This is a preparatory PR to add additional APIs (Pinning Service API) easily
to cluster.
Instead of copy-pasting most of what the REST API does, I have refactored so
that the whole configuration, routing and request-handling utilities can be
re-used.
The worst part has been to divide the test between tests that test core
(common.API) functionality and tests that test specific REST API endpoint
functionality. I could not get away without an additional common/test package
to provide functions that are used from both places. This is a side effect of
testing both http and libp2p endpoints for every request etc.
* Libp2p protectors no longer needed, use PSK directly
* Generate cluster 32-byte secret here (helper gone from pnet)
* Switch to go-log/v2 in all places
* DHT bootstrapping not needed. Adjust DHT options for tests.
* Do not rely on dissappeared CidToDsKey and DsKeyToCid functions fro dshelp.
* Disable QUIC (does not support private networks)
* Fix tests: autodiscovery started working properly
* add ipv6 listening addresses to the default config
* ipfsproxy: support multiple listeners. Add default ipv6.
* mm
* restapi: support multiple listen addresses. enable ipv6
* cluster_config: format default listen addresses
* commands: update for multiple listeners. Fix randomports for udp and ipv6.
* ipfs-cluster-service: fix randomports test
* multiple listeners: fix remaining tests
* golint
* Disable ipv6 in defaults
It is not supported by docker by default. It is not supported in travis-CI
build environments. User can enable it now manually.
* proxy: disable ipv6 in test
* ipfshttp: fix test
Co-authored-by: @RubenKelevra <cyrond@gmail.com>
This adds a new "crdt" consensus component using go-ds-crdt.
This implies several refactors to fully make cluster consensus-component
independent:
* Delete mapstate and fully adopt dsstate (after people have migrated).
* Return errors from state methods rather than ignoring them.
* Add a new "datastore" modules so that we can configure datastores in the
main configuration like other components.
* Let the consensus components fully define the "state.State". Thus, they do
not receive the state, they receive the storage where we put the state (a
go-datastore).
* Allow to customize how the monitor component obtains Peers() (the current
peerset), including avoiding using the current peerset. At the moment the
crdt consensus uses the monitoring component to define the current peerset.
Therefore the monitor component cannot rely on the consensus component to
produce a peerset.
* Re-factor/re-implementation of "ipfs-cluster-service state"
operations. Includes the dissapearance of the "migrate" one.
The CRDT consensus component defines creates a crdt-datastore (with ipfs-lite)
and uses it to intitialize a dssate. Thus the crdt-store is elegantly
wrapped. Any modifications to the state get automatically replicated to other
peers. We store all the CRDT DAG blocks in the local datastore.
The consensus components only expose a ReadOnly state, as any modifications to
the shared state should happen through them.
DHT and PubSub facilities must now be created outside of Cluster and passed in
so they can be re-used by different components.
This commit adds support for OpenCensus tracing
and metrics collection. This required support for
context.Context propogation throughout the cluster
codebase, and in particular, the ipfscluster component
interfaces.
The tracing propogates across RPC and HTTP boundaries.
The current default tracing backend is Jaeger.
The metrics currently exports the metrics exposed by
the opencensus http plugin as well as the pprof metrics
to a prometheus endpoint for scraping.
The current default metrics backend is Prometheus.
Metrics are currently exposed by default due to low
overhead, can be turned off if desired, whereas tracing
is off by default as it has a much higher performance
overhead, though the extent of the performance hit can be
adjusted with smaller sampling rates.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
We should deprecate passing in Host/Port in the config,
but in the meantime, it hardcoded /dns4/, meaning that if
someone placed an ipv6 address in there things would break badly
and weirdly.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Before we resolved all /dns*/ multiaddresses before we used them.
When using HTTPs, the Go HTTP Client only sees the resolved IP address
and it is unable to negotiate TLS with a cerficate because the request
is not going to the hostname the certificate is signed for, but to
the IP. This leverages a recent feature in go-multiaddr-net
and uses directly the user-provided hostname.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This removes PeerAddr and uses APIAddr directly, figuring out if it is
a Peer multiaddress or not.
PeerAddr is actually kept for compatiblity.
It also fixes a bad panic when resolving returned 0 results
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
The IPFS() methods returns an ipfs Shell pointing to the ipfs-cluster
proxy endpoint. The location can be customized (via ProxyAddr configuration
option) or it is assumed to be the same as the PeerAddr/APIAddr, with
a different port (the default).
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This adds support for libp2p-tunneled http to the rest api component.
If PeerAddr is specified in the configuration, then we will create a
libp2p host and communicate with the API using that.
Tests run now in both http and libp2p mode.
Note: pnet support not included, but coming up
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This allows taking advantage of connection keep alive by having the
api client re-use the same connection. Additionally, an option
to close connections after every request is provided.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
* Set default logging facility
* Remove old keep-alive comment in tests
* Use a port for TestPeersWithErrors which is not default
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This adds the pakage api/rest/client which implements a go-client
for the REST API component. It also update the ipfs-cluster-ctl
tool to rely on it.
Originally, I wanted this to live it in it's own separate repository,
but the api client uses /api/types.go, which is part of cluster.
Therefore it would need to import all of cluster as a dependency.
ipfs-cluster-ctl would also need to import go-ipfs-cluster-api-client
as a dependency, creating circular gx deps which would be a mess to
maintain.
Only the splitting of cluster in multiple repositories (at least for
api, rest, ipfs-cluster-ctl, rest/client and test) would allow better
dependency management by allowing rest/client and the ctl tool
to only import what is needed, but this is something which brings
maintenance costs and can probably wait a bit until cluster is more stable.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
