This commit adds support for OpenCensus tracing
and metrics collection. This required support for
context.Context propogation throughout the cluster
codebase, and in particular, the ipfscluster component
interfaces.
The tracing propogates across RPC and HTTP boundaries.
The current default tracing backend is Jaeger.
The metrics currently exports the metrics exposed by
the opencensus http plugin as well as the pprof metrics
to a prometheus endpoint for scraping.
The current default metrics backend is Prometheus.
Metrics are currently exposed by default due to low
overhead, can be turned off if desired, whereas tracing
is off by default as it has a much higher performance
overhead, though the extent of the performance hit can be
adjusted with smaller sampling rates.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
Added a fail case where an invalid filter is passed in.
Update `api.rpcClient.Call` to `api.rpcClient.CallContext` and pass
in the Request context r.Context() so that context can be cancelled
when the request is cancelled by caller
Fixes#445
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Added filter option to `ipfs-cluster-ctl status`
When the --filter is passed, it will only fetch the peer information
where status of the pin matches with the filter value.
Valid filter values are tracker status types(i.e., "pinned",
"pin_error", "unpinning" etc), an alias of tracker status type (i.e.,
"queued" or "error"), comma separated list of tracker status type
and/or it aliases(i.e., "error,pinning")
On passing invalid filter value no status information will be shown
In particular, the filter would remove elements from []GlobalPinInfo
when none of the peers in GlobalPinInfo match the filter. If one peer
in the GlobalPinInfo matches the filter, the whole object is returned,
including the information for the other peers which may or not match it.
filter option works on statusAll("GET /pins"). For fetching pin status
for a CID("GET /pins/<cid>"), filter option would have no effect
Fixes#445
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Move ctl-health sharness tests to apprpriate file
Since the API is using the RPC mock to request metrics and it always
returns a mocked test metric we might just do c.Metrics("somemetricstype")
and check that there is no error. Here we just want to check that the
client is hitting an API endpoint (and understands the response).
Fixes#587
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Added API and client tests for GET /monitor/metrics/{metrics_type}
Fixes#587
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
This straigthens some mistakes with the outputs of the /add endpoints.
Currently, we had exactly the same output format which:
* was not exactly the ipfs API output format but was sort of similar
* made some weird concessions to be compatible (like having a string-type "size")
* was not aligned with Cluster API conventions (lowercase keys)
This corrects all this:
* The Cluster API /add output format now uses the right types and lowercase keys.
* `Hash` is now `Cid`, because the field carries a Cid.
* We copy error handling with request trailers from IPFS, and avoid carrying the
errors in the output objects.
* The proxy now returns exactly the types as ipfs would
* We add the X-Chunked-Output: 1 header, which is custom and redundant, but
otherwise breaks js-ipfs-api integrations with the /add endpoint.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.
This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
api client test
api test
one sharness test
refactoring of testingData for access from other packages
always wrap files added by cluster
remove unused flag 'progress'
changes to support hidden flag
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
This uses go-libp2p-kad-dht as routing provider for the Cluster Peers.
This means that:
* A cluster peer can discover other Cluster peers even if they are
not in their peerstore file.
* We remove a bunch of code sending and receiving peers multiaddresses
when a new peer was added to the Cluster.
* PeerAdd now takes an ID and not a multiaddress. We do not need to
ask the new peer which is our external multiaddress nor broadcast
the new multiaddress to everyone. This will fix problems when bootstrapping
a new peer to the Cluster while not all the other peers are online.
* Adding a new peer does not mean to open connections to all peers
anymore. The number of connections will be made according to the DHT
parameters (this is good to have for future work)
The that detecting a peer addition in the watchPeers() function does
no longer mean that we have connected to it or that we know its
multiaddresses. Therefore it's no point to save the peerstore in these
events anymore.
Here a question opens, should we save the peerstore at all, and should we
save multiaddresses only for cluster peers, or for everyone known?
Currently, the peerstore is only updated on clean shutdown,
and it is updated with all the multiaddresses known, and not limited to
peer IDs in the cluster, (because, why not).
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This adds support for libp2p-tunneled http to the rest api component.
If PeerAddr is specified in the configuration, then we will create a
libp2p host and communicate with the API using that.
Tests run now in both http and libp2p mode.
Note: pnet support not included, but coming up
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Added go tests
Refactored cluster connect graph to new file
Refactored dot file printing to new repo
Fixed code climate issues
Added sharness test
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>