The go-ds-crdt upgrade disables multi-head-processing by default again. We see
this causes a lot of branching.
We however increase the number of workers. With large deltas, it may be
possible that all the 5 workers are busy downloading a delta or processing
them, while we potentially have hundreds of children in the DAG. Thus it is
not bad to attempt to do more things in parallel.
This fixes#1213. It adds partial support for the Pinning Services API with some caveats:
* RequestIDs == CIDs. This is a violation of the spec, as request IDs will not be unique etc.
* Pagination, name matching, metadata matching, ordering etc. are not supported in the List endpoint.
* The List endpoint only supports status filtering and cid query parameter.
* Created time property is not supported and always set to Now()
There is more work to do here: cleanup, extract useful types etc. and TESTS.
This will facilitate building outputs for the Pinning Services API, saving a
round trip to query the cluster State, since all the needed information
already comes from the PinTracker, which has already accessed the state.
Since the pintracker already included a state attribute (Name), we are simply
going down that path.
This allows to specifically request status for several CIDs as
provided in the "cids" query parameter, instead of request status for
all CIDs.
In this case, the filter is ignored.
It is not good to add something locally only to pin it somewhere else:
* The locally used space is not GCed automatically or anything and is lost
* Pay the penalty of having to copy things somewhere else
Given that every pin and block/put writes something to IPFS and thus increases
the repo size, a while ago we added a check to let the IPFS connector directly
trigger the sending of metrics every 10 of such requests. This was meant to
update the metrics more often so that balancing happened more granularly
(particularly the freespace one).
In practice, on a cluster that receives several hundreds of pin/adds
operations in a few seconds, this is just bad.
So:
* We disable by default the whole thing.
* We add a new InformerTriggerInterval configuration option to enable the thing.
* Fix a bug that made this always call the first informer, which may not
have been the freespace one).
Fixes#1554
Fixes: peer names unset for remote peers
This adds an IPFS field to pin status information (PinInfoShort).
It has not been easy to add this, given that the IPFS ID is something that
comes from outside of cluster (unlike the peer name). After several tries I
have settled in the following things:
- Use the ping metric to send out peer names and IPFS IDs to the peers in the
cluster.
- Cache the latest known IPFS ID (if IPFS dies we should still be setting
the ID).
- Provide an RPC method for the Pintracker to obtain IPFS ID from the cache.
- Given we now know information for peernames and IPFS IDs from other peers,
we can use that information even if the requests to them error or we are not
contacting (i.e. peers allocated as remote are not queried for status). We can
use the information from the last received ping metric.
- This means we should keep metrics around even if peers go away, at least for
a while rather than deleting them as soon as we detect that they have expired.
Puting it all together we now have a system to gossip peer information around on top
of the ping metrics.