This commit fixes#810 and adds block streaming to the final destinations when
adding. This should add major performance gains when adding data to clusters.
Before, everytime cluster issued a block, it was broadcasted individually to
all destinations (new libp2p stream), where it was block/put to IPFS (a single
block/put http roundtrip per block).
Now, blocks are streamed all the way from the adder module to the ipfs daemon,
by making every block as it arrives a single part in a multipart block/put
request.
Before, block-broadcast needed to wait for all destinations to finish in order
to process the next block. Now, buffers allow some destinations to be faster
than others while sending and receiving blocks.
Before, if a block put request failed to be broadcasted everywhere, an error
would happen at that moment.
Now, we keep streaming until the end and only then report any errors. The
operation succeeds as long as at least one stream finished successfully.
Errors block/putting to IPFS will not abort streams. Instead, subsequent
blocks are retried with a new request, although the method will return an
error when the stream finishes if there were errors at any point.
This commit makes all the changes to make Peers() a streaming call.
While Peers is usually a non problematic call, for consistency, all calls
returning collections assembled through broadcast to cluster peers are now
streaming calls.
This commit continues the work of taking advantage of the streaming
capabilities in go-libp2p-gorpc by improving the ipfsconnector and pintracker
components.
StatusAll and RecoverAll methods are now streaming methods, with the REST API
output changing accordingly to produce a stream of GlobalPinInfos rather than
a json array.
pin/ls request to the ipfs daemon now use ?stream=true and avoid having to
load the full pinset map on memory. StatusAllLocal and RecoverAllLocal
requests to the pin tracker stream all the way and no longer store the full
pinset, and the full PinInfo status slice before sending it out.
We have additionally switched to a pattern where streaming methods receive the
channel as an argument, allowing the caller to decide on whether to launch a
goroutine, do buffering etc.
This commit introduces the new go-libp2p-gorpc streaming capabilities for
Cluster. The main aim is to work towards heavily reducing memory usage when
working with very large pinsets.
As a side-effect, it takes the chance to revampt all types for all public
methods so that pointers to static what should be static objects are not used
anymore. This should heavily reduce heap allocations and GC activity.
The main change is that state.List now returns a channel from which to read
the pins, rather than pins being all loaded into a huge slice.
Things reading pins have been all updated to iterate on the channel rather
than on the slice. The full pinset is no longer fully loaded onto memory for
things that run regularly like StateSync().
Additionally, the /allocations endpoint of the rest API no longer returns an
array of pins, but rather streams json-encoded pin objects directly. This
change has extended to the restapi client (which puts pins into a channel as
they arrive) and to ipfs-cluster-ctl.
There are still pending improvements like StatusAll() calls which should also
stream responses, and specially BlockPut calls which should stream blocks
directly into IPFS on a single call.
These are coming up in future commits.
This adds a new allocations field to add response objects which
provides the cluster peers to which the content has been allocated.
In the case of sharded dags, it provides peers for the current shard.
pinsvcapi: do not cache peer information here as all the needed information is
in the status objects.
This adds ipfs_addresses as a field broadcasted with the ping metrics.
This does 3 things:
- Add a NoPin option to the adder. When set to true, the adding process does not
send a pin in the end.
- When user-allocations are set and local=true happens, we do not overwrite
the allocations returned by the allocator to include the local peer
anymore, as this could alter user-allocations.
- Some code improvement (remove pointers).
This fixes#1213. It adds partial support for the Pinning Services API with some caveats:
* RequestIDs == CIDs. This is a violation of the spec, as request IDs will not be unique etc.
* Pagination, name matching, metadata matching, ordering etc. are not supported in the List endpoint.
* The List endpoint only supports status filtering and cid query parameter.
* Created time property is not supported and always set to Now()
There is more work to do here: cleanup, extract useful types etc. and TESTS.
This will facilitate building outputs for the Pinning Services API, saving a
round trip to query the cluster State, since all the needed information
already comes from the PinTracker, which has already accessed the state.
Since the pintracker already included a state attribute (Name), we are simply
going down that path.
This allows to specifically request status for several CIDs as
provided in the "cids" query parameter, instead of request status for
all CIDs.
In this case, the filter is ignored.
Fixes#1554
Fixes: peer names unset for remote peers
This adds an IPFS field to pin status information (PinInfoShort).
It has not been easy to add this, given that the IPFS ID is something that
comes from outside of cluster (unlike the peer name). After several tries I
have settled in the following things:
- Use the ping metric to send out peer names and IPFS IDs to the peers in the
cluster.
- Cache the latest known IPFS ID (if IPFS dies we should still be setting
the ID).
- Provide an RPC method for the Pintracker to obtain IPFS ID from the cache.
- Given we now know information for peernames and IPFS IDs from other peers,
we can use that information even if the requests to them error or we are not
contacting (i.e. peers allocated as remote are not queried for status). We can
use the information from the last received ping metric.
- This means we should keep metrics around even if peers go away, at least for
a while rather than deleting them as soon as we detect that they have expired.
Puting it all together we now have a system to gossip peer information around on top
of the ping metrics.
The pin objects now include an Origin field which was set to
multiaddr.Multiaddr.
This type is an interface and has no idea how to deserialize itself. As a
result, an state json export cannot be deseralized.
We had already added a wrapper Multiaddr type to handle these issues. I
believe this fix does not affect anything else other than fixing
UnmarshalJSON. PB types are deserialized by hand and it should not make a
difference.
This adds a Timestamp field to the pin objects. This allows to track when they were pinned.
This:
* Allows the pin-tracker to actually show accurate information on when the pin
entered the system for pins that are not part of ongoing operations
(currently it shows time.Now())
* Adds support for reporting timestamp on a pinning services api.
This fixes the issue about partitions not being picked based
on the amount of freespace available in them.
It additionally removes the metrics registry and carries information directly
in the metric.
Metrics have two additional fields: Weight and Partitionable.
Informers have been updated to make use of these fields. Partitions have
weights that equals to the weight of the metrics under them.
Older cluster versions will not set these fields. Partitionable is false by
default and weight has a GetWeight() function to convert value->weight when
unset. This provides backwards compatibility for the freespace metric.
