The operation tracker was not setting some fields correctly when producing
PinInfo objects. Additionally, recover operations were submitted with empty
pin objects, which resulted in the status for pins sent on recover operations
to be missing fields.
These changes fix a few bugs in the PinSVC API and have been discovered by
running the compliance tool at https://github.com/ipfs-shipyard/pinning-service-compliance.
In particular, the error objects did not respect the spec, filters and counts
were not done right. An additional PR will follow with fixes to the pintracker
because some pin status information was not correctly set.
This deprecates the Metadata protobuf map and starts serializing metadata as an
array that is always sorted by the metadata key.
This should resolve the issue that an state export file can be imported
resulting in two different CRDT dags because the binary representation of the
pin can arbitrarily change depending on how the keys in the map are listed,
and thus the CID of a delta is impacted.
So with this, a state export should result in exactly the same DAG regardless
of where it is imported.
I think this should fix the issue. As solution we make every retry with a
temporary channel and copy results to the final channel which is only closed
by us. This only affects streaming methods.
This new component broadcasts metrics about the current size of the pinqueue,
which can in turn be used to inform allocations.
It has a weight_bucket_size option that serves to divide the actual size by a
given factor. This allows considering peers with similar queue sizes to have
the same weight.
Additionally, some changes have been made to the balanced allocator so that a
combination of tags, pinqueue sizes and free-spaces can be used. When
allocating by [<tag>, pinqueue, freespace], the allocator will prioritize
choosing peers with the smallest pin queue weight first, and of those with the
same weight, it will allocate based on freespace.
* Gauge for total number of ipfs pins
* Counter for pin/add
* Counter for pin/add errors
* Counter for Block/Puts
* Counter for blocks added
* Counter for block/added size
* Counted for block/added errors
We were currently tracking this metrics as a counter (SUM). The number is
good, but conceptually this is more a gauge (LastValue), given it can go down.
Thus we switch it by tracking the aggregation numbers directy in the operation
tracker.
Adding keeps retrying indefinitely to send blocks to ipfs. When ipfs is down this:
* Stores all errors
* Keeps retrying
* Additionally, forgot to close bodies
This resulted in memory leaks. The new behaviour does not keep retrying. And
response bodies are closed after reading.