This fixes two bugs. First, the "blockPut response CID does not match the
multihash" warning was coming up when it shouldn't. Particularly, the
multipart reader called Node() several times for the same block, resulting in
CIDs been removed from the Seen set, and causing the warning when there were
several blocks (usually the empty dir block).
This also means we were counting Blockputs (and total data added) wrong in the
metrics, double-counting some blocks as these were recorded in Node() calls.
The fix makes the tracking in Next(), which is only called once for each
block. To avoid timing issues between Block reads from the channel and
blockput responses, the Seen set now stores how many times we have seen a
block. Thus a duplicated block that will get two BlockPut responses will not
trigger a warning regardless of the time when those responses arrive.
Fixes#1706.
It has been observed that some peers have a growing number of goroutines,
usually stuck in go-libp2p-gorpc.MultiStream() function, which is waiting to
read items from the arguments channel.
We suspect this is due to aborted /add requests. In situations when the add
request is aborted or fails, Finalize() is never called and the blocks channel
stays open, so MultiStream() can never exit, and the BlockStreamer can never
stop streaming etc.
As a fix, we added the requirement to call Close() when we stop using a
ClusterDAGService (error or not). This should ensure that the blocks channel
is always closed and not just on Finalize().
This implements committing batches on shutdown properly.
Now the batchWorker will only finish when there are no more things queued to
be included in the final batch(es).
LogPin/Unpin operations will fail while we are shutting down and they cannot
be included in the batch.
This attempt to commit any pending batches when the crdt component is being
shutudown. A commit should succeed if the new DAG node is created, heads are
replaced and broadcast.
The latest version of CRDT ensure that the datastore does not unnecessarily
gets marked as dirty when a broadcasted head cannot be fetched/processed, so
the side effect of publishing a head before shutting down should be under
control at least.
The Pinning Services API standard mandates Bearer token authentication.
This adds JWT bearer token authentication to the IPFS Cluster REST and PINSVC
APIs.
The basic_auth_credentials configuration option needs to be not null and have
at least one username/passwords entry.
A user authenticated via Basic Auth can then "POST /token" and obtain a json
object:
```json { "token" : "<JWTtoken>" } ```
The JWT token has the "iss" (issuer) field set to the Basic auth user that
authorized its creation and is HMAC-signed with its password.
When basic-auth-credentials are set, the APIs will verify that requests come
with either Basic Auth authorization header or with a Bearer token
authorization header.
Bearer tokens will be decoded and the signature will be verified against the
password of the issuer.
At the moment we provide no support to revoke tokens, set "expiration date",
"not before" etc, but this may come in the future.
The operation tracker was not setting some fields correctly when producing
PinInfo objects. Additionally, recover operations were submitted with empty
pin objects, which resulted in the status for pins sent on recover operations
to be missing fields.
These changes fix a few bugs in the PinSVC API and have been discovered by
running the compliance tool at https://github.com/ipfs-shipyard/pinning-service-compliance.
In particular, the error objects did not respect the spec, filters and counts
were not done right. An additional PR will follow with fixes to the pintracker
because some pin status information was not correctly set.
