4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
This commit attempts to fix race issues in the maptracker since the
introduction of the OperationTracker.
There were two main problems:
* Duplicity tracking the state both in the state map and the opTracker
* Non atomiciy of operations with different threads being able to affect
other threads operations.
A test performing random Track/Untracks on the same Cid quickly showed
that items would sometimes stay as pin_queued or pin_unqueued. That happened
because operations could be cancelled under the hood by a different request,
while leaving the map status untouched.
It was not simply to deal with this issues without a refactoring.
First, the state map has been removed, and the operation tracker now provides
status information for any Cid. This implies that the tracker keeps all
operations and operations have a `PhaseDone`. There's also a
new `OperationRemote` type.
Secondly, operations are only created in the tracker and can only be removed
by their creators (they can be overwritten by other operations though).
Operations cannot be accessed directly and modifications are limited to setting
Error for PhaseDone operations.
After created, *Operations are queued in the pinWorker queues which handle any
status updates. This means, that, even when an operation has been removed from
the tracker, status updates will not interfere with any other newer operations.
In the maptracker, only the Unpin worker Cleans operations once processed. A
sucessful unpin is the only way that a delete() happens in the tracker map.
Otherwise, operations stay there until a newer operation for the Cid arrives
and 1) cancels the existing one 2) takes its place. The tracker refuses to
create a new operation if a similar "ongoing" operation of the same type
exists.
The final change is that Recover and RecoverAll() are not async and play by the
same rules as Track() and Untrack(), queueing the items to be recovered.
Note: for stateless pintracker, the tracker will need to Clean() operation
of type OperationPin as well, and complement the Status reported
by the tracker with those coming from IPFS.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
When StateSync() runs and triggers Untrack() on items
that have just been removed from the state but on which
Untrack() is underway, the operation tracker would be
reset to phase queued and in some cases stay so.
Also happened for Track()
This caused failures of TestClustersPin as SyncStatus()
is triggered regularly while Tracks() and Untracks() happen.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Avoid writing tests which will hang indefinitely on failure conditions.
Introduce TODOs.
Rename some vars to more explicit names.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This creates a race condition where the items may have been
already pinned before the operation is registered in the tracker.
This may result in operations being left in the tracker and potentially
never completed.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
come off the pin/unpin channels.
Also fix a race condition in the operationTracker.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
The TrackerStatuses were starting to be used to convey the inflight
status of an 'operation', instead of just the status of the Pin.
I have separated out any thing related to 'operations' and
an operation's 'phases'.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
ipfshttp: cancel POST request when timeout reached
ipfshttp/config: fix config test
ipfshttp: use struct styling for multi-line func calls
ipfshttp/config: add general ClientTimeout
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
This fixes#326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.
It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.
Documentation for the configuration file has been updated.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This PR replaces ReplicationFactor with ReplicationFactorMax
and ReplicationFactor min.
This allows a CID to be pinned even though the desired
replication factor (max) is not reached, and prevents triggering
re-pinnings when the replication factor has not crossed the
lower threshold (min).
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This adds API, RPC calls to support RecoverAllLocal() (and expose RecoverLocal()
on the Rest API too). cluster-ctl is updated accordingly.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This also generates a default configuration section when it
doesn't exist, so it's backwards compatible.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
