ipfs-cluster/pintracker/optracker/operation.go

314 lines
7.2 KiB
Go
Raw Normal View History

Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
package optracker
import (
"context"
"fmt"
"strings"
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
"sync"
"time"
cid "github.com/ipfs/go-cid"
"github.com/ipfs/ipfs-cluster/api"
"go.opencensus.io/trace"
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
)
//go:generate stringer -type=OperationType
// OperationType represents the kinds of operations that the PinTracker
// performs and the operationTracker tracks the status of.
type OperationType int
const (
// OperationUnknown represents an unknown operation.
OperationUnknown OperationType = iota
// OperationPin represents a pin operation.
OperationPin
// OperationUnpin represents an unpin operation.
OperationUnpin
// OperationRemote represents an noop operation
OperationRemote
// OperationShard represents a meta pin. We don't
// pin these.
OperationShard
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
)
//go:generate stringer -type=Phase
// Phase represents the multiple phase that an operation can be in.
type Phase int
const (
// PhaseError represents an error state.
PhaseError Phase = iota
// PhaseQueued represents the queued phase of an operation.
PhaseQueued
// PhaseInProgress represents the operation as in progress.
PhaseInProgress
// PhaseDone represents the operation once finished.
PhaseDone
)
// Operation represents an ongoing operation involving a
// particular Cid. It provides the type and phase of operation
// and a way to mark the operation finished (also used to cancel).
type Operation struct {
ctx context.Context
cancel func()
// RO fields
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
opType OperationType
pin api.Pin
// RW fields
mu sync.RWMutex
phase Phase
attemptCount int
priority bool
error string
ts time.Time
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
// NewOperation creates a new Operation.
func NewOperation(ctx context.Context, pin api.Pin, typ OperationType, ph Phase) *Operation {
ctx, span := trace.StartSpan(ctx, "optracker/NewOperation")
defer span.End()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
ctx, cancel := context.WithCancel(ctx)
return &Operation{
ctx: ctx,
cancel: cancel,
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
pin: pin,
opType: typ,
phase: ph,
attemptCount: 0,
priority: false,
ts: time.Now(),
error: "",
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
}
// String returns a string representation of an Operation.
func (op *Operation) String() string {
var b strings.Builder
fmt.Fprintf(&b, "type: %s\n", op.Type().String())
fmt.Fprint(&b, "pin:\n")
pinstr := op.Pin().String()
pinstrs := strings.Split(pinstr, "\n")
for _, s := range pinstrs {
fmt.Fprintf(&b, "\t%s\n", s)
}
fmt.Fprintf(&b, "phase: %s\n", op.Phase().String())
fmt.Fprintf(&b, "attemptCount: %d\n", op.AttemptCount())
fmt.Fprintf(&b, "error: %s\n", op.Error())
fmt.Fprintf(&b, "timestamp: %s\n", op.Timestamp().String())
return b.String()
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
// Cid returns the Cid associated to this operation.
func (op *Operation) Cid() cid.Cid {
2022-01-28 18:35:46 +00:00
return op.pin.Cid
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
// Context returns the context associated to this operation.
func (op *Operation) Context() context.Context {
return op.ctx
}
// Cancel will cancel the context associated to this operation.
func (op *Operation) Cancel() {
_, span := trace.StartSpan(op.ctx, "optracker/Cancel")
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
op.cancel()
span.End()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
// Phase returns the Phase.
func (op *Operation) Phase() Phase {
var ph Phase
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
op.mu.RLock()
ph = op.phase
op.mu.RUnlock()
return ph
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
// SetPhase changes the Phase and updates the timestamp.
func (op *Operation) SetPhase(ph Phase) {
_, span := trace.StartSpan(op.ctx, "optracker/SetPhase")
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
op.mu.Lock()
{
op.phase = ph
op.ts = time.Now()
}
op.mu.Unlock()
span.End()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
// AttemptCount returns the number of times that this operation has been in
// progress.
func (op *Operation) AttemptCount() int {
var retries int
op.mu.RLock()
retries = op.attemptCount
op.mu.RUnlock()
return retries
}
// IncAttempt does a plus-one on the AttemptCount.
func (op *Operation) IncAttempt() {
op.mu.Lock()
op.attemptCount++
op.mu.Unlock()
}
// PriorityPin returns true if the pin has been marked as priority pin.
func (op *Operation) PriorityPin() bool {
var p bool
op.mu.RLock()
p = op.priority
op.mu.RUnlock()
return p
}
// SetPriorityPin returns true if the pin has been marked as priority pin.
func (op *Operation) SetPriorityPin(p bool) {
op.mu.Lock()
op.priority = p
op.mu.Unlock()
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
// Error returns any error message attached to the operation.
func (op *Operation) Error() string {
var err string
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
op.mu.RLock()
err = op.error
op.mu.RUnlock()
return err
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
// SetError sets the phase to PhaseError along with
// an error message. It updates the timestamp.
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
func (op *Operation) SetError(err error) {
_, span := trace.StartSpan(op.ctx, "optracker/SetError")
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
op.mu.Lock()
{
op.phase = PhaseError
op.error = err.Error()
op.ts = time.Now()
}
op.mu.Unlock()
span.End()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
// Type returns the operation Type.
func (op *Operation) Type() OperationType {
return op.opType
}
// Pin returns the Pin object associated to the operation.
func (op *Operation) Pin() api.Pin {
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
return op.pin
}
// Timestamp returns the time when this operation was
// last modified (phase changed, error was set...).
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
func (op *Operation) Timestamp() time.Time {
var ts time.Time
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
op.mu.RLock()
ts = op.ts
op.mu.RUnlock()
return ts
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
// Cancelled returns whether the context for this
// operation has been cancelled.
func (op *Operation) Cancelled() bool {
ctx, span := trace.StartSpan(op.ctx, "optracker/Cancelled")
_ = ctx
defer span.End()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
select {
case <-op.ctx.Done():
return true
default:
return false
}
}
// ToTrackerStatus returns an api.TrackerStatus reflecting
// the current status of this operation. It's a translation
// from the Type and the Phase.
func (op *Operation) ToTrackerStatus() api.TrackerStatus {
typ := op.Type()
ph := op.Phase()
switch typ {
case OperationPin:
switch ph {
case PhaseError:
return api.TrackerStatusPinError
case PhaseQueued:
return api.TrackerStatusPinQueued
case PhaseInProgress:
return api.TrackerStatusPinning
case PhaseDone:
return api.TrackerStatusPinned
default:
return api.TrackerStatusUndefined
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
case OperationUnpin:
switch ph {
case PhaseError:
return api.TrackerStatusUnpinError
case PhaseQueued:
return api.TrackerStatusUnpinQueued
case PhaseInProgress:
return api.TrackerStatusUnpinning
case PhaseDone:
return api.TrackerStatusUnpinned
default:
return api.TrackerStatusUndefined
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
case OperationRemote:
return api.TrackerStatusRemote
case OperationShard:
return api.TrackerStatusSharded
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
default:
return api.TrackerStatusUndefined
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
// TrackerStatusToOperationPhase takes an api.TrackerStatus and
// converts it to an OpType and Phase.
func TrackerStatusToOperationPhase(status api.TrackerStatus) (OperationType, Phase) {
switch status {
case api.TrackerStatusPinError:
return OperationPin, PhaseError
case api.TrackerStatusPinQueued:
return OperationPin, PhaseQueued
case api.TrackerStatusPinning:
return OperationPin, PhaseInProgress
case api.TrackerStatusPinned:
return OperationPin, PhaseDone
case api.TrackerStatusUnpinError:
return OperationUnpin, PhaseError
case api.TrackerStatusUnpinQueued:
return OperationUnpin, PhaseQueued
case api.TrackerStatusUnpinning:
return OperationUnpin, PhaseInProgress
case api.TrackerStatusUnpinned:
return OperationUnpin, PhaseDone
case api.TrackerStatusRemote:
return OperationRemote, PhaseDone
case api.TrackerStatusSharded:
return OperationShard, PhaseDone
default:
return OperationUnknown, PhaseError
}
}