ipfs-cluster/pintracker/optracker/operationtracker_test.go

259 lines
7.2 KiB
Go
Raw Permalink Normal View History

package optracker
import (
"context"
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
"errors"
"testing"
"github.com/ipfs-cluster/ipfs-cluster/api"
"github.com/ipfs-cluster/ipfs-cluster/test"
)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
func testOperationTracker(t *testing.T) *OperationTracker {
ctx := context.Background()
return NewOperationTracker(ctx, test.PeerID1, test.PeerName1)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
func TestOperationTracker_TrackNewOperation(t *testing.T) {
ctx := context.Background()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
opt := testOperationTracker(t)
op := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationPin, PhaseQueued)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Run("track new operation", func(t *testing.T) {
if op == nil {
t.Fatal("nil op")
}
if op.Phase() != PhaseQueued {
t.Error("bad phase")
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if op.Type() != OperationPin {
t.Error("bad type")
}
if op.Canceled() != false {
t.Error("should not be canceled")
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if op.ToTrackerStatus() != api.TrackerStatusPinQueued {
t.Error("bad state")
}
})
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Run("track when ongoing operation", func(t *testing.T) {
op2 := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationPin, PhaseInProgress)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if op2 != nil {
t.Fatal("should not have created new operation")
}
})
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Run("track of different type", func(t *testing.T) {
op2 := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationUnpin, PhaseQueued)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if op2 == nil {
t.Fatal("should have created a new operation")
}
if !op.Canceled() {
t.Fatal("should have canceled the original operation")
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
})
t.Run("track of same type when done", func(t *testing.T) {
op2 := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationPin, PhaseDone)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if op2 == nil {
t.Fatal("should have created a new operation")
}
op3 := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationPin, PhaseQueued)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if op3 == nil {
t.Fatal("should have created a new operation when other is in Done")
}
})
t.Run("track of same type when error", func(t *testing.T) {
op4 := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationUnpin, PhaseError)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if op4 == nil {
t.Fatal("should have created a new operation")
}
op5 := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationUnpin, PhaseQueued)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if op5 == nil {
t.Fatal("should have created a new operation")
}
})
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
func TestOperationTracker_Clean(t *testing.T) {
ctx := context.Background()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
opt := testOperationTracker(t)
op := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationPin, PhaseQueued)
op2 := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationUnpin, PhaseQueued)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Run("clean older operation", func(t *testing.T) {
opt.Clean(ctx, op)
st, ok := opt.Status(ctx, test.Cid1)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if !ok || st != api.TrackerStatusUnpinQueued {
t.Fatal("should not have cleaned the latest op")
}
})
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Run("clean current operation", func(t *testing.T) {
opt.Clean(ctx, op2)
_, ok := opt.Status(ctx, test.Cid1)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if ok {
t.Fatal("should have cleaned the latest op")
}
})
}
func TestOperationTracker_Status(t *testing.T) {
ctx := context.Background()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
opt := testOperationTracker(t)
opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationRemote, PhaseDone)
st, ok := opt.Status(ctx, test.Cid1)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if !ok || st != api.TrackerStatusRemote {
t.Error("should provide status remote")
}
_, ok = opt.Status(ctx, test.Cid1)
if !ok {
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Error("should signal unexistent status")
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
func TestOperationTracker_SetError(t *testing.T) {
ctx := context.Background()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
opt := testOperationTracker(t)
opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationPin, PhaseDone)
opt.SetError(ctx, test.Cid1, errors.New("fake error"))
pinfo := opt.Get(ctx, test.Cid1, api.IPFSID{})
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if pinfo.Status != api.TrackerStatusPinError {
t.Error("should have updated the status")
}
if pinfo.Error != "fake error" {
t.Error("should have set the error message")
}
opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationUnpin, PhaseQueued)
opt.SetError(ctx, test.Cid1, errors.New("fake error"))
st, ok := opt.Status(ctx, test.Cid1)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if !ok || st != api.TrackerStatusUnpinQueued {
t.Error("should not have set an error on in-flight items")
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
func TestOperationTracker_Get(t *testing.T) {
ctx := context.Background()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
opt := testOperationTracker(t)
opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationPin, PhaseDone)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Run("Get with existing item", func(t *testing.T) {
pinfo := opt.Get(ctx, test.Cid1, api.IPFSID{})
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if pinfo.Status != api.TrackerStatusPinned {
t.Error("bad status")
}
if !pinfo.Cid.Equals(test.Cid1) {
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Error("bad cid")
}
if pinfo.Peer != test.PeerID1 {
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Error("bad peer ID")
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
})
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Run("Get with unexisting item", func(t *testing.T) {
pinfo := opt.Get(ctx, test.Cid2, api.IPFSID{})
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if pinfo.Status != api.TrackerStatusUnpinned {
t.Error("bad status")
}
if !pinfo.Cid.Equals(test.Cid2) {
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Error("bad cid")
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if pinfo.Peer != test.PeerID1 {
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
t.Error("bad peer ID")
}
})
}
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
func TestOperationTracker_GetAll(t *testing.T) {
ctx := context.Background()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
opt := testOperationTracker(t)
opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationPin, PhaseInProgress)
pinfos := opt.GetAll(ctx, api.IPFSID{})
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if len(pinfos) != 1 {
t.Fatal("expected 1 item")
}
if pinfos[0].Status != api.TrackerStatusPinning {
t.Fatal("bad status")
}
}
func TestOperationTracker_OpContext(t *testing.T) {
ctx := context.Background()
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
opt := testOperationTracker(t)
op := opt.TrackNewOperation(ctx, api.PinCid(test.Cid1), OperationPin, PhaseInProgress)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
ctx1 := op.Context()
ctx2 := opt.OpContext(ctx, test.Cid1)
Fix: maptracker race issues This commit attempts to fix race issues in the maptracker since the introduction of the OperationTracker. There were two main problems: * Duplicity tracking the state both in the state map and the opTracker * Non atomiciy of operations with different threads being able to affect other threads operations. A test performing random Track/Untracks on the same Cid quickly showed that items would sometimes stay as pin_queued or pin_unqueued. That happened because operations could be cancelled under the hood by a different request, while leaving the map status untouched. It was not simply to deal with this issues without a refactoring. First, the state map has been removed, and the operation tracker now provides status information for any Cid. This implies that the tracker keeps all operations and operations have a `PhaseDone`. There's also a new `OperationRemote` type. Secondly, operations are only created in the tracker and can only be removed by their creators (they can be overwritten by other operations though). Operations cannot be accessed directly and modifications are limited to setting Error for PhaseDone operations. After created, *Operations are queued in the pinWorker queues which handle any status updates. This means, that, even when an operation has been removed from the tracker, status updates will not interfere with any other newer operations. In the maptracker, only the Unpin worker Cleans operations once processed. A sucessful unpin is the only way that a delete() happens in the tracker map. Otherwise, operations stay there until a newer operation for the Cid arrives and 1) cancels the existing one 2) takes its place. The tracker refuses to create a new operation if a similar "ongoing" operation of the same type exists. The final change is that Recover and RecoverAll() are not async and play by the same rules as Track() and Untrack(), queueing the items to be recovered. Note: for stateless pintracker, the tracker will need to Clean() operation of type OperationPin as well, and complement the Status reported by the tracker with those coming from IPFS. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 16:32:10 +00:00
if ctx1 != ctx2 {
t.Fatal("didn't get the right context")
}
}
func TestOperationTracker_filterOps(t *testing.T) {
ctx := context.Background()
testOpsMap := map[api.Cid]*Operation{
test.Cid1: {pin: api.PinCid(test.Cid1), opType: OperationPin, phase: PhaseQueued},
test.Cid2: {pin: api.PinCid(test.Cid2), opType: OperationPin, phase: PhaseInProgress},
test.Cid3: {pin: api.PinCid(test.Cid3), opType: OperationUnpin, phase: PhaseInProgress},
}
opt := &OperationTracker{ctx: ctx, operations: testOpsMap}
t.Run("filter ops to pin operations", func(t *testing.T) {
wantLen := 2
wantOp := OperationPin
got := opt.filterOps(ctx, wantOp)
if len(got) != wantLen {
t.Errorf("want: %d %s operations; got: %d", wantLen, wantOp.String(), len(got))
}
for i := range got {
if got[i].Type() != wantOp {
t.Errorf("want: %v; got: %v", wantOp.String(), got[i])
}
}
})
t.Run("filter ops to in progress phase", func(t *testing.T) {
wantLen := 2
wantPhase := PhaseInProgress
got := opt.filterOps(ctx, PhaseInProgress)
if len(got) != wantLen {
t.Errorf("want: %d %s operations; got: %d", wantLen, wantPhase.String(), len(got))
}
for i := range got {
if got[i].Phase() != wantPhase {
t.Errorf("want: %s; got: %v", wantPhase.String(), got[i])
}
}
})
t.Run("filter ops to queued pins", func(t *testing.T) {
wantLen := 1
wantPhase := PhaseQueued
wantOp := OperationPin
got := opt.filterOps(ctx, OperationPin, PhaseQueued)
if len(got) != wantLen {
t.Errorf("want: %d %s operations; got: %d", wantLen, wantPhase.String(), len(got))
}
for i := range got {
if got[i].Phase() != wantPhase {
t.Errorf("want: %s; got: %v", wantPhase.String(), got[i])
}
if got[i].Type() != wantOp {
t.Errorf("want: %s; got: %v", wantOp.String(), got[i])
}
}
})
}