This removes mappintracker and sets stateless tracker as the default (and only) pintracker component.
Because the stateless tracker matches the cluster state with only ongoing operations being kept on memory, and additional information provided by ipfs-pin-ls, syncing operations are not necessary. Therefore the Sync/SyncAll operations are removed cluster-wide.
This introduces the possiblity of running Cluster with multiple informer components. The first one in the list is the used for "allocations". This is just groundwork for working with several informers in the future.
- Cluster peers will now be able dial and listen using QUIC
- By default QUIC is enabled, to disable it remove QUIC listen address
from service.json
- This commit also adds a config option for whether to act as relay or
not, EnableRelayHop
This adds a new PinOption: ExpireAt.
The StateSync ticker will check and unpin expired pins from the Cluster.
ipfs-cluster-ctl supports an "expire-in" which gives a duration.
- cluster method, ipfs connector method, rpc and rest apis,
command, etc for repo gc
- Remove extra space from policy generator
- Added special timeout for `/repo/gc` call to IPFS
- Added `RepoGCLocal` cluster rpc method, which will be used to run gc
on local IPFS daemon
- Added peer name to the repo gc struct
- Sorted with peer ids, while formatting(only affects cli
results)
- Special timeout setting where timeout gets checked from last update
- Added `local` argument, which would run gc only on contacted peer
* pin() should not allocate if allocations are already provided
* pin() should not skip pinning if the exact same pin exists
* Additionally this was unreliable as it allocated it before
so the pin may have existed but the allocations may have been
artificially changed.
* pin() re-uses existing pin when pin options are the same and thus
avoids changing the allocations of a pin.
As a side effect, this fixes re-allocations which were broken: peers
called `shouldPeerRepinCid()` and instead of repinning that single
cid proceeded to repin the full state. For every pin.
Additionally tests have been adapted. It may be that some re-alloc tests
were very unreliable for the problems above.
This introduces a pin/update operation which allows to Pin a new item to
cluster indicating that said pin is an update to an already-existing pin.
When this is the case, all the configuration for the existing pin is copied to
the new one (including allocations). The IPFS connector will then trigger
pin/update directly in IPFS, allowing an efficient pinning based on
DAG-differences. Since the allocations where the same for both pins,
the pin/update can proceed.
PinUpdate does not unpin the previous pin (it is not possible to do this
atomically in cluster like it happens in IPFS). The user can manually do it
after the pin/update is done.
Internally, after a lot of deliberations on what the optimal way for this is,
I opted for adding a `PinUpdate` option to the `PinOptions` type (carries the
CID to update from). In order to carry this option from the REST API to the
IPFS Connector, it is serialized in the Protobuf (and stored in the
datastore). There is no other way to do this in a simple fashion since the Pin
object is piece of information that is sent around.
Additionally, making it a PinOption plays well with the Pin/PinPath APIs which
need little changes. Effectively, you are pinning a new thing. You are just
indicating that it should be configured from an existing one.
Fixes#732
Peers configured with follower_mode = true fail to add/pin/unpin.
Additionally they do not contact other peers when doing Status, Sync or
Recover and report on themselves.
They still contact other peers when doing "peers ls", as this is an OpenRPC
endpoint.
This is merely improving user interaction with a cluster peer and avoids
getting into confusing places:
* pin/unpin seems to work even no one trusts them
* status will query all peers in the peerset only to get auth errors and
ignore them, becoming way slower than it could be
This is not a security feature.
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
Specifying "*" as part of "trusted_peers" in the configuration will
result in trusting all peers.
This is useful for private clusters where we don't want to list every
peer ID in the config.
This fixes multiple issues in and around tests while
increasing ttls and delays in 100ms. Multiple issues, including
races, tests not running with consensus-crdt missing log messages
and better initialization have been fixed.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This adds a new "crdt" consensus component using go-ds-crdt.
This implies several refactors to fully make cluster consensus-component
independent:
* Delete mapstate and fully adopt dsstate (after people have migrated).
* Return errors from state methods rather than ignoring them.
* Add a new "datastore" modules so that we can configure datastores in the
main configuration like other components.
* Let the consensus components fully define the "state.State". Thus, they do
not receive the state, they receive the storage where we put the state (a
go-datastore).
* Allow to customize how the monitor component obtains Peers() (the current
peerset), including avoiding using the current peerset. At the moment the
crdt consensus uses the monitoring component to define the current peerset.
Therefore the monitor component cannot rely on the consensus component to
produce a peerset.
* Re-factor/re-implementation of "ipfs-cluster-service state"
operations. Includes the dissapearance of the "migrate" one.
The CRDT consensus component defines creates a crdt-datastore (with ipfs-lite)
and uses it to intitialize a dssate. Thus the crdt-store is elegantly
wrapped. Any modifications to the state get automatically replicated to other
peers. We store all the CRDT DAG blocks in the local datastore.
The consensus components only expose a ReadOnly state, as any modifications to
the shared state should happen through them.
DHT and PubSub facilities must now be created outside of Cluster and passed in
so they can be re-used by different components.
Remove basic monitor
This commit removes `basic` monitor component, because it is not being
used by default since few releases ago pubsub monitor was introduced.
Issue #689