2016-12-28 15:25:24 +00:00
|
|
|
// Package ipfscluster implements a wrapper for the IPFS deamon which
|
|
|
|
// allows to orchestrate pinning operations among several IPFS nodes.
|
2016-12-02 18:33:39 +00:00
|
|
|
//
|
2019-05-04 20:36:10 +00:00
|
|
|
// IPFS Cluster peers form a separate libp2p swarm. A Cluster peer uses
|
2020-02-03 09:30:04 +00:00
|
|
|
// multiple Cluster Components which perform different tasks like managing
|
2019-05-04 20:36:10 +00:00
|
|
|
// the underlying IPFS daemons, or providing APIs for external control.
|
2016-12-02 18:33:39 +00:00
|
|
|
package ipfscluster
|
|
|
|
|
|
|
|
import (
|
2018-04-16 09:01:20 +00:00
|
|
|
"context"
|
|
|
|
|
2017-03-14 15:37:29 +00:00
|
|
|
"github.com/ipfs/ipfs-cluster/api"
|
|
|
|
"github.com/ipfs/ipfs-cluster/state"
|
|
|
|
|
2019-06-14 10:41:11 +00:00
|
|
|
peer "github.com/libp2p/go-libp2p-core/peer"
|
2018-10-17 13:28:03 +00:00
|
|
|
rpc "github.com/libp2p/go-libp2p-gorpc"
|
2016-12-02 18:33:39 +00:00
|
|
|
)
|
|
|
|
|
2016-12-29 17:38:09 +00:00
|
|
|
// Component represents a piece of ipfscluster. Cluster components
|
2016-12-08 16:24:38 +00:00
|
|
|
// usually run their own goroutines (a http server for example). They
|
2016-12-23 18:35:37 +00:00
|
|
|
// communicate with the main Cluster component and other components
|
|
|
|
// (both local and remote), using an instance of rpc.Client.
|
2016-12-29 17:38:09 +00:00
|
|
|
type Component interface {
|
2016-12-23 18:35:37 +00:00
|
|
|
SetClient(*rpc.Client)
|
2018-06-27 04:03:15 +00:00
|
|
|
Shutdown(context.Context) error
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
2017-03-14 15:37:29 +00:00
|
|
|
// Consensus is a component which keeps a shared state in
|
|
|
|
// IPFS Cluster and triggers actions on updates to that state.
|
|
|
|
// Currently, Consensus needs to be able to elect/provide a
|
|
|
|
// Cluster Leader and the implementation is very tight to
|
|
|
|
// the Cluster main component.
|
|
|
|
type Consensus interface {
|
|
|
|
Component
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
// Returns a channel to signal that the consensus layer is ready
|
|
|
|
// allowing the main component to wait for it during start.
|
2018-06-27 04:03:15 +00:00
|
|
|
Ready(context.Context) <-chan struct{}
|
2019-05-13 21:20:03 +00:00
|
|
|
// Logs a pin operation.
|
2022-03-19 01:52:46 +00:00
|
|
|
LogPin(context.Context, api.Pin) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// Logs an unpin operation.
|
2022-03-19 01:52:46 +00:00
|
|
|
LogUnpin(context.Context, api.Pin) error
|
2019-07-12 14:40:29 +00:00
|
|
|
AddPeer(context.Context, peer.ID) error
|
|
|
|
RmPeer(context.Context, peer.ID) error
|
2019-02-20 14:24:25 +00:00
|
|
|
State(context.Context) (state.ReadOnly, error)
|
2017-03-14 15:37:29 +00:00
|
|
|
// Provide a node which is responsible to perform
|
2019-05-13 21:20:03 +00:00
|
|
|
// specific tasks which must only run in 1 cluster peer.
|
2018-06-27 04:03:15 +00:00
|
|
|
Leader(context.Context) (peer.ID, error)
|
2017-03-14 15:37:29 +00:00
|
|
|
// Only returns when the consensus state has all log
|
2019-05-13 21:20:03 +00:00
|
|
|
// updates applied to it.
|
2018-06-27 04:03:15 +00:00
|
|
|
WaitForSync(context.Context) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// Clean removes all consensus data.
|
2018-06-27 04:03:15 +00:00
|
|
|
Clean(context.Context) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// Peers returns the peerset participating in the Consensus.
|
2018-06-27 04:03:15 +00:00
|
|
|
Peers(context.Context) ([]peer.ID, error)
|
2019-05-13 21:20:03 +00:00
|
|
|
// IsTrustedPeer returns true if the given peer is "trusted".
|
2019-05-09 13:14:26 +00:00
|
|
|
// This will grant access to more rpc endpoints and a
|
|
|
|
// non-trusted one. This should be fast as it will be
|
2020-02-03 09:30:04 +00:00
|
|
|
// called repeatedly for every remote RPC request.
|
2019-05-09 13:14:26 +00:00
|
|
|
IsTrustedPeer(context.Context, peer.ID) bool
|
2019-05-13 21:20:03 +00:00
|
|
|
// Trust marks a peer as "trusted".
|
2019-05-09 17:48:40 +00:00
|
|
|
Trust(context.Context, peer.ID) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// Distrust removes a peer from the "trusted" set.
|
2019-05-09 17:48:40 +00:00
|
|
|
Distrust(context.Context, peer.ID) error
|
2017-03-14 15:37:29 +00:00
|
|
|
}
|
|
|
|
|
2016-12-15 18:08:46 +00:00
|
|
|
// API is a component which offers an API for Cluster. This is
|
2016-12-02 18:33:39 +00:00
|
|
|
// a base component.
|
2016-12-15 18:08:46 +00:00
|
|
|
type API interface {
|
2016-12-29 17:38:09 +00:00
|
|
|
Component
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// IPFSConnector is a component which allows cluster to interact with
|
2016-12-08 16:24:38 +00:00
|
|
|
// an IPFS daemon. This is a base component.
|
2016-12-02 18:33:39 +00:00
|
|
|
type IPFSConnector interface {
|
2016-12-29 17:38:09 +00:00
|
|
|
Component
|
2022-03-19 01:52:46 +00:00
|
|
|
ID(context.Context) (api.IPFSID, error)
|
|
|
|
Pin(context.Context, api.Pin) error
|
2022-04-07 11:53:30 +00:00
|
|
|
Unpin(context.Context, api.Cid) error
|
2022-03-19 01:52:46 +00:00
|
|
|
PinLsCid(context.Context, api.Pin) (api.IPFSPinStatus, error)
|
2022-03-22 09:56:16 +00:00
|
|
|
// PinLs returns pins in the pinset of the given types (recursive, direct...)
|
|
|
|
PinLs(ctx context.Context, typeFilters []string, out chan<- api.IPFSPinInfo) error
|
2017-03-23 18:34:33 +00:00
|
|
|
// ConnectSwarms make sure this peer's IPFS daemon is connected to
|
|
|
|
// other peers IPFS daemons.
|
2018-06-27 04:03:15 +00:00
|
|
|
ConnectSwarms(context.Context) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// SwarmPeers returns the IPFS daemon's swarm peers.
|
2019-02-27 17:04:35 +00:00
|
|
|
SwarmPeers(context.Context) ([]peer.ID, error)
|
2017-03-27 13:07:12 +00:00
|
|
|
// ConfigKey returns the value for a configuration key.
|
|
|
|
// Subobjects are reached with keypaths as "Parent/Child/GrandChild...".
|
|
|
|
ConfigKey(keypath string) (interface{}, error)
|
2018-08-20 18:43:27 +00:00
|
|
|
// RepoStat returns the current repository size and max limit as
|
|
|
|
// provided by "repo stat".
|
2022-03-19 01:52:46 +00:00
|
|
|
RepoStat(context.Context) (api.IPFSRepoStat, error)
|
2019-10-22 05:40:32 +00:00
|
|
|
// RepoGC performs garbage collection sweep on the IPFS repo.
|
2022-03-19 01:52:46 +00:00
|
|
|
RepoGC(context.Context) (api.RepoGC, error)
|
2019-05-13 21:20:03 +00:00
|
|
|
// Resolve returns a cid given a path.
|
2022-04-07 11:53:30 +00:00
|
|
|
Resolve(context.Context, string) (api.Cid, error)
|
Adders: stream blocks to destinations
This commit fixes #810 and adds block streaming to the final destinations when
adding. This should add major performance gains when adding data to clusters.
Before, everytime cluster issued a block, it was broadcasted individually to
all destinations (new libp2p stream), where it was block/put to IPFS (a single
block/put http roundtrip per block).
Now, blocks are streamed all the way from the adder module to the ipfs daemon,
by making every block as it arrives a single part in a multipart block/put
request.
Before, block-broadcast needed to wait for all destinations to finish in order
to process the next block. Now, buffers allow some destinations to be faster
than others while sending and receiving blocks.
Before, if a block put request failed to be broadcasted everywhere, an error
would happen at that moment.
Now, we keep streaming until the end and only then report any errors. The
operation succeeds as long as at least one stream finished successfully.
Errors block/putting to IPFS will not abort streams. Instead, subsequent
blocks are retried with a new request, although the method will return an
error when the stream finishes if there were errors at any point.
2022-03-24 01:17:10 +00:00
|
|
|
// BlockStream adds a stream of blocks to IPFS.
|
|
|
|
BlockStream(context.Context, <-chan api.NodeWithMeta) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// BlockGet retrieves the raw data of an IPFS block.
|
2022-04-07 11:53:30 +00:00
|
|
|
BlockGet(context.Context, api.Cid) ([]byte, error)
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Peered represents a component which needs to be aware of the peers
|
2016-12-08 16:24:38 +00:00
|
|
|
// in the Cluster and of any changes to the peer set.
|
2016-12-02 18:33:39 +00:00
|
|
|
type Peered interface {
|
2018-06-27 04:03:15 +00:00
|
|
|
AddPeer(ctx context.Context, p peer.ID)
|
|
|
|
RmPeer(ctx context.Context, p peer.ID)
|
2017-01-30 12:12:25 +00:00
|
|
|
//SetPeers(peers []peer.ID)
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
2016-12-06 21:29:59 +00:00
|
|
|
// PinTracker represents a component which tracks the status of
|
|
|
|
// the pins in this cluster and ensures they are in sync with the
|
|
|
|
// IPFS daemon. This component should be thread safe.
|
|
|
|
type PinTracker interface {
|
2016-12-29 17:38:09 +00:00
|
|
|
Component
|
2016-12-19 17:35:24 +00:00
|
|
|
// Track tells the tracker that a Cid is now under its supervision
|
|
|
|
// The tracker may decide to perform an IPFS pin.
|
2022-03-19 01:52:46 +00:00
|
|
|
Track(context.Context, api.Pin) error
|
2016-12-19 17:35:24 +00:00
|
|
|
// Untrack tells the tracker that a Cid is to be forgotten. The tracker
|
|
|
|
// may perform an IPFS unpin operation.
|
2022-04-07 11:53:30 +00:00
|
|
|
Untrack(context.Context, api.Cid) error
|
2021-07-04 21:40:15 +00:00
|
|
|
// StatusAll returns the list of pins with their local status. Takes a
|
|
|
|
// filter to specify which statuses to report.
|
2022-03-22 09:56:16 +00:00
|
|
|
StatusAll(context.Context, api.TrackerStatus, chan<- api.PinInfo) error
|
2017-01-25 18:38:23 +00:00
|
|
|
// Status returns the local status of a given Cid.
|
2022-04-07 11:53:30 +00:00
|
|
|
Status(context.Context, api.Cid) api.PinInfo
|
2022-03-22 09:56:16 +00:00
|
|
|
// RecoverAll calls Recover() for all pins tracked.
|
|
|
|
RecoverAll(context.Context, chan<- api.PinInfo) error
|
2018-04-09 02:19:47 +00:00
|
|
|
// Recover retriggers a Pin/Unpin operation in a Cids with error status.
|
2022-04-07 11:53:30 +00:00
|
|
|
Recover(context.Context, api.Cid) (api.PinInfo, error)
|
2017-01-30 12:12:25 +00:00
|
|
|
}
|
2017-02-13 15:46:53 +00:00
|
|
|
|
2017-02-15 14:16:16 +00:00
|
|
|
// Informer provides Metric information from a peer. The metrics produced by
|
2017-02-13 15:46:53 +00:00
|
|
|
// informers are then passed to a PinAllocator which will use them to
|
2017-02-15 14:16:16 +00:00
|
|
|
// determine where to pin content. The metric is agnostic to the rest of
|
|
|
|
// Cluster.
|
2017-02-13 15:46:53 +00:00
|
|
|
type Informer interface {
|
|
|
|
Component
|
|
|
|
Name() string
|
2021-10-05 12:04:28 +00:00
|
|
|
// GetMetrics returns the metrics obtained by this Informer. It must
|
|
|
|
// always return at least one metric.
|
2022-03-19 01:52:46 +00:00
|
|
|
GetMetrics(context.Context) []api.Metric
|
2017-02-13 15:46:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// PinAllocator decides where to pin certain content. In order to make such
|
|
|
|
// decision, it receives the pin arguments, the peers which are currently
|
|
|
|
// allocated to the content and metrics available for all peers which could
|
|
|
|
// allocate the content.
|
|
|
|
type PinAllocator interface {
|
|
|
|
Component
|
|
|
|
// Allocate returns the list of peers that should be assigned to
|
2018-04-09 02:19:47 +00:00
|
|
|
// Pin content in order of preference (from the most preferred to the
|
2017-02-15 14:16:16 +00:00
|
|
|
// least). The "current" map contains valid metrics for peers
|
2017-02-13 15:46:53 +00:00
|
|
|
// which are currently pinning the content. The candidates map
|
2017-02-15 14:16:16 +00:00
|
|
|
// contains the metrics for all peers which are eligible for pinning
|
2017-02-13 15:46:53 +00:00
|
|
|
// the content.
|
2022-04-07 11:53:30 +00:00
|
|
|
Allocate(ctx context.Context, c api.Cid, current, candidates, priority api.MetricsSet) ([]peer.ID, error)
|
2021-09-13 12:14:50 +00:00
|
|
|
// Metrics returns the list of metrics that the allocator needs.
|
|
|
|
Metrics() []string
|
2017-02-13 15:46:53 +00:00
|
|
|
}
|
|
|
|
|
2018-05-08 11:10:38 +00:00
|
|
|
// PeerMonitor is a component in charge of publishing a peer's metrics and
|
2018-05-01 16:01:59 +00:00
|
|
|
// reading metrics from other peers in the cluster. The PinAllocator will
|
|
|
|
// use the metrics provided by the monitor as candidates for Pin allocations.
|
|
|
|
//
|
|
|
|
// The PeerMonitor component also provides an Alert channel which is signaled
|
2018-05-08 11:10:38 +00:00
|
|
|
// when a metric is no longer received and the monitor identifies it
|
2018-05-01 16:01:59 +00:00
|
|
|
// as a problem.
|
2017-02-13 15:46:53 +00:00
|
|
|
type PeerMonitor interface {
|
|
|
|
Component
|
2018-05-01 16:01:59 +00:00
|
|
|
// LogMetric stores a metric. It can be used to manually inject
|
|
|
|
// a metric to a monitor.
|
2022-03-19 01:52:46 +00:00
|
|
|
LogMetric(context.Context, api.Metric) error
|
2018-05-01 16:01:59 +00:00
|
|
|
// PublishMetric sends a metric to the rest of the peers.
|
|
|
|
// How to send it, and to who, is to be decided by the implementation.
|
2022-03-19 01:52:46 +00:00
|
|
|
PublishMetric(context.Context, api.Metric) error
|
2022-01-28 15:38:29 +00:00
|
|
|
// LatestMetrics returns a map with the latest valid metrics of matching
|
2021-10-05 12:04:28 +00:00
|
|
|
// name for the current cluster peers. The result should only contain
|
|
|
|
// one metric per peer at most.
|
2022-03-19 01:52:46 +00:00
|
|
|
LatestMetrics(ctx context.Context, name string) []api.Metric
|
2022-01-28 15:38:29 +00:00
|
|
|
// Returns the latest metric received from a peer. It may be expired.
|
2022-03-19 01:52:46 +00:00
|
|
|
LatestForPeer(ctx context.Context, name string, pid peer.ID) api.Metric
|
2019-10-24 14:37:26 +00:00
|
|
|
// MetricNames returns a list of metric names.
|
|
|
|
MetricNames(ctx context.Context) []string
|
2017-02-13 15:46:53 +00:00
|
|
|
// Alerts delivers alerts generated when this peer monitor detects
|
2018-05-01 16:01:59 +00:00
|
|
|
// a problem (i.e. metrics not arriving as expected). Alerts can be used
|
|
|
|
// to trigger self-healing measures or re-pinnings of content.
|
2022-03-19 01:52:46 +00:00
|
|
|
Alerts() <-chan api.Alert
|
2017-02-13 15:46:53 +00:00
|
|
|
}
|
2018-06-27 04:03:15 +00:00
|
|
|
|
|
|
|
// Tracer implements Component as a way
|
|
|
|
// to shutdown and flush and remaining traces.
|
|
|
|
type Tracer interface {
|
|
|
|
Component
|
|
|
|
}
|