5fb2b6ae95
Squashed commit of the following: commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e Merge:d125f69
aaada42 Author: Hector Sanjuan <hsanjuan@users.noreply.github.com> Date: Wed Feb 20 11:02:00 2019 +0000 Merge pull request #634 from ipfs/issue_450 Support PinPath, UnpinPath (resolve before pinning) commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Feb 19 22:16:25 2019 +0530 formatResponse accepts api.Pin and not api.PinSerial commit b5da4bea045865814cc422da71827b44ddd44b90 Merge: ba59036cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Feb 19 21:36:46 2019 +0530 Merge branch 'master' into issue_450 commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f Merge: f002914 d59880c Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Mon Feb 18 08:41:11 2019 +0530 Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450 commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Mon Feb 18 08:31:39 2019 +0530 PinPath: more improvements Added tracing for new methods commit d59880c338eaa8214fe06b4f930a540793d78407 Merge: 0ca4c7cb4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com> Date: Wed Feb 13 15:22:49 2019 +0000 Merge branch 'master' into issue_450 commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030 Merge: d35017aecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Feb 12 13:10:13 2019 +0530 Merge branch 'master' into issue_450 commit d35017a8de91ca9fc9a9a047c48c75134cee9f98 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Feb 12 13:07:25 2019 +0530 PinPath: more improvements - Worth having `PinOptions` as a separate field in the struct and constructing the query in the test with ToQuery() - sharness: "intialization" line can be placed outside the tests at the top commit 68e3b90417ffbad89d41a70ac81d85f9037f8848 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Sun Feb 10 21:43:50 2019 +0530 Using if-continue pattern instead of if-else License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 3c29799f3b85be328b27508332ab92049d8b82f3 Merge: 956790b4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Thu Feb 7 10:25:52 2019 +0530 Merge branch 'master' into issue_450 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 956790b381db9858e4194f983e898b07dc51ba66 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Feb 6 21:11:20 2019 +0530 Removing resolved path License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 7191cc46cedfbec116a9746937e28881b50ca044 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Feb 6 16:45:07 2019 +0530 Fix go vet License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Feb 6 16:07:03 2019 +0530 Fixed linting error License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 23c57eb467755a1f21387a1615a7f34e97348053 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Feb 6 09:20:41 2019 +0530 Fixed tests License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4 Merge: 17e555e5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Feb 6 00:07:10 2019 +0530 Merge branch 'master' into issue_450 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 17e555e4a7c574413df90aac70c5cc29cab98f54 Author: Hector Sanjuan <code@hector.link> Date: Tue Feb 5 16:58:50 2019 +0000 PinPath: address some feedback + improvements * Changed client's Pin() API and PinPath to be consistent * Added helper methods to turn PinPath to query and back * Make code and tests build * Use TestCidResolved everywhere * Fix cluster.PinPath arguments * Fix formatting of responses with --no-status * Make tests readable and call Fatal when needed * Use a pathTestCases variable commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Feb 5 18:34:26 2019 +0530 Support PinPath, UnpinPath(resolve before pinning) Addressed review comments as in https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit a8b4f181d2d7afed32ee41331dfaab19fd66a173 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Jan 29 22:41:27 2019 +0530 Fixing tests License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit e39b95ca19e4d75506f4f492678245ef13936a44 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Jan 29 14:52:53 2019 +0530 Support PinPath, UnpinPath(resolve before pinning) - PinPath and UnpinPath should return api.Pin - PinPath should accept pin options - Removing duplicate logic for Resolve from cluster - And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit d146075126320896665ba58d337a13789f68ea86 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Jan 23 17:08:41 2019 +0530 Support PinPath, UnpinPath(resolve before pinning) PinPath(in both rest and rpc) should return a serializable struct in the form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid" to marshal and unmarshal) License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 1f4869568a8adb450275257154ea3a26d03a30f3 Merge: 7acfd28a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Jan 23 07:18:56 2019 +0530 Merge branch 'master' into issue_450 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Jan 22 18:14:32 2019 +0530 Support PinPath, UnpinPath(resolve before pinning) - RPC must always use serializable structs - In command, just use pin with path as cid is also a valid path - Addressing many other small review comments as in https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Jan 16 09:36:42 2019 +0530 Support PinPath, UnpinPath(resolve before pinning) Extra logic for path checking should go into resolve so that it can be properly reused Added sharness tests License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 9116bda3534e77bb391d873051bb520a1b01a326 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Jan 16 08:08:07 2019 +0530 Support PinPath, UnpinPath(resolve before pinning) error strings should not be capitalized Fixes #450 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit ca7e61861374f456300a85ddc0374e594f74f963 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Jan 15 23:40:25 2019 +0530 Support PinPath, UnpinPath(resolve before pinning) Tests Fixes #450 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 522fbcd899f01c01680375561a32a87464157c0a Merge: f1a56abf7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Jan 15 10:40:54 2019 +0530 Merge branch 'master' into issue_450 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit f1a56ab925fb74c0c44273a4524afa4843cf757f Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Mon Jan 14 20:58:17 2019 +0530 Support PinPath, UnpinPath(resolve before pinning) - IPFS Connector should act as a pure IPFS client, any extra logic should go to cluster.go - Use cid.Undef, instead of cid.Cid{} Fixes #450 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Thu Jan 10 08:57:17 2019 +0530 Support PinPath, UnpinPath(resolve before pinning) - Separate handlers, methods and rpc apis for PinPath and UnpinPath from Pin and Unpin - Support ipld paths as well Fixes #450 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c Merge: 91ceb47 21170c4 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Jan 9 19:38:35 2019 +0530 Merge branch 'issue_450_old' into HEAD License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 91ceb4796259ca7ef2974ec43e6a278a12796b13 Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Wed Jan 9 19:36:41 2019 +0530 Revert "WIP: Figure out why test does not impleme" This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75. License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> commit 28a3a3f25dce6f296c8cbef86221644c099a7e75 Author: cd10012 <ced361@nyu.edu> Date: Tue Jul 24 23:23:10 2018 -0400 WIP: Figure out why test does not implement IPFSConnector interface... License: MIT Signed-off-by: cd10012 <ced361@nyu.edu> commit 21170c48e77e69583db64544b08120a9baf40d8d Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Date: Tue Jan 8 10:37:59 2019 +0530 Support PinPath, UnpinPath (resolve before pinning) This commit adds API support for pinning using path `POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>` will resolve the path into a cid and perform perform pinning or unpinning Fixes #450 License: MIT Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com> Co-authored-by: Hector Sanjuan <hector@protocol.ai> License: MIT Signed-off-by: Hector Sanjuan <hector@protocol.ai>
183 lines
7.4 KiB
Go
183 lines
7.4 KiB
Go
// Package ipfscluster implements a wrapper for the IPFS deamon which
|
|
// allows to orchestrate pinning operations among several IPFS nodes.
|
|
//
|
|
// IPFS Cluster uses a go-libp2p-raft to keep a shared state between
|
|
// the different cluster peers. It also uses LibP2P to enable
|
|
// communication between its different components, which perform different
|
|
// tasks like managing the underlying IPFS daemons, or providing APIs for
|
|
// external control.
|
|
package ipfscluster
|
|
|
|
import (
|
|
"context"
|
|
|
|
"github.com/ipfs/ipfs-cluster/api"
|
|
"github.com/ipfs/ipfs-cluster/state"
|
|
|
|
cid "github.com/ipfs/go-cid"
|
|
rpc "github.com/libp2p/go-libp2p-gorpc"
|
|
peer "github.com/libp2p/go-libp2p-peer"
|
|
)
|
|
|
|
// Component represents a piece of ipfscluster. Cluster components
|
|
// usually run their own goroutines (a http server for example). They
|
|
// communicate with the main Cluster component and other components
|
|
// (both local and remote), using an instance of rpc.Client.
|
|
type Component interface {
|
|
SetClient(*rpc.Client)
|
|
Shutdown(context.Context) error
|
|
}
|
|
|
|
// Consensus is a component which keeps a shared state in
|
|
// IPFS Cluster and triggers actions on updates to that state.
|
|
// Currently, Consensus needs to be able to elect/provide a
|
|
// Cluster Leader and the implementation is very tight to
|
|
// the Cluster main component.
|
|
type Consensus interface {
|
|
Component
|
|
// Returns a channel to signal that the consensus layer is ready
|
|
// allowing the main component to wait for it during start.
|
|
Ready(context.Context) <-chan struct{}
|
|
// Logs a pin operation
|
|
LogPin(ctx context.Context, c api.Pin) error
|
|
// Logs an unpin operation
|
|
LogUnpin(ctx context.Context, c api.Pin) error
|
|
AddPeer(ctx context.Context, p peer.ID) error
|
|
RmPeer(ctx context.Context, p peer.ID) error
|
|
State(context.Context) (state.State, error)
|
|
// Provide a node which is responsible to perform
|
|
// specific tasks which must only run in 1 cluster peer
|
|
Leader(context.Context) (peer.ID, error)
|
|
// Only returns when the consensus state has all log
|
|
// updates applied to it
|
|
WaitForSync(context.Context) error
|
|
// Clean removes all consensus data
|
|
Clean(context.Context) error
|
|
// Peers returns the peerset participating in the Consensus
|
|
Peers(context.Context) ([]peer.ID, error)
|
|
}
|
|
|
|
// API is a component which offers an API for Cluster. This is
|
|
// a base component.
|
|
type API interface {
|
|
Component
|
|
}
|
|
|
|
// IPFSConnector is a component which allows cluster to interact with
|
|
// an IPFS daemon. This is a base component.
|
|
type IPFSConnector interface {
|
|
Component
|
|
ID(context.Context) (api.IPFSID, error)
|
|
Pin(context.Context, cid.Cid, int) error
|
|
Unpin(context.Context, cid.Cid) error
|
|
PinLsCid(context.Context, cid.Cid) (api.IPFSPinStatus, error)
|
|
PinLs(ctx context.Context, typeFilter string) (map[string]api.IPFSPinStatus, error)
|
|
// ConnectSwarms make sure this peer's IPFS daemon is connected to
|
|
// other peers IPFS daemons.
|
|
ConnectSwarms(context.Context) error
|
|
// SwarmPeers returns the IPFS daemon's swarm peers
|
|
SwarmPeers(context.Context) (api.SwarmPeers, error)
|
|
// ConfigKey returns the value for a configuration key.
|
|
// Subobjects are reached with keypaths as "Parent/Child/GrandChild...".
|
|
ConfigKey(keypath string) (interface{}, error)
|
|
// RepoStat returns the current repository size and max limit as
|
|
// provided by "repo stat".
|
|
RepoStat(context.Context) (api.IPFSRepoStat, error)
|
|
// Resolve returns a cid given a path
|
|
Resolve(context.Context, string) (cid.Cid, error)
|
|
// BlockPut directly adds a block of data to the IPFS repo
|
|
BlockPut(context.Context, api.NodeWithMeta) error
|
|
// BlockGet retrieves the raw data of an IPFS block
|
|
BlockGet(context.Context, cid.Cid) ([]byte, error)
|
|
}
|
|
|
|
// Peered represents a component which needs to be aware of the peers
|
|
// in the Cluster and of any changes to the peer set.
|
|
type Peered interface {
|
|
AddPeer(ctx context.Context, p peer.ID)
|
|
RmPeer(ctx context.Context, p peer.ID)
|
|
//SetPeers(peers []peer.ID)
|
|
}
|
|
|
|
// PinTracker represents a component which tracks the status of
|
|
// the pins in this cluster and ensures they are in sync with the
|
|
// IPFS daemon. This component should be thread safe.
|
|
type PinTracker interface {
|
|
Component
|
|
// Track tells the tracker that a Cid is now under its supervision
|
|
// The tracker may decide to perform an IPFS pin.
|
|
Track(context.Context, api.Pin) error
|
|
// Untrack tells the tracker that a Cid is to be forgotten. The tracker
|
|
// may perform an IPFS unpin operation.
|
|
Untrack(context.Context, cid.Cid) error
|
|
// StatusAll returns the list of pins with their local status.
|
|
StatusAll(context.Context) []api.PinInfo
|
|
// Status returns the local status of a given Cid.
|
|
Status(context.Context, cid.Cid) api.PinInfo
|
|
// SyncAll makes sure that all tracked Cids reflect the real IPFS status.
|
|
// It returns the list of pins which were updated by the call.
|
|
SyncAll(context.Context) ([]api.PinInfo, error)
|
|
// Sync makes sure that the Cid status reflect the real IPFS status.
|
|
// It returns the local status of the Cid.
|
|
Sync(context.Context, cid.Cid) (api.PinInfo, error)
|
|
// RecoverAll calls Recover() for all pins tracked.
|
|
RecoverAll(context.Context) ([]api.PinInfo, error)
|
|
// Recover retriggers a Pin/Unpin operation in a Cids with error status.
|
|
Recover(context.Context, cid.Cid) (api.PinInfo, error)
|
|
}
|
|
|
|
// Informer provides Metric information from a peer. The metrics produced by
|
|
// informers are then passed to a PinAllocator which will use them to
|
|
// determine where to pin content. The metric is agnostic to the rest of
|
|
// Cluster.
|
|
type Informer interface {
|
|
Component
|
|
Name() string
|
|
GetMetric(context.Context) api.Metric
|
|
}
|
|
|
|
// PinAllocator decides where to pin certain content. In order to make such
|
|
// decision, it receives the pin arguments, the peers which are currently
|
|
// allocated to the content and metrics available for all peers which could
|
|
// allocate the content.
|
|
type PinAllocator interface {
|
|
Component
|
|
// Allocate returns the list of peers that should be assigned to
|
|
// Pin content in order of preference (from the most preferred to the
|
|
// least). The "current" map contains valid metrics for peers
|
|
// which are currently pinning the content. The candidates map
|
|
// contains the metrics for all peers which are eligible for pinning
|
|
// the content.
|
|
Allocate(ctx context.Context, c cid.Cid, current, candidates, priority map[peer.ID]api.Metric) ([]peer.ID, error)
|
|
}
|
|
|
|
// PeerMonitor is a component in charge of publishing a peer's metrics and
|
|
// reading metrics from other peers in the cluster. The PinAllocator will
|
|
// use the metrics provided by the monitor as candidates for Pin allocations.
|
|
//
|
|
// The PeerMonitor component also provides an Alert channel which is signaled
|
|
// when a metric is no longer received and the monitor identifies it
|
|
// as a problem.
|
|
type PeerMonitor interface {
|
|
Component
|
|
// LogMetric stores a metric. It can be used to manually inject
|
|
// a metric to a monitor.
|
|
LogMetric(context.Context, api.Metric) error
|
|
// PublishMetric sends a metric to the rest of the peers.
|
|
// How to send it, and to who, is to be decided by the implementation.
|
|
PublishMetric(context.Context, api.Metric) error
|
|
// LatestMetrics returns a map with the latest metrics of matching name
|
|
// for the current cluster peers.
|
|
LatestMetrics(ctx context.Context, name string) []api.Metric
|
|
// Alerts delivers alerts generated when this peer monitor detects
|
|
// a problem (i.e. metrics not arriving as expected). Alerts can be used
|
|
// to trigger self-healing measures or re-pinnings of content.
|
|
Alerts() <-chan api.Alert
|
|
}
|
|
|
|
// Tracer implements Component as a way
|
|
// to shutdown and flush and remaining traces.
|
|
type Tracer interface {
|
|
Component
|
|
}
|