2016-12-28 15:25:24 +00:00
|
|
|
// Package ipfscluster implements a wrapper for the IPFS deamon which
|
|
|
|
// allows to orchestrate pinning operations among several IPFS nodes.
|
2016-12-02 18:33:39 +00:00
|
|
|
//
|
2019-05-04 20:36:10 +00:00
|
|
|
// IPFS Cluster peers form a separate libp2p swarm. A Cluster peer uses
|
|
|
|
// multiple Cluster Componenets which perform different tasks like managing
|
|
|
|
// the underlying IPFS daemons, or providing APIs for external control.
|
2016-12-02 18:33:39 +00:00
|
|
|
package ipfscluster
|
|
|
|
|
|
|
|
import (
|
2018-04-16 09:01:20 +00:00
|
|
|
"context"
|
|
|
|
|
2017-03-14 15:37:29 +00:00
|
|
|
"github.com/ipfs/ipfs-cluster/api"
|
|
|
|
"github.com/ipfs/ipfs-cluster/state"
|
|
|
|
|
2016-12-16 11:40:28 +00:00
|
|
|
cid "github.com/ipfs/go-cid"
|
2019-06-14 10:41:11 +00:00
|
|
|
peer "github.com/libp2p/go-libp2p-core/peer"
|
2018-10-17 13:28:03 +00:00
|
|
|
rpc "github.com/libp2p/go-libp2p-gorpc"
|
2016-12-02 18:33:39 +00:00
|
|
|
)
|
|
|
|
|
2016-12-29 17:38:09 +00:00
|
|
|
// Component represents a piece of ipfscluster. Cluster components
|
2016-12-08 16:24:38 +00:00
|
|
|
// usually run their own goroutines (a http server for example). They
|
2016-12-23 18:35:37 +00:00
|
|
|
// communicate with the main Cluster component and other components
|
|
|
|
// (both local and remote), using an instance of rpc.Client.
|
2016-12-29 17:38:09 +00:00
|
|
|
type Component interface {
|
2016-12-23 18:35:37 +00:00
|
|
|
SetClient(*rpc.Client)
|
2018-06-27 04:03:15 +00:00
|
|
|
Shutdown(context.Context) error
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
2017-03-14 15:37:29 +00:00
|
|
|
// Consensus is a component which keeps a shared state in
|
|
|
|
// IPFS Cluster and triggers actions on updates to that state.
|
|
|
|
// Currently, Consensus needs to be able to elect/provide a
|
|
|
|
// Cluster Leader and the implementation is very tight to
|
|
|
|
// the Cluster main component.
|
|
|
|
type Consensus interface {
|
|
|
|
Component
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
// Returns a channel to signal that the consensus layer is ready
|
|
|
|
// allowing the main component to wait for it during start.
|
2018-06-27 04:03:15 +00:00
|
|
|
Ready(context.Context) <-chan struct{}
|
2019-05-13 21:20:03 +00:00
|
|
|
// Logs a pin operation.
|
2019-07-12 14:40:29 +00:00
|
|
|
LogPin(context.Context, *api.Pin) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// Logs an unpin operation.
|
2019-07-12 14:40:29 +00:00
|
|
|
LogUnpin(context.Context, *api.Pin) error
|
|
|
|
AddPeer(context.Context, peer.ID) error
|
|
|
|
RmPeer(context.Context, peer.ID) error
|
2019-02-20 14:24:25 +00:00
|
|
|
State(context.Context) (state.ReadOnly, error)
|
2017-03-14 15:37:29 +00:00
|
|
|
// Provide a node which is responsible to perform
|
2019-05-13 21:20:03 +00:00
|
|
|
// specific tasks which must only run in 1 cluster peer.
|
2018-06-27 04:03:15 +00:00
|
|
|
Leader(context.Context) (peer.ID, error)
|
2017-03-14 15:37:29 +00:00
|
|
|
// Only returns when the consensus state has all log
|
2019-05-13 21:20:03 +00:00
|
|
|
// updates applied to it.
|
2018-06-27 04:03:15 +00:00
|
|
|
WaitForSync(context.Context) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// Clean removes all consensus data.
|
2018-06-27 04:03:15 +00:00
|
|
|
Clean(context.Context) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// Peers returns the peerset participating in the Consensus.
|
2018-06-27 04:03:15 +00:00
|
|
|
Peers(context.Context) ([]peer.ID, error)
|
2019-05-13 21:20:03 +00:00
|
|
|
// IsTrustedPeer returns true if the given peer is "trusted".
|
2019-05-09 13:14:26 +00:00
|
|
|
// This will grant access to more rpc endpoints and a
|
|
|
|
// non-trusted one. This should be fast as it will be
|
|
|
|
// called repeteadly for every remote RPC request.
|
|
|
|
IsTrustedPeer(context.Context, peer.ID) bool
|
2019-05-13 21:20:03 +00:00
|
|
|
// Trust marks a peer as "trusted".
|
2019-05-09 17:48:40 +00:00
|
|
|
Trust(context.Context, peer.ID) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// Distrust removes a peer from the "trusted" set.
|
2019-05-09 17:48:40 +00:00
|
|
|
Distrust(context.Context, peer.ID) error
|
2017-03-14 15:37:29 +00:00
|
|
|
}
|
|
|
|
|
2016-12-15 18:08:46 +00:00
|
|
|
// API is a component which offers an API for Cluster. This is
|
2016-12-02 18:33:39 +00:00
|
|
|
// a base component.
|
2016-12-15 18:08:46 +00:00
|
|
|
type API interface {
|
2016-12-29 17:38:09 +00:00
|
|
|
Component
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// IPFSConnector is a component which allows cluster to interact with
|
2016-12-08 16:24:38 +00:00
|
|
|
// an IPFS daemon. This is a base component.
|
2016-12-02 18:33:39 +00:00
|
|
|
type IPFSConnector interface {
|
2016-12-29 17:38:09 +00:00
|
|
|
Component
|
2019-02-27 17:04:35 +00:00
|
|
|
ID(context.Context) (*api.IPFSID, error)
|
2019-07-12 14:40:29 +00:00
|
|
|
Pin(context.Context, *api.Pin) error
|
2018-09-22 01:00:10 +00:00
|
|
|
Unpin(context.Context, cid.Cid) error
|
|
|
|
PinLsCid(context.Context, cid.Cid) (api.IPFSPinStatus, error)
|
2018-04-16 09:01:20 +00:00
|
|
|
PinLs(ctx context.Context, typeFilter string) (map[string]api.IPFSPinStatus, error)
|
2017-03-23 18:34:33 +00:00
|
|
|
// ConnectSwarms make sure this peer's IPFS daemon is connected to
|
|
|
|
// other peers IPFS daemons.
|
2018-06-27 04:03:15 +00:00
|
|
|
ConnectSwarms(context.Context) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// SwarmPeers returns the IPFS daemon's swarm peers.
|
2019-02-27 17:04:35 +00:00
|
|
|
SwarmPeers(context.Context) ([]peer.ID, error)
|
2017-03-27 13:07:12 +00:00
|
|
|
// ConfigKey returns the value for a configuration key.
|
|
|
|
// Subobjects are reached with keypaths as "Parent/Child/GrandChild...".
|
|
|
|
ConfigKey(keypath string) (interface{}, error)
|
2018-08-20 18:43:27 +00:00
|
|
|
// RepoStat returns the current repository size and max limit as
|
|
|
|
// provided by "repo stat".
|
2019-02-27 17:04:35 +00:00
|
|
|
RepoStat(context.Context) (*api.IPFSRepoStat, error)
|
2019-05-13 21:20:03 +00:00
|
|
|
// Resolve returns a cid given a path.
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
Resolve(context.Context, string) (cid.Cid, error)
|
2019-05-13 21:20:03 +00:00
|
|
|
// BlockPut directly adds a block of data to the IPFS repo.
|
2019-02-27 17:04:35 +00:00
|
|
|
BlockPut(context.Context, *api.NodeWithMeta) error
|
2019-05-13 21:20:03 +00:00
|
|
|
// BlockGet retrieves the raw data of an IPFS block.
|
2018-06-27 04:03:15 +00:00
|
|
|
BlockGet(context.Context, cid.Cid) ([]byte, error)
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Peered represents a component which needs to be aware of the peers
|
2016-12-08 16:24:38 +00:00
|
|
|
// in the Cluster and of any changes to the peer set.
|
2016-12-02 18:33:39 +00:00
|
|
|
type Peered interface {
|
2018-06-27 04:03:15 +00:00
|
|
|
AddPeer(ctx context.Context, p peer.ID)
|
|
|
|
RmPeer(ctx context.Context, p peer.ID)
|
2017-01-30 12:12:25 +00:00
|
|
|
//SetPeers(peers []peer.ID)
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
2016-12-06 21:29:59 +00:00
|
|
|
// PinTracker represents a component which tracks the status of
|
|
|
|
// the pins in this cluster and ensures they are in sync with the
|
|
|
|
// IPFS daemon. This component should be thread safe.
|
|
|
|
type PinTracker interface {
|
2016-12-29 17:38:09 +00:00
|
|
|
Component
|
2016-12-19 17:35:24 +00:00
|
|
|
// Track tells the tracker that a Cid is now under its supervision
|
|
|
|
// The tracker may decide to perform an IPFS pin.
|
2019-02-27 17:04:35 +00:00
|
|
|
Track(context.Context, *api.Pin) error
|
2016-12-19 17:35:24 +00:00
|
|
|
// Untrack tells the tracker that a Cid is to be forgotten. The tracker
|
|
|
|
// may perform an IPFS unpin operation.
|
2018-06-27 04:03:15 +00:00
|
|
|
Untrack(context.Context, cid.Cid) error
|
2017-01-25 18:38:23 +00:00
|
|
|
// StatusAll returns the list of pins with their local status.
|
2019-02-27 17:04:35 +00:00
|
|
|
StatusAll(context.Context) []*api.PinInfo
|
2017-01-25 18:38:23 +00:00
|
|
|
// Status returns the local status of a given Cid.
|
2019-02-27 17:04:35 +00:00
|
|
|
Status(context.Context, cid.Cid) *api.PinInfo
|
2017-01-25 18:38:23 +00:00
|
|
|
// SyncAll makes sure that all tracked Cids reflect the real IPFS status.
|
2017-01-25 17:07:19 +00:00
|
|
|
// It returns the list of pins which were updated by the call.
|
2019-02-27 17:04:35 +00:00
|
|
|
SyncAll(context.Context) ([]*api.PinInfo, error)
|
2017-01-25 18:38:23 +00:00
|
|
|
// Sync makes sure that the Cid status reflect the real IPFS status.
|
|
|
|
// It returns the local status of the Cid.
|
2019-02-27 17:04:35 +00:00
|
|
|
Sync(context.Context, cid.Cid) (*api.PinInfo, error)
|
2017-11-30 00:53:31 +00:00
|
|
|
// RecoverAll calls Recover() for all pins tracked.
|
2019-02-27 17:04:35 +00:00
|
|
|
RecoverAll(context.Context) ([]*api.PinInfo, error)
|
2018-04-09 02:19:47 +00:00
|
|
|
// Recover retriggers a Pin/Unpin operation in a Cids with error status.
|
2019-02-27 17:04:35 +00:00
|
|
|
Recover(context.Context, cid.Cid) (*api.PinInfo, error)
|
2017-01-30 12:12:25 +00:00
|
|
|
}
|
2017-02-13 15:46:53 +00:00
|
|
|
|
2017-02-15 14:16:16 +00:00
|
|
|
// Informer provides Metric information from a peer. The metrics produced by
|
2017-02-13 15:46:53 +00:00
|
|
|
// informers are then passed to a PinAllocator which will use them to
|
2017-02-15 14:16:16 +00:00
|
|
|
// determine where to pin content. The metric is agnostic to the rest of
|
|
|
|
// Cluster.
|
2017-02-13 15:46:53 +00:00
|
|
|
type Informer interface {
|
|
|
|
Component
|
|
|
|
Name() string
|
2019-02-27 17:04:35 +00:00
|
|
|
GetMetric(context.Context) *api.Metric
|
2017-02-13 15:46:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// PinAllocator decides where to pin certain content. In order to make such
|
|
|
|
// decision, it receives the pin arguments, the peers which are currently
|
|
|
|
// allocated to the content and metrics available for all peers which could
|
|
|
|
// allocate the content.
|
|
|
|
type PinAllocator interface {
|
|
|
|
Component
|
|
|
|
// Allocate returns the list of peers that should be assigned to
|
2018-04-09 02:19:47 +00:00
|
|
|
// Pin content in order of preference (from the most preferred to the
|
2017-02-15 14:16:16 +00:00
|
|
|
// least). The "current" map contains valid metrics for peers
|
2017-02-13 15:46:53 +00:00
|
|
|
// which are currently pinning the content. The candidates map
|
2017-02-15 14:16:16 +00:00
|
|
|
// contains the metrics for all peers which are eligible for pinning
|
2017-02-13 15:46:53 +00:00
|
|
|
// the content.
|
2019-02-27 17:04:35 +00:00
|
|
|
Allocate(ctx context.Context, c cid.Cid, current, candidates, priority map[peer.ID]*api.Metric) ([]peer.ID, error)
|
2017-02-13 15:46:53 +00:00
|
|
|
}
|
|
|
|
|
2018-05-08 11:10:38 +00:00
|
|
|
// PeerMonitor is a component in charge of publishing a peer's metrics and
|
2018-05-01 16:01:59 +00:00
|
|
|
// reading metrics from other peers in the cluster. The PinAllocator will
|
|
|
|
// use the metrics provided by the monitor as candidates for Pin allocations.
|
|
|
|
//
|
|
|
|
// The PeerMonitor component also provides an Alert channel which is signaled
|
2018-05-08 11:10:38 +00:00
|
|
|
// when a metric is no longer received and the monitor identifies it
|
2018-05-01 16:01:59 +00:00
|
|
|
// as a problem.
|
2017-02-13 15:46:53 +00:00
|
|
|
type PeerMonitor interface {
|
|
|
|
Component
|
2018-05-01 16:01:59 +00:00
|
|
|
// LogMetric stores a metric. It can be used to manually inject
|
|
|
|
// a metric to a monitor.
|
2019-02-27 17:04:35 +00:00
|
|
|
LogMetric(context.Context, *api.Metric) error
|
2018-05-01 16:01:59 +00:00
|
|
|
// PublishMetric sends a metric to the rest of the peers.
|
|
|
|
// How to send it, and to who, is to be decided by the implementation.
|
2019-02-27 17:04:35 +00:00
|
|
|
PublishMetric(context.Context, *api.Metric) error
|
2018-05-08 10:44:14 +00:00
|
|
|
// LatestMetrics returns a map with the latest metrics of matching name
|
2017-02-13 15:46:53 +00:00
|
|
|
// for the current cluster peers.
|
2019-02-27 17:04:35 +00:00
|
|
|
LatestMetrics(ctx context.Context, name string) []*api.Metric
|
2017-02-13 15:46:53 +00:00
|
|
|
// Alerts delivers alerts generated when this peer monitor detects
|
2018-05-01 16:01:59 +00:00
|
|
|
// a problem (i.e. metrics not arriving as expected). Alerts can be used
|
|
|
|
// to trigger self-healing measures or re-pinnings of content.
|
2019-02-27 17:04:35 +00:00
|
|
|
Alerts() <-chan *api.Alert
|
2017-02-13 15:46:53 +00:00
|
|
|
}
|
2018-06-27 04:03:15 +00:00
|
|
|
|
|
|
|
// Tracer implements Component as a way
|
|
|
|
// to shutdown and flush and remaining traces.
|
|
|
|
type Tracer interface {
|
|
|
|
Component
|
|
|
|
}
|