2016-12-02 18:33:39 +00:00
|
|
|
package ipfscluster
|
|
|
|
|
|
|
|
import (
|
|
|
|
"context"
|
2017-02-13 15:46:53 +00:00
|
|
|
"errors"
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
"fmt"
|
2018-04-25 02:12:30 +00:00
|
|
|
"mime/multipart"
|
2019-04-01 05:33:58 +00:00
|
|
|
"sort"
|
2016-12-15 13:07:19 +00:00
|
|
|
"sync"
|
2017-01-23 19:29:05 +00:00
|
|
|
"time"
|
2016-12-02 18:33:39 +00:00
|
|
|
|
2018-07-04 16:30:24 +00:00
|
|
|
"github.com/ipfs/ipfs-cluster/adder"
|
|
|
|
"github.com/ipfs/ipfs-cluster/adder/sharding"
|
2019-08-28 09:50:55 +00:00
|
|
|
"github.com/ipfs/ipfs-cluster/adder/single"
|
2017-02-08 17:04:08 +00:00
|
|
|
"github.com/ipfs/ipfs-cluster/api"
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
"github.com/ipfs/ipfs-cluster/pstoremgr"
|
2018-05-01 16:01:59 +00:00
|
|
|
"github.com/ipfs/ipfs-cluster/rpcutil"
|
2017-03-10 16:24:25 +00:00
|
|
|
"github.com/ipfs/ipfs-cluster/state"
|
2018-12-18 14:44:11 +00:00
|
|
|
"github.com/ipfs/ipfs-cluster/version"
|
2017-02-08 17:04:08 +00:00
|
|
|
|
2016-12-16 21:00:08 +00:00
|
|
|
cid "github.com/ipfs/go-cid"
|
2019-02-20 14:24:25 +00:00
|
|
|
ds "github.com/ipfs/go-datastore"
|
2019-06-14 10:41:11 +00:00
|
|
|
host "github.com/libp2p/go-libp2p-core/host"
|
|
|
|
peer "github.com/libp2p/go-libp2p-core/peer"
|
|
|
|
peerstore "github.com/libp2p/go-libp2p-core/peerstore"
|
2018-10-17 13:28:03 +00:00
|
|
|
rpc "github.com/libp2p/go-libp2p-gorpc"
|
2018-07-17 10:51:31 +00:00
|
|
|
dht "github.com/libp2p/go-libp2p-kad-dht"
|
2019-08-24 15:09:54 +00:00
|
|
|
"github.com/libp2p/go-libp2p/p2p/discovery"
|
2017-01-24 15:19:23 +00:00
|
|
|
ma "github.com/multiformats/go-multiaddr"
|
2019-02-20 14:24:25 +00:00
|
|
|
|
|
|
|
ocgorpc "github.com/lanzafame/go-libp2p-ocgorpc"
|
|
|
|
trace "go.opencensus.io/trace"
|
2016-12-02 18:33:39 +00:00
|
|
|
)
|
|
|
|
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
// ReadyTimeout specifies the time before giving up
|
|
|
|
// during startup (waiting for consensus to be ready)
|
|
|
|
// It may need adjustment according to timeouts in the
|
|
|
|
// consensus layer.
|
|
|
|
var ReadyTimeout = 30 * time.Second
|
2018-03-20 13:27:49 +00:00
|
|
|
|
Fix #787: Connectivity fixes
Currently, unless doing Join() (--bootstrap), we do not connect to any peers on startup.
We however loaded up the peerstore file and Raft will automatically connect
older peers to figure out who is the leader etc. DHT bootstrap, after Raft
was working, did the rest.
For CRDTs we need to connect to people on a normal boot as otherwise, unless
bootstrapping, this does not happen, even if the peerstore contains known peers.
This introduces a number of changes:
* Move peerstore file management back inside the Cluster component, which was
already in charge of saving the peerstore file.
* We keep saving all "known addresses" but we load them with a non permanent
TTL, so that there will be clean up of peers we're not connected to for long.
* "Bootstrap" (connect) to a small number of peers during Cluster component creation.
* Bootstrap the DHT asap after this, so that other cluster components can
initialize with a working peer discovery mechanism.
* CRDT Trust() method will now:
* Protect the trusted Peer ID in the conn manager
* Give top priority in the PeerManager to that Peer (see below)
* Mark addresses as permanent in the Peerstore
The PeerManager now attaches priorities to peers when importing them and is
able to order them according to that priority. The result is that peers with
high priority are saved first in the peerstore file. When we load the peerstore
file, the first entries in it are given the highest priority.
This means that during startup we will connect to "trusted peers" first
(because they have been tagged with priority in the previous run and saved at
the top of the list). Once connected to a small number of peers, we let the
DHT bootstrap process in the background do the rest and discover the network.
All this makes the peerstore file a "bootstrap" list for CRDTs and we will attempt
to connect to peers on that list until some of those connections succeed.
2019-05-23 16:41:33 +00:00
|
|
|
const (
|
2019-06-09 11:26:18 +00:00
|
|
|
pingMetricName = "ping"
|
|
|
|
bootstrapCount = 3
|
|
|
|
reBootstrapInterval = 30 * time.Second
|
2019-08-24 15:09:54 +00:00
|
|
|
mdnsServiceTag = "_ipfs-cluster-discovery._udp"
|
Fix #787: Connectivity fixes
Currently, unless doing Join() (--bootstrap), we do not connect to any peers on startup.
We however loaded up the peerstore file and Raft will automatically connect
older peers to figure out who is the leader etc. DHT bootstrap, after Raft
was working, did the rest.
For CRDTs we need to connect to people on a normal boot as otherwise, unless
bootstrapping, this does not happen, even if the peerstore contains known peers.
This introduces a number of changes:
* Move peerstore file management back inside the Cluster component, which was
already in charge of saving the peerstore file.
* We keep saving all "known addresses" but we load them with a non permanent
TTL, so that there will be clean up of peers we're not connected to for long.
* "Bootstrap" (connect) to a small number of peers during Cluster component creation.
* Bootstrap the DHT asap after this, so that other cluster components can
initialize with a working peer discovery mechanism.
* CRDT Trust() method will now:
* Protect the trusted Peer ID in the conn manager
* Give top priority in the PeerManager to that Peer (see below)
* Mark addresses as permanent in the Peerstore
The PeerManager now attaches priorities to peers when importing them and is
able to order them according to that priority. The result is that peers with
high priority are saved first in the peerstore file. When we load the peerstore
file, the first entries in it are given the highest priority.
This means that during startup we will connect to "trusted peers" first
(because they have been tagged with priority in the previous run and saved at
the top of the list). Once connected to a small number of peers, we let the
DHT bootstrap process in the background do the rest and discover the network.
All this makes the peerstore file a "bootstrap" list for CRDTs and we will attempt
to connect to peers on that list until some of those connections succeed.
2019-05-23 16:41:33 +00:00
|
|
|
)
|
2018-09-26 11:42:20 +00:00
|
|
|
|
2019-07-30 17:42:26 +00:00
|
|
|
var (
|
2019-08-07 08:12:46 +00:00
|
|
|
errFollowerMode = errors.New("this peer is configured to be in follower mode. Write operations are disabled")
|
2019-07-30 17:42:26 +00:00
|
|
|
)
|
|
|
|
|
2016-12-02 18:33:39 +00:00
|
|
|
// Cluster is the main IPFS cluster component. It provides
|
2017-01-24 11:39:08 +00:00
|
|
|
// the go-API for it and orchestrates the components that make up the system.
|
2016-12-02 18:33:39 +00:00
|
|
|
type Cluster struct {
|
2017-02-09 15:29:17 +00:00
|
|
|
ctx context.Context
|
|
|
|
cancel func()
|
2016-12-02 18:33:39 +00:00
|
|
|
|
2019-02-20 14:24:25 +00:00
|
|
|
id peer.ID
|
|
|
|
config *Config
|
|
|
|
host host.Host
|
|
|
|
dht *dht.IpfsDHT
|
2019-08-24 15:09:54 +00:00
|
|
|
discovery discovery.Service
|
2019-02-20 14:24:25 +00:00
|
|
|
datastore ds.Datastore
|
|
|
|
|
2017-01-30 12:12:25 +00:00
|
|
|
rpcServer *rpc.Server
|
|
|
|
rpcClient *rpc.Client
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
peerManager *pstoremgr.Manager
|
2016-12-02 18:33:39 +00:00
|
|
|
|
2017-03-14 15:37:29 +00:00
|
|
|
consensus Consensus
|
2018-11-01 10:24:05 +00:00
|
|
|
apis []API
|
2016-12-02 18:33:39 +00:00
|
|
|
ipfs IPFSConnector
|
2016-12-06 21:29:59 +00:00
|
|
|
tracker PinTracker
|
2017-02-13 15:46:53 +00:00
|
|
|
monitor PeerMonitor
|
|
|
|
allocator PinAllocator
|
2019-12-05 14:08:43 +00:00
|
|
|
informers []Informer
|
2018-06-27 04:03:15 +00:00
|
|
|
tracer Tracer
|
2016-12-15 13:07:19 +00:00
|
|
|
|
2018-10-30 10:54:32 +00:00
|
|
|
doneCh chan struct{}
|
|
|
|
readyCh chan struct{}
|
|
|
|
readyB bool
|
|
|
|
wg sync.WaitGroup
|
|
|
|
|
|
|
|
// peerAdd
|
|
|
|
paMux sync.Mutex
|
|
|
|
|
|
|
|
// shutdown function and related variables
|
2016-12-15 13:07:19 +00:00
|
|
|
shutdownLock sync.Mutex
|
2017-10-31 10:20:14 +00:00
|
|
|
shutdownB bool
|
2017-11-01 12:25:28 +00:00
|
|
|
removed bool
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
2017-01-30 12:12:25 +00:00
|
|
|
// NewCluster builds a new IPFS Cluster peer. It initializes a LibP2P host,
|
|
|
|
// creates and RPC Server and client and sets up all components.
|
|
|
|
//
|
|
|
|
// The new cluster peer may still be performing initialization tasks when
|
|
|
|
// this call returns (consensus may still be bootstrapping). Use Cluster.Ready()
|
|
|
|
// if you need to wait until the peer is fully up.
|
2017-02-13 15:46:53 +00:00
|
|
|
func NewCluster(
|
2019-04-29 07:58:28 +00:00
|
|
|
ctx context.Context,
|
2018-03-13 17:16:15 +00:00
|
|
|
host host.Host,
|
2019-02-20 14:24:25 +00:00
|
|
|
dht *dht.IpfsDHT,
|
2017-02-13 15:46:53 +00:00
|
|
|
cfg *Config,
|
2019-02-20 14:24:25 +00:00
|
|
|
datastore ds.Datastore,
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
consensus Consensus,
|
2018-11-01 10:24:05 +00:00
|
|
|
apis []API,
|
2017-02-13 15:46:53 +00:00
|
|
|
ipfs IPFSConnector,
|
|
|
|
tracker PinTracker,
|
|
|
|
monitor PeerMonitor,
|
|
|
|
allocator PinAllocator,
|
2019-12-05 14:08:43 +00:00
|
|
|
informers []Informer,
|
2018-06-27 04:03:15 +00:00
|
|
|
tracer Tracer,
|
2018-07-04 16:30:24 +00:00
|
|
|
) (*Cluster, error) {
|
Issue #162: Rework configuration format
The following commit reimplements ipfs-cluster configuration under
the following premises:
* Each component is initialized with a configuration object
defined by its module
* Each component decides how the JSON representation of its
configuration looks like
* Each component parses and validates its own configuration
* Each component exposes its own defaults
* Component configurations are make the sections of a
central JSON configuration file (which replaces the current
JSON format)
* Component configurations implement a common interface
(config.ComponentConfig) with a set of common operations
* The central configuration file is managed by a
config.ConfigManager which:
* Registers ComponentConfigs
* Assigns the correspondent sections from the JSON file to each
component and delegates the parsing
* Delegates the JSON generation for each section
* Can be notified when the configuration is updated and must be
saved to disk
The new service.json would then look as follows:
```json
{
"cluster": {
"id": "QmTVW8NoRxC5wBhV7WtAYtRn7itipEESfozWN5KmXUQnk2",
"private_key": "<...>",
"secret": "00224102ae6aaf94f2606abf69a0e278251ecc1d64815b617ff19d6d2841f786",
"peers": [],
"bootstrap": [],
"leave_on_shutdown": false,
"listen_multiaddress": "/ip4/0.0.0.0/tcp/9096",
"state_sync_interval": "1m0s",
"ipfs_sync_interval": "2m10s",
"replication_factor": -1,
"monitor_ping_interval": "15s"
},
"consensus": {
"raft": {
"heartbeat_timeout": "1s",
"election_timeout": "1s",
"commit_timeout": "50ms",
"max_append_entries": 64,
"trailing_logs": 10240,
"snapshot_interval": "2m0s",
"snapshot_threshold": 8192,
"leader_lease_timeout": "500ms"
}
},
"api": {
"restapi": {
"listen_multiaddress": "/ip4/127.0.0.1/tcp/9094",
"read_timeout": "30s",
"read_header_timeout": "5s",
"write_timeout": "1m0s",
"idle_timeout": "2m0s"
}
},
"ipfs_connector": {
"ipfshttp": {
"proxy_listen_multiaddress": "/ip4/127.0.0.1/tcp/9095",
"node_multiaddress": "/ip4/127.0.0.1/tcp/5001",
"connect_swarms_delay": "7s",
"proxy_read_timeout": "10m0s",
"proxy_read_header_timeout": "5s",
"proxy_write_timeout": "10m0s",
"proxy_idle_timeout": "1m0s"
}
},
"monitor": {
"monbasic": {
"check_interval": "15s"
}
},
"informer": {
"disk": {
"metric_ttl": "30s",
"metric_type": "freespace"
},
"numpin": {
"metric_ttl": "10s"
}
}
}
```
This new format aims to be easily extensible per component. As such,
it already surfaces quite a few new options which were hardcoded
before.
Additionally, since Go API have changed, some redundant methods have been
removed and small refactoring has happened to take advantage of the new
way.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-10-11 18:23:03 +00:00
|
|
|
err := cfg.Validate()
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2018-03-13 17:16:15 +00:00
|
|
|
if host == nil {
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
return nil, errors.New("cluster host is nil")
|
|
|
|
}
|
|
|
|
|
2019-12-05 14:08:43 +00:00
|
|
|
if len(informers) == 0 {
|
|
|
|
return nil, errors.New("no informers are passed")
|
|
|
|
}
|
|
|
|
|
2019-04-29 07:58:28 +00:00
|
|
|
ctx, cancel := context.WithCancel(ctx)
|
2018-07-17 10:51:31 +00:00
|
|
|
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
listenAddrs := ""
|
|
|
|
for _, addr := range host.Addrs() {
|
2019-08-15 11:19:26 +00:00
|
|
|
listenAddrs += fmt.Sprintf(" %s/p2p/%s\n", addr, host.ID().Pretty())
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
2018-12-18 14:44:11 +00:00
|
|
|
logger.Infof("IPFS Cluster v%s listening on:\n%s\n", version.Version, listenAddrs)
|
2016-12-02 18:33:39 +00:00
|
|
|
|
Fix #787: Connectivity fixes
Currently, unless doing Join() (--bootstrap), we do not connect to any peers on startup.
We however loaded up the peerstore file and Raft will automatically connect
older peers to figure out who is the leader etc. DHT bootstrap, after Raft
was working, did the rest.
For CRDTs we need to connect to people on a normal boot as otherwise, unless
bootstrapping, this does not happen, even if the peerstore contains known peers.
This introduces a number of changes:
* Move peerstore file management back inside the Cluster component, which was
already in charge of saving the peerstore file.
* We keep saving all "known addresses" but we load them with a non permanent
TTL, so that there will be clean up of peers we're not connected to for long.
* "Bootstrap" (connect) to a small number of peers during Cluster component creation.
* Bootstrap the DHT asap after this, so that other cluster components can
initialize with a working peer discovery mechanism.
* CRDT Trust() method will now:
* Protect the trusted Peer ID in the conn manager
* Give top priority in the PeerManager to that Peer (see below)
* Mark addresses as permanent in the Peerstore
The PeerManager now attaches priorities to peers when importing them and is
able to order them according to that priority. The result is that peers with
high priority are saved first in the peerstore file. When we load the peerstore
file, the first entries in it are given the highest priority.
This means that during startup we will connect to "trusted peers" first
(because they have been tagged with priority in the previous run and saved at
the top of the list). Once connected to a small number of peers, we let the
DHT bootstrap process in the background do the rest and discover the network.
All this makes the peerstore file a "bootstrap" list for CRDTs and we will attempt
to connect to peers on that list until some of those connections succeed.
2019-05-23 16:41:33 +00:00
|
|
|
peerManager := pstoremgr.New(ctx, host, cfg.GetPeerstorePath())
|
2017-11-08 19:04:04 +00:00
|
|
|
|
2019-08-24 15:09:54 +00:00
|
|
|
var mdns discovery.Service
|
|
|
|
if cfg.MDNSInterval > 0 {
|
|
|
|
mdns, err := discovery.NewMdnsService(ctx, host, cfg.MDNSInterval, mdnsServiceTag)
|
|
|
|
if err != nil {
|
|
|
|
cancel()
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
mdns.RegisterNotifee(peerManager)
|
|
|
|
}
|
|
|
|
|
2017-02-02 22:52:06 +00:00
|
|
|
c := &Cluster{
|
2017-11-08 19:04:04 +00:00
|
|
|
ctx: ctx,
|
|
|
|
cancel: cancel,
|
|
|
|
id: host.ID(),
|
|
|
|
config: cfg,
|
2019-02-20 14:24:25 +00:00
|
|
|
host: host,
|
|
|
|
dht: dht,
|
2019-08-24 15:09:54 +00:00
|
|
|
discovery: mdns,
|
2019-02-20 14:24:25 +00:00
|
|
|
datastore: datastore,
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
consensus: consensus,
|
2018-11-01 10:24:05 +00:00
|
|
|
apis: apis,
|
2017-11-08 19:04:04 +00:00
|
|
|
ipfs: ipfs,
|
|
|
|
tracker: tracker,
|
|
|
|
monitor: monitor,
|
|
|
|
allocator: allocator,
|
2019-12-05 14:08:43 +00:00
|
|
|
informers: informers,
|
2018-06-27 04:03:15 +00:00
|
|
|
tracer: tracer,
|
2017-11-08 19:04:04 +00:00
|
|
|
peerManager: peerManager,
|
|
|
|
shutdownB: false,
|
|
|
|
removed: false,
|
|
|
|
doneCh: make(chan struct{}),
|
|
|
|
readyCh: make(chan struct{}),
|
|
|
|
readyB: false,
|
|
|
|
}
|
|
|
|
|
2019-12-02 11:52:24 +00:00
|
|
|
// Import known cluster peers from peerstore file and config. Set
|
Fix #787: Connectivity fixes
Currently, unless doing Join() (--bootstrap), we do not connect to any peers on startup.
We however loaded up the peerstore file and Raft will automatically connect
older peers to figure out who is the leader etc. DHT bootstrap, after Raft
was working, did the rest.
For CRDTs we need to connect to people on a normal boot as otherwise, unless
bootstrapping, this does not happen, even if the peerstore contains known peers.
This introduces a number of changes:
* Move peerstore file management back inside the Cluster component, which was
already in charge of saving the peerstore file.
* We keep saving all "known addresses" but we load them with a non permanent
TTL, so that there will be clean up of peers we're not connected to for long.
* "Bootstrap" (connect) to a small number of peers during Cluster component creation.
* Bootstrap the DHT asap after this, so that other cluster components can
initialize with a working peer discovery mechanism.
* CRDT Trust() method will now:
* Protect the trusted Peer ID in the conn manager
* Give top priority in the PeerManager to that Peer (see below)
* Mark addresses as permanent in the Peerstore
The PeerManager now attaches priorities to peers when importing them and is
able to order them according to that priority. The result is that peers with
high priority are saved first in the peerstore file. When we load the peerstore
file, the first entries in it are given the highest priority.
This means that during startup we will connect to "trusted peers" first
(because they have been tagged with priority in the previous run and saved at
the top of the list). Once connected to a small number of peers, we let the
DHT bootstrap process in the background do the rest and discover the network.
All this makes the peerstore file a "bootstrap" list for CRDTs and we will attempt
to connect to peers on that list until some of those connections succeed.
2019-05-23 16:41:33 +00:00
|
|
|
// a non permanent TTL.
|
|
|
|
c.peerManager.ImportPeersFromPeerstore(false, peerstore.AddressTTL)
|
2019-12-02 11:52:24 +00:00
|
|
|
c.peerManager.ImportPeers(c.config.PeerAddresses, false, peerstore.AddressTTL)
|
Fix #787: Connectivity fixes
Currently, unless doing Join() (--bootstrap), we do not connect to any peers on startup.
We however loaded up the peerstore file and Raft will automatically connect
older peers to figure out who is the leader etc. DHT bootstrap, after Raft
was working, did the rest.
For CRDTs we need to connect to people on a normal boot as otherwise, unless
bootstrapping, this does not happen, even if the peerstore contains known peers.
This introduces a number of changes:
* Move peerstore file management back inside the Cluster component, which was
already in charge of saving the peerstore file.
* We keep saving all "known addresses" but we load them with a non permanent
TTL, so that there will be clean up of peers we're not connected to for long.
* "Bootstrap" (connect) to a small number of peers during Cluster component creation.
* Bootstrap the DHT asap after this, so that other cluster components can
initialize with a working peer discovery mechanism.
* CRDT Trust() method will now:
* Protect the trusted Peer ID in the conn manager
* Give top priority in the PeerManager to that Peer (see below)
* Mark addresses as permanent in the Peerstore
The PeerManager now attaches priorities to peers when importing them and is
able to order them according to that priority. The result is that peers with
high priority are saved first in the peerstore file. When we load the peerstore
file, the first entries in it are given the highest priority.
This means that during startup we will connect to "trusted peers" first
(because they have been tagged with priority in the previous run and saved at
the top of the list). Once connected to a small number of peers, we let the
DHT bootstrap process in the background do the rest and discover the network.
All this makes the peerstore file a "bootstrap" list for CRDTs and we will attempt
to connect to peers on that list until some of those connections succeed.
2019-05-23 16:41:33 +00:00
|
|
|
// Attempt to connect to some peers (up to bootstrapCount)
|
2019-06-09 11:14:02 +00:00
|
|
|
connectedPeers := c.peerManager.Bootstrap(bootstrapCount)
|
|
|
|
// We cannot warn when count is low as this as this is normal if going
|
|
|
|
// to Join() later.
|
|
|
|
logger.Debugf("bootstrap count %d", len(connectedPeers))
|
|
|
|
// Log a ping metric for every connected peer. This will make them
|
|
|
|
// visible as peers without having to wait for them to send one.
|
|
|
|
for _, p := range connectedPeers {
|
|
|
|
if err := c.logPingMetric(ctx, p); err != nil {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn(err)
|
2019-06-09 11:14:02 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Fix #787: Connectivity fixes
Currently, unless doing Join() (--bootstrap), we do not connect to any peers on startup.
We however loaded up the peerstore file and Raft will automatically connect
older peers to figure out who is the leader etc. DHT bootstrap, after Raft
was working, did the rest.
For CRDTs we need to connect to people on a normal boot as otherwise, unless
bootstrapping, this does not happen, even if the peerstore contains known peers.
This introduces a number of changes:
* Move peerstore file management back inside the Cluster component, which was
already in charge of saving the peerstore file.
* We keep saving all "known addresses" but we load them with a non permanent
TTL, so that there will be clean up of peers we're not connected to for long.
* "Bootstrap" (connect) to a small number of peers during Cluster component creation.
* Bootstrap the DHT asap after this, so that other cluster components can
initialize with a working peer discovery mechanism.
* CRDT Trust() method will now:
* Protect the trusted Peer ID in the conn manager
* Give top priority in the PeerManager to that Peer (see below)
* Mark addresses as permanent in the Peerstore
The PeerManager now attaches priorities to peers when importing them and is
able to order them according to that priority. The result is that peers with
high priority are saved first in the peerstore file. When we load the peerstore
file, the first entries in it are given the highest priority.
This means that during startup we will connect to "trusted peers" first
(because they have been tagged with priority in the previous run and saved at
the top of the list). Once connected to a small number of peers, we let the
DHT bootstrap process in the background do the rest and discover the network.
All this makes the peerstore file a "bootstrap" list for CRDTs and we will attempt
to connect to peers on that list until some of those connections succeed.
2019-05-23 16:41:33 +00:00
|
|
|
// After setupRPC components can do their tasks with a fully operative
|
|
|
|
// routed libp2p host with some connections and a working DHT (hopefully).
|
2017-02-02 22:52:06 +00:00
|
|
|
err = c.setupRPC()
|
2016-12-23 18:35:37 +00:00
|
|
|
if err != nil {
|
2018-06-27 04:03:15 +00:00
|
|
|
c.Shutdown(ctx)
|
2016-12-23 18:35:37 +00:00
|
|
|
return nil, err
|
|
|
|
}
|
2017-02-02 22:52:06 +00:00
|
|
|
c.setupRPCClients()
|
2019-06-09 11:18:11 +00:00
|
|
|
|
|
|
|
// Note: It is very important to first call Add() once in a non-racy
|
|
|
|
// place
|
|
|
|
c.wg.Add(1)
|
2017-02-13 15:46:53 +00:00
|
|
|
go func() {
|
2019-06-09 11:18:11 +00:00
|
|
|
defer c.wg.Done()
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
c.ready(ReadyTimeout)
|
2017-02-13 15:46:53 +00:00
|
|
|
c.run()
|
|
|
|
}()
|
2018-06-27 04:03:15 +00:00
|
|
|
|
2017-02-02 22:52:06 +00:00
|
|
|
return c, nil
|
|
|
|
}
|
2017-01-30 12:12:25 +00:00
|
|
|
|
2017-02-02 22:52:06 +00:00
|
|
|
func (c *Cluster) setupRPC() error {
|
2019-05-04 20:36:10 +00:00
|
|
|
rpcServer, err := newRPCServer(c)
|
2017-02-02 22:52:06 +00:00
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
c.rpcServer = rpcServer
|
2018-06-27 04:03:15 +00:00
|
|
|
|
|
|
|
var rpcClient *rpc.Client
|
|
|
|
if c.config.Tracing {
|
|
|
|
csh := &ocgorpc.ClientHandler{}
|
2019-05-04 20:36:10 +00:00
|
|
|
rpcClient = rpc.NewClientWithServer(
|
|
|
|
c.host,
|
|
|
|
version.RPCProtocol,
|
|
|
|
rpcServer,
|
|
|
|
rpc.WithClientStatsHandler(csh),
|
|
|
|
)
|
2018-06-27 04:03:15 +00:00
|
|
|
} else {
|
|
|
|
rpcClient = rpc.NewClientWithServer(c.host, version.RPCProtocol, rpcServer)
|
|
|
|
}
|
2017-02-02 22:52:06 +00:00
|
|
|
c.rpcClient = rpcClient
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
|
|
|
func (c *Cluster) setupRPCClients() {
|
|
|
|
c.tracker.SetClient(c.rpcClient)
|
|
|
|
c.ipfs.SetClient(c.rpcClient)
|
2018-11-01 10:24:05 +00:00
|
|
|
for _, api := range c.apis {
|
|
|
|
api.SetClient(c.rpcClient)
|
|
|
|
}
|
2017-02-02 22:52:06 +00:00
|
|
|
c.consensus.SetClient(c.rpcClient)
|
2017-02-13 15:46:53 +00:00
|
|
|
c.monitor.SetClient(c.rpcClient)
|
|
|
|
c.allocator.SetClient(c.rpcClient)
|
2019-12-05 14:08:43 +00:00
|
|
|
for _, informer := range c.informers {
|
|
|
|
informer.SetClient(c.rpcClient)
|
|
|
|
}
|
2017-02-02 22:52:06 +00:00
|
|
|
}
|
2016-12-09 19:54:46 +00:00
|
|
|
|
2019-12-12 20:22:54 +00:00
|
|
|
// watchPinset triggers recurrent operations that loop on the pinset.
|
|
|
|
func (c *Cluster) watchPinset() {
|
|
|
|
ctx, span := trace.StartSpan(c.ctx, "cluster/watchPinset")
|
2018-06-27 04:03:15 +00:00
|
|
|
defer span.End()
|
|
|
|
|
Issue #162: Rework configuration format
The following commit reimplements ipfs-cluster configuration under
the following premises:
* Each component is initialized with a configuration object
defined by its module
* Each component decides how the JSON representation of its
configuration looks like
* Each component parses and validates its own configuration
* Each component exposes its own defaults
* Component configurations are make the sections of a
central JSON configuration file (which replaces the current
JSON format)
* Component configurations implement a common interface
(config.ComponentConfig) with a set of common operations
* The central configuration file is managed by a
config.ConfigManager which:
* Registers ComponentConfigs
* Assigns the correspondent sections from the JSON file to each
component and delegates the parsing
* Delegates the JSON generation for each section
* Can be notified when the configuration is updated and must be
saved to disk
The new service.json would then look as follows:
```json
{
"cluster": {
"id": "QmTVW8NoRxC5wBhV7WtAYtRn7itipEESfozWN5KmXUQnk2",
"private_key": "<...>",
"secret": "00224102ae6aaf94f2606abf69a0e278251ecc1d64815b617ff19d6d2841f786",
"peers": [],
"bootstrap": [],
"leave_on_shutdown": false,
"listen_multiaddress": "/ip4/0.0.0.0/tcp/9096",
"state_sync_interval": "1m0s",
"ipfs_sync_interval": "2m10s",
"replication_factor": -1,
"monitor_ping_interval": "15s"
},
"consensus": {
"raft": {
"heartbeat_timeout": "1s",
"election_timeout": "1s",
"commit_timeout": "50ms",
"max_append_entries": 64,
"trailing_logs": 10240,
"snapshot_interval": "2m0s",
"snapshot_threshold": 8192,
"leader_lease_timeout": "500ms"
}
},
"api": {
"restapi": {
"listen_multiaddress": "/ip4/127.0.0.1/tcp/9094",
"read_timeout": "30s",
"read_header_timeout": "5s",
"write_timeout": "1m0s",
"idle_timeout": "2m0s"
}
},
"ipfs_connector": {
"ipfshttp": {
"proxy_listen_multiaddress": "/ip4/127.0.0.1/tcp/9095",
"node_multiaddress": "/ip4/127.0.0.1/tcp/5001",
"connect_swarms_delay": "7s",
"proxy_read_timeout": "10m0s",
"proxy_read_header_timeout": "5s",
"proxy_write_timeout": "10m0s",
"proxy_idle_timeout": "1m0s"
}
},
"monitor": {
"monbasic": {
"check_interval": "15s"
}
},
"informer": {
"disk": {
"metric_ttl": "30s",
"metric_type": "freespace"
},
"numpin": {
"metric_ttl": "10s"
}
}
}
```
This new format aims to be easily extensible per component. As such,
it already surfaces quite a few new options which were hardcoded
before.
Additionally, since Go API have changed, some redundant methods have been
removed and small refactoring has happened to take advantage of the new
way.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-10-11 18:23:03 +00:00
|
|
|
stateSyncTicker := time.NewTicker(c.config.StateSyncInterval)
|
2019-07-08 14:55:05 +00:00
|
|
|
recoverTicker := time.NewTicker(c.config.PinRecoverInterval)
|
2017-04-05 21:29:22 +00:00
|
|
|
|
2017-02-02 22:52:06 +00:00
|
|
|
for {
|
|
|
|
select {
|
|
|
|
case <-stateSyncTicker.C:
|
2017-04-05 21:29:22 +00:00
|
|
|
logger.Debug("auto-triggering StateSync()")
|
2018-06-27 04:03:15 +00:00
|
|
|
c.StateSync(ctx)
|
2019-07-08 14:55:05 +00:00
|
|
|
case <-recoverTicker.C:
|
|
|
|
logger.Debug("auto-triggering RecoverAllLocal()")
|
|
|
|
c.RecoverAllLocal(ctx)
|
2017-02-02 22:52:06 +00:00
|
|
|
case <-c.ctx.Done():
|
|
|
|
stateSyncTicker.Stop()
|
2019-12-12 20:22:54 +00:00
|
|
|
recoverTicker.Stop()
|
2017-02-02 22:52:06 +00:00
|
|
|
return
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-12-05 14:08:43 +00:00
|
|
|
func (c *Cluster) sendInformerMetric(ctx context.Context, informer Informer) (*api.Metric, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/sendInformerMetric")
|
|
|
|
defer span.End()
|
|
|
|
|
2019-12-05 14:08:43 +00:00
|
|
|
metric := informer.GetMetric(ctx)
|
2018-08-07 09:49:31 +00:00
|
|
|
metric.Peer = c.id
|
2018-06-27 04:03:15 +00:00
|
|
|
return metric, c.monitor.PublishMetric(ctx, metric)
|
2018-08-07 09:49:31 +00:00
|
|
|
}
|
|
|
|
|
2019-12-05 14:08:43 +00:00
|
|
|
func (c *Cluster) sendInformersMetrics(ctx context.Context) ([]*api.Metric, error) {
|
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/sendInformersMetrics")
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
var metrics []*api.Metric
|
|
|
|
for _, informer := range c.informers {
|
|
|
|
m, err := c.sendInformerMetric(ctx, informer)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
metrics = append(metrics, m)
|
|
|
|
}
|
|
|
|
return metrics, nil
|
|
|
|
}
|
|
|
|
|
2018-05-02 07:43:16 +00:00
|
|
|
// pushInformerMetrics loops and publishes informers metrics using the
|
|
|
|
// cluster monitor. Metrics are pushed normally at a TTL/2 rate. If an error
|
|
|
|
// occurs, they are pushed at a TTL/4 rate.
|
2019-12-05 14:08:43 +00:00
|
|
|
func (c *Cluster) pushInformerMetrics(ctx context.Context, informer Informer) {
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/pushInformerMetrics")
|
|
|
|
defer span.End()
|
|
|
|
|
2017-02-13 15:46:53 +00:00
|
|
|
timer := time.NewTimer(0) // fire immediately first
|
2018-05-02 07:43:16 +00:00
|
|
|
|
|
|
|
// retries counts how many retries we have made
|
2017-10-23 11:46:37 +00:00
|
|
|
retries := 0
|
2018-05-02 07:43:16 +00:00
|
|
|
// retryWarnMod controls how often do we log
|
|
|
|
// "error broadcasting metric".
|
|
|
|
// It will do it in the first error, and then on every
|
|
|
|
// 10th.
|
|
|
|
retryWarnMod := 10
|
|
|
|
|
2017-02-13 15:46:53 +00:00
|
|
|
for {
|
|
|
|
select {
|
2018-06-27 04:03:15 +00:00
|
|
|
case <-ctx.Done():
|
2017-02-13 15:46:53 +00:00
|
|
|
return
|
|
|
|
case <-timer.C:
|
|
|
|
// wait
|
|
|
|
}
|
|
|
|
|
2019-12-05 14:08:43 +00:00
|
|
|
metric, err := c.sendInformerMetric(ctx, informer)
|
2017-02-13 15:46:53 +00:00
|
|
|
|
|
|
|
if err != nil {
|
2017-10-23 11:46:37 +00:00
|
|
|
if (retries % retryWarnMod) == 0 {
|
|
|
|
logger.Errorf("error broadcasting metric: %s", err)
|
|
|
|
retries++
|
|
|
|
}
|
2018-05-01 16:01:59 +00:00
|
|
|
// retry sooner
|
2018-03-16 16:37:39 +00:00
|
|
|
timer.Reset(metric.GetTTL() / 4)
|
2017-02-13 15:46:53 +00:00
|
|
|
continue
|
|
|
|
}
|
|
|
|
|
2017-10-23 11:46:37 +00:00
|
|
|
retries = 0
|
|
|
|
// send metric again in TTL/2
|
2017-02-28 15:01:26 +00:00
|
|
|
timer.Reset(metric.GetTTL() / 2)
|
|
|
|
}
|
|
|
|
}
|
2017-02-13 15:46:53 +00:00
|
|
|
|
2019-05-09 12:19:07 +00:00
|
|
|
func (c *Cluster) sendPingMetric(ctx context.Context) (*api.Metric, error) {
|
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/sendPingMetric")
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
metric := &api.Metric{
|
|
|
|
Name: pingMetricName,
|
|
|
|
Peer: c.id,
|
|
|
|
Valid: true,
|
|
|
|
}
|
|
|
|
metric.SetTTL(c.config.MonitorPingInterval * 2)
|
|
|
|
return metric, c.monitor.PublishMetric(ctx, metric)
|
|
|
|
}
|
|
|
|
|
2019-06-09 11:14:02 +00:00
|
|
|
// logPingMetric logs a ping metric as if it had been sent from PID. It is
|
|
|
|
// used to make peers appear available as soon as we connect to them (without
|
|
|
|
// having to wait for them to broadcast a metric).
|
|
|
|
//
|
|
|
|
// We avoid specifically sending a metric to a peer when we "connect" to it
|
|
|
|
// because: a) this requires an extra. OPEN RPC endpoint (LogMetric) that can
|
|
|
|
// be called by everyone b) We have no way of verifying that the peer ID in a
|
|
|
|
// metric pushed is actually the issuer of the metric (something the regular
|
|
|
|
// "pubsub" way of pushing metrics allows (by verifying the signature on the
|
|
|
|
// message). Thus, this reduces chances of abuse until we have something
|
|
|
|
// better.
|
|
|
|
func (c *Cluster) logPingMetric(ctx context.Context, pid peer.ID) error {
|
|
|
|
m := &api.Metric{
|
|
|
|
Name: pingMetricName,
|
|
|
|
Peer: pid,
|
|
|
|
Valid: true,
|
|
|
|
}
|
|
|
|
m.SetTTL(c.config.MonitorPingInterval * 2)
|
|
|
|
return c.monitor.LogMetric(ctx, m)
|
|
|
|
}
|
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
func (c *Cluster) pushPingMetrics(ctx context.Context) {
|
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/pushPingMetrics")
|
|
|
|
defer span.End()
|
|
|
|
|
Issue #162: Rework configuration format
The following commit reimplements ipfs-cluster configuration under
the following premises:
* Each component is initialized with a configuration object
defined by its module
* Each component decides how the JSON representation of its
configuration looks like
* Each component parses and validates its own configuration
* Each component exposes its own defaults
* Component configurations are make the sections of a
central JSON configuration file (which replaces the current
JSON format)
* Component configurations implement a common interface
(config.ComponentConfig) with a set of common operations
* The central configuration file is managed by a
config.ConfigManager which:
* Registers ComponentConfigs
* Assigns the correspondent sections from the JSON file to each
component and delegates the parsing
* Delegates the JSON generation for each section
* Can be notified when the configuration is updated and must be
saved to disk
The new service.json would then look as follows:
```json
{
"cluster": {
"id": "QmTVW8NoRxC5wBhV7WtAYtRn7itipEESfozWN5KmXUQnk2",
"private_key": "<...>",
"secret": "00224102ae6aaf94f2606abf69a0e278251ecc1d64815b617ff19d6d2841f786",
"peers": [],
"bootstrap": [],
"leave_on_shutdown": false,
"listen_multiaddress": "/ip4/0.0.0.0/tcp/9096",
"state_sync_interval": "1m0s",
"ipfs_sync_interval": "2m10s",
"replication_factor": -1,
"monitor_ping_interval": "15s"
},
"consensus": {
"raft": {
"heartbeat_timeout": "1s",
"election_timeout": "1s",
"commit_timeout": "50ms",
"max_append_entries": 64,
"trailing_logs": 10240,
"snapshot_interval": "2m0s",
"snapshot_threshold": 8192,
"leader_lease_timeout": "500ms"
}
},
"api": {
"restapi": {
"listen_multiaddress": "/ip4/127.0.0.1/tcp/9094",
"read_timeout": "30s",
"read_header_timeout": "5s",
"write_timeout": "1m0s",
"idle_timeout": "2m0s"
}
},
"ipfs_connector": {
"ipfshttp": {
"proxy_listen_multiaddress": "/ip4/127.0.0.1/tcp/9095",
"node_multiaddress": "/ip4/127.0.0.1/tcp/5001",
"connect_swarms_delay": "7s",
"proxy_read_timeout": "10m0s",
"proxy_read_header_timeout": "5s",
"proxy_write_timeout": "10m0s",
"proxy_idle_timeout": "1m0s"
}
},
"monitor": {
"monbasic": {
"check_interval": "15s"
}
},
"informer": {
"disk": {
"metric_ttl": "30s",
"metric_type": "freespace"
},
"numpin": {
"metric_ttl": "10s"
}
}
}
```
This new format aims to be easily extensible per component. As such,
it already surfaces quite a few new options which were hardcoded
before.
Additionally, since Go API have changed, some redundant methods have been
removed and small refactoring has happened to take advantage of the new
way.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-10-11 18:23:03 +00:00
|
|
|
ticker := time.NewTicker(c.config.MonitorPingInterval)
|
2017-02-28 15:01:26 +00:00
|
|
|
for {
|
2019-05-09 12:19:07 +00:00
|
|
|
c.sendPingMetric(ctx)
|
2017-02-13 15:46:53 +00:00
|
|
|
|
2017-02-28 15:01:26 +00:00
|
|
|
select {
|
2018-06-27 04:03:15 +00:00
|
|
|
case <-ctx.Done():
|
2017-02-28 15:01:26 +00:00
|
|
|
return
|
|
|
|
case <-ticker.C:
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2017-02-13 15:46:53 +00:00
|
|
|
|
2017-02-28 15:01:26 +00:00
|
|
|
// read the alerts channel from the monitor and triggers repins
|
|
|
|
func (c *Cluster) alertsHandler() {
|
|
|
|
for {
|
|
|
|
select {
|
|
|
|
case <-c.ctx.Done():
|
|
|
|
return
|
|
|
|
case alrt := <-c.monitor.Alerts():
|
2019-12-16 12:38:17 +00:00
|
|
|
// Follower peers do not care about alerts.
|
|
|
|
// They can do nothing about them.
|
|
|
|
if c.config.FollowerMode {
|
|
|
|
continue
|
|
|
|
}
|
|
|
|
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warnf("metric alert for %s: Peer: %s.", alrt.MetricName, alrt.Peer)
|
2018-08-15 10:30:00 +00:00
|
|
|
if alrt.MetricName != pingMetricName {
|
|
|
|
continue // only handle ping alerts
|
|
|
|
}
|
|
|
|
|
2019-09-06 15:56:00 +00:00
|
|
|
if c.config.DisableRepinning {
|
|
|
|
logger.Debugf("repinning is disabled. Will not re-allocate pins on alerts")
|
|
|
|
return
|
|
|
|
}
|
|
|
|
|
2019-04-18 07:41:32 +00:00
|
|
|
cState, err := c.consensus.State(c.ctx)
|
|
|
|
if err != nil {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn(err)
|
2019-04-18 07:41:32 +00:00
|
|
|
return
|
|
|
|
}
|
|
|
|
list, err := cState.List(c.ctx)
|
|
|
|
if err != nil {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn(err)
|
2019-04-18 07:41:32 +00:00
|
|
|
return
|
|
|
|
}
|
2019-04-01 05:33:58 +00:00
|
|
|
for _, pin := range list {
|
2019-04-17 06:39:00 +00:00
|
|
|
if len(pin.Allocations) == 1 && containsPeer(pin.Allocations, alrt.Peer) {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn("a pin with only one allocation cannot be repinned")
|
|
|
|
logger.Warn("to make repinning possible, pin with a replication factor of 2+")
|
2019-04-01 05:33:58 +00:00
|
|
|
continue
|
|
|
|
}
|
|
|
|
if c.shouldPeerRepinCid(alrt.Peer, pin) {
|
2019-09-06 15:56:00 +00:00
|
|
|
c.repinFromPeer(c.ctx, alrt.Peer, pin)
|
2017-02-28 15:01:26 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-04-01 05:33:58 +00:00
|
|
|
// shouldPeerRepinCid returns true if the current peer is the top of the
|
|
|
|
// allocs list. The failed peer is ignored, i.e. if current peer is
|
|
|
|
// second and the failed peer is first, the function will still
|
|
|
|
// return true.
|
|
|
|
func (c *Cluster) shouldPeerRepinCid(failed peer.ID, pin *api.Pin) bool {
|
|
|
|
if containsPeer(pin.Allocations, failed) && containsPeer(pin.Allocations, c.id) {
|
2019-04-17 06:39:00 +00:00
|
|
|
allocs := peer.IDSlice(pin.Allocations)
|
|
|
|
sort.Sort(allocs)
|
|
|
|
if allocs[0] == c.id {
|
2019-04-01 05:33:58 +00:00
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
2019-04-17 06:39:00 +00:00
|
|
|
if allocs[1] == c.id && allocs[0] == failed {
|
2019-04-01 05:33:58 +00:00
|
|
|
return true
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
|
2017-11-08 19:04:04 +00:00
|
|
|
// detects any changes in the peerset and saves the configuration. When it
|
|
|
|
// detects that we have been removed from the peerset, it shuts down this peer.
|
|
|
|
func (c *Cluster) watchPeers() {
|
2018-03-16 16:36:43 +00:00
|
|
|
ticker := time.NewTicker(c.config.PeerWatchInterval)
|
2019-06-09 11:26:18 +00:00
|
|
|
defer ticker.Stop()
|
2017-11-08 19:04:04 +00:00
|
|
|
|
|
|
|
for {
|
|
|
|
select {
|
|
|
|
case <-c.ctx.Done():
|
|
|
|
return
|
|
|
|
case <-ticker.C:
|
2019-09-06 15:56:00 +00:00
|
|
|
// logger.Debugf("%s watching peers", c.id)
|
2017-11-08 19:04:04 +00:00
|
|
|
hasMe := false
|
2018-06-27 04:03:15 +00:00
|
|
|
peers, err := c.consensus.Peers(c.ctx)
|
2017-11-08 19:04:04 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
continue
|
|
|
|
}
|
|
|
|
for _, p := range peers {
|
|
|
|
if p == c.id {
|
|
|
|
hasMe = true
|
|
|
|
break
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if !hasMe {
|
2018-10-26 12:45:16 +00:00
|
|
|
c.shutdownLock.Lock()
|
|
|
|
defer c.shutdownLock.Unlock()
|
2018-08-15 10:30:00 +00:00
|
|
|
logger.Info("peer no longer in peerset. Initiating shutdown")
|
2017-11-08 19:04:04 +00:00
|
|
|
c.removed = true
|
2018-06-27 04:03:15 +00:00
|
|
|
go c.Shutdown(c.ctx)
|
2017-11-08 19:04:04 +00:00
|
|
|
return
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-02-03 09:30:04 +00:00
|
|
|
// reBootstrap regularly attempts to bootstrap (re-connect to peers from the
|
2019-06-09 11:26:18 +00:00
|
|
|
// peerstore). This should ensure that we auto-recover from situations in
|
|
|
|
// which the network was completely gone and we lost all peers.
|
|
|
|
func (c *Cluster) reBootstrap() {
|
|
|
|
ticker := time.NewTicker(reBootstrapInterval)
|
|
|
|
defer ticker.Stop()
|
|
|
|
|
|
|
|
for {
|
|
|
|
select {
|
|
|
|
case <-c.ctx.Done():
|
|
|
|
return
|
|
|
|
case <-ticker.C:
|
|
|
|
connected := c.peerManager.Bootstrap(bootstrapCount)
|
|
|
|
for _, p := range connected {
|
|
|
|
logger.Infof("reconnected to %s", p)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-07-03 15:45:22 +00:00
|
|
|
// find all Cids pinned to a given peer and triggers re-pins on them.
|
2019-09-06 15:56:00 +00:00
|
|
|
func (c *Cluster) vacatePeer(ctx context.Context, p peer.ID) {
|
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/vacatePeer")
|
2018-06-27 04:03:15 +00:00
|
|
|
defer span.End()
|
|
|
|
|
2018-04-22 16:25:18 +00:00
|
|
|
if c.config.DisableRepinning {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warnf("repinning is disabled. Will not re-allocate cids from %s", p.Pretty())
|
2018-04-22 16:25:18 +00:00
|
|
|
return
|
|
|
|
}
|
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
cState, err := c.consensus.State(ctx)
|
2017-02-28 15:01:26 +00:00
|
|
|
if err != nil {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn(err)
|
2017-02-28 15:01:26 +00:00
|
|
|
return
|
|
|
|
}
|
2019-02-20 14:24:25 +00:00
|
|
|
list, err := cState.List(ctx)
|
|
|
|
if err != nil {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn(err)
|
2019-02-20 14:24:25 +00:00
|
|
|
return
|
|
|
|
}
|
2017-03-08 17:28:43 +00:00
|
|
|
for _, pin := range list {
|
2017-07-05 14:28:52 +00:00
|
|
|
if containsPeer(pin.Allocations, p) {
|
2019-09-06 15:56:00 +00:00
|
|
|
c.repinFromPeer(ctx, p, pin)
|
2017-02-28 15:01:26 +00:00
|
|
|
}
|
2017-02-13 15:46:53 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-09-06 15:56:00 +00:00
|
|
|
// repinFromPeer triggers a repin on a given pin object blacklisting one of the
|
|
|
|
// allocations.
|
|
|
|
func (c *Cluster) repinFromPeer(ctx context.Context, p peer.ID, pin *api.Pin) {
|
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/repinFromPeer")
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
pin.Allocations = nil // force re-allocations
|
|
|
|
_, ok, err := c.pin(ctx, pin, []peer.ID{p})
|
|
|
|
if ok && err == nil {
|
|
|
|
logger.Infof("repinned %s out of %s", pin.Cid, p.Pretty())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-11-08 19:04:04 +00:00
|
|
|
// run launches some go-routines which live throughout the cluster's life
|
2017-02-02 22:52:06 +00:00
|
|
|
func (c *Cluster) run() {
|
2019-06-09 11:18:11 +00:00
|
|
|
c.wg.Add(1)
|
|
|
|
go func() {
|
|
|
|
defer c.wg.Done()
|
2019-12-12 20:22:54 +00:00
|
|
|
c.watchPinset()
|
2019-06-09 11:18:11 +00:00
|
|
|
}()
|
|
|
|
|
|
|
|
c.wg.Add(1)
|
|
|
|
go func() {
|
|
|
|
defer c.wg.Done()
|
|
|
|
c.pushPingMetrics(c.ctx)
|
|
|
|
}()
|
|
|
|
|
2019-12-05 14:08:43 +00:00
|
|
|
c.wg.Add(len(c.informers))
|
|
|
|
for _, informer := range c.informers {
|
|
|
|
go func(inf Informer) {
|
|
|
|
defer c.wg.Done()
|
|
|
|
c.pushInformerMetrics(c.ctx, inf)
|
|
|
|
}(informer)
|
|
|
|
}
|
2019-06-09 11:18:11 +00:00
|
|
|
|
|
|
|
c.wg.Add(1)
|
|
|
|
go func() {
|
|
|
|
defer c.wg.Done()
|
|
|
|
c.watchPeers()
|
|
|
|
}()
|
|
|
|
|
|
|
|
c.wg.Add(1)
|
|
|
|
go func() {
|
|
|
|
defer c.wg.Done()
|
|
|
|
c.alertsHandler()
|
|
|
|
}()
|
2019-06-09 11:26:18 +00:00
|
|
|
|
|
|
|
c.wg.Add(1)
|
|
|
|
go func() {
|
|
|
|
defer c.wg.Done()
|
|
|
|
c.reBootstrap()
|
|
|
|
}()
|
2017-02-02 22:52:06 +00:00
|
|
|
}
|
|
|
|
|
2018-03-12 21:45:29 +00:00
|
|
|
func (c *Cluster) ready(timeout time.Duration) {
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx, span := trace.StartSpan(c.ctx, "cluster/ready")
|
|
|
|
defer span.End()
|
|
|
|
|
2017-02-02 22:52:06 +00:00
|
|
|
// We bootstrapped first because with dirty state consensus
|
|
|
|
// may have a peerset and not find a leader so we cannot wait
|
|
|
|
// for it.
|
2018-03-12 21:45:29 +00:00
|
|
|
timer := time.NewTimer(timeout)
|
2017-02-02 22:52:06 +00:00
|
|
|
select {
|
|
|
|
case <-timer.C:
|
2018-03-13 09:37:47 +00:00
|
|
|
logger.Error("***** ipfs-cluster consensus start timed out (tips below) *****")
|
|
|
|
logger.Error(`
|
|
|
|
**************************************************
|
|
|
|
This peer was not able to become part of the cluster.
|
|
|
|
This might be due to one or several causes:
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
- Check the logs above this message for errors
|
2018-03-20 13:27:49 +00:00
|
|
|
- Check that there is connectivity to the "peers" multiaddresses
|
2018-03-13 09:37:47 +00:00
|
|
|
- Check that all cluster peers are using the same "secret"
|
2018-03-20 13:27:49 +00:00
|
|
|
- Check that this peer is reachable on its "listen_multiaddress" by all peers
|
|
|
|
- Check that the current cluster is healthy (has a leader). Otherwise make
|
|
|
|
sure to start enough peers so that a leader election can happen.
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
- Check that the peer(s) you are trying to connect to is running the
|
2018-04-24 12:27:01 +00:00
|
|
|
same version of IPFS-cluster.
|
2018-03-13 09:37:47 +00:00
|
|
|
**************************************************
|
|
|
|
`)
|
2018-06-27 04:03:15 +00:00
|
|
|
c.Shutdown(ctx)
|
2017-02-02 22:52:06 +00:00
|
|
|
return
|
2018-06-27 04:03:15 +00:00
|
|
|
case <-c.consensus.Ready(ctx):
|
2019-12-13 11:00:03 +00:00
|
|
|
// Consensus ready means the state is up to date. Every item
|
|
|
|
// in the state that is not pinned will appear as PinError so
|
|
|
|
// we can proceed to recover all of those in the tracker.
|
|
|
|
c.RecoverAllLocal(ctx)
|
2017-02-02 22:52:06 +00:00
|
|
|
case <-c.ctx.Done():
|
|
|
|
return
|
|
|
|
}
|
|
|
|
|
|
|
|
// Cluster is ready.
|
2019-02-20 14:24:25 +00:00
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
peers, err := c.consensus.Peers(ctx)
|
2017-11-08 19:04:04 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
2018-06-27 04:03:15 +00:00
|
|
|
c.Shutdown(ctx)
|
2017-11-08 19:04:04 +00:00
|
|
|
return
|
|
|
|
}
|
|
|
|
|
|
|
|
logger.Info("Cluster Peers (without including ourselves):")
|
|
|
|
if len(peers) == 1 {
|
2017-02-02 22:52:06 +00:00
|
|
|
logger.Info(" - No other peers")
|
|
|
|
}
|
2018-03-16 16:36:43 +00:00
|
|
|
|
2017-11-08 19:04:04 +00:00
|
|
|
for _, p := range peers {
|
|
|
|
if p != c.id {
|
|
|
|
logger.Infof(" - %s", p.Pretty())
|
|
|
|
}
|
2017-02-02 22:52:06 +00:00
|
|
|
}
|
2018-03-16 16:36:43 +00:00
|
|
|
|
2017-02-13 15:46:53 +00:00
|
|
|
close(c.readyCh)
|
2018-08-15 10:27:01 +00:00
|
|
|
c.shutdownLock.Lock()
|
2017-10-30 11:17:39 +00:00
|
|
|
c.readyB = true
|
2018-08-15 10:27:01 +00:00
|
|
|
c.shutdownLock.Unlock()
|
2017-12-12 16:47:21 +00:00
|
|
|
logger.Info("** IPFS Cluster is READY **")
|
2017-02-02 22:52:06 +00:00
|
|
|
}
|
|
|
|
|
2017-01-30 12:12:25 +00:00
|
|
|
// Ready returns a channel which signals when this peer is
|
|
|
|
// fully initialized (including consensus).
|
|
|
|
func (c *Cluster) Ready() <-chan struct{} {
|
2017-02-02 22:52:06 +00:00
|
|
|
return c.readyCh
|
2017-01-30 12:12:25 +00:00
|
|
|
}
|
|
|
|
|
2019-12-13 08:47:42 +00:00
|
|
|
// Shutdown performs all the necessary operations to shutdown
|
|
|
|
// the IPFS Cluster peer:
|
|
|
|
// * Save peerstore with the current peers
|
|
|
|
// * Remove itself from consensus when LeaveOnShutdown is set
|
|
|
|
// * It Shutdowns all the components
|
|
|
|
// * Closes the datastore
|
|
|
|
// * Collects all goroutines
|
|
|
|
//
|
|
|
|
// Shutdown does not closes the libp2p host or the DHT.
|
2018-06-27 04:03:15 +00:00
|
|
|
func (c *Cluster) Shutdown(ctx context.Context) error {
|
|
|
|
_, span := trace.StartSpan(ctx, "cluster/Shutdown")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2016-12-15 13:07:19 +00:00
|
|
|
c.shutdownLock.Lock()
|
|
|
|
defer c.shutdownLock.Unlock()
|
2017-02-02 22:52:06 +00:00
|
|
|
|
2017-10-31 10:20:14 +00:00
|
|
|
if c.shutdownB {
|
|
|
|
logger.Debug("Cluster is already shutdown")
|
2016-12-15 13:07:19 +00:00
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
2017-10-31 10:20:14 +00:00
|
|
|
logger.Info("shutting down Cluster")
|
2017-02-02 22:52:06 +00:00
|
|
|
|
2019-08-24 15:09:54 +00:00
|
|
|
// Cancel discovery service (this shutdowns announcing). Handling
|
|
|
|
// entries is cancelled along with the context below.
|
|
|
|
if c.discovery != nil {
|
|
|
|
c.discovery.Close()
|
|
|
|
}
|
|
|
|
|
2018-07-17 10:51:31 +00:00
|
|
|
// Try to store peerset file for all known peers whatsoever
|
|
|
|
// if we got ready (otherwise, don't overwrite anything)
|
|
|
|
if c.readyB {
|
2019-07-25 08:47:44 +00:00
|
|
|
// Ignoring error since it's a best-effort
|
2018-07-17 10:51:31 +00:00
|
|
|
c.peerManager.SavePeerstoreForPeers(c.host.Peerstore().Peers())
|
|
|
|
}
|
|
|
|
|
2017-11-01 12:41:55 +00:00
|
|
|
// Only attempt to leave if:
|
|
|
|
// - consensus is initialized
|
|
|
|
// - cluster was ready (no bootstrapping error)
|
Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.
Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
* The current cluster peerset, as required by the implementation
* The current cluster pinset (shared state)
While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.
The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.
In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.
By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:
* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.
* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.
* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.
* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.
* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.
* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically
* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.
While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.
As a side effect, the PR also:
* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-28 22:22:23 +00:00
|
|
|
// - We are not removed already (means watchPeers() called us)
|
2017-11-01 12:41:55 +00:00
|
|
|
if c.consensus != nil && c.config.LeaveOnShutdown && c.readyB && !c.removed {
|
|
|
|
c.removed = true
|
2018-06-27 04:03:15 +00:00
|
|
|
_, err := c.consensus.Peers(ctx)
|
2017-11-08 19:04:04 +00:00
|
|
|
if err == nil {
|
|
|
|
// best effort
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn("attempting to leave the cluster. This may take some seconds")
|
2018-06-27 04:03:15 +00:00
|
|
|
err := c.consensus.RmPeer(ctx, c.id)
|
2017-11-08 19:04:04 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error("leaving cluster: " + err.Error())
|
|
|
|
}
|
2017-02-02 22:52:06 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-01-30 12:12:25 +00:00
|
|
|
if con := c.consensus; con != nil {
|
2018-06-27 04:03:15 +00:00
|
|
|
if err := con.Shutdown(ctx); err != nil {
|
2017-01-30 12:12:25 +00:00
|
|
|
logger.Errorf("error stopping consensus: %s", err)
|
|
|
|
return err
|
|
|
|
}
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
2017-02-02 22:52:06 +00:00
|
|
|
|
2019-02-20 14:24:25 +00:00
|
|
|
// We left the cluster or were removed. Remove any consensus-specific
|
|
|
|
// state.
|
2017-11-01 12:25:28 +00:00
|
|
|
if c.removed && c.readyB {
|
2018-06-27 04:03:15 +00:00
|
|
|
err := c.consensus.Clean(ctx)
|
2017-11-01 12:25:28 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error("cleaning consensus: ", err)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
if err := c.monitor.Shutdown(ctx); err != nil {
|
2017-02-28 15:01:26 +00:00
|
|
|
logger.Errorf("error stopping monitor: %s", err)
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
2018-11-01 10:24:05 +00:00
|
|
|
for _, api := range c.apis {
|
2018-06-27 04:03:15 +00:00
|
|
|
if err := api.Shutdown(ctx); err != nil {
|
2018-11-01 10:24:05 +00:00
|
|
|
logger.Errorf("error stopping API: %s", err)
|
|
|
|
return err
|
|
|
|
}
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
2018-11-01 10:24:05 +00:00
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
if err := c.ipfs.Shutdown(ctx); err != nil {
|
2016-12-15 13:19:41 +00:00
|
|
|
logger.Errorf("error stopping IPFS Connector: %s", err)
|
2016-12-02 18:33:39 +00:00
|
|
|
return err
|
|
|
|
}
|
2016-12-07 16:21:29 +00:00
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
if err := c.tracker.Shutdown(ctx); err != nil {
|
2016-12-15 13:19:41 +00:00
|
|
|
logger.Errorf("error stopping PinTracker: %s", err)
|
2016-12-07 16:21:29 +00:00
|
|
|
return err
|
|
|
|
}
|
2017-11-15 01:33:46 +00:00
|
|
|
|
2019-12-13 08:48:12 +00:00
|
|
|
for _, inf := range c.informers {
|
|
|
|
if err := inf.Shutdown(ctx); err != nil {
|
|
|
|
logger.Errorf("error stopping informer: %s", err)
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
if err := c.tracer.Shutdown(ctx); err != nil {
|
|
|
|
logger.Errorf("error stopping Tracer: %s", err)
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
2017-11-15 01:33:46 +00:00
|
|
|
c.cancel()
|
|
|
|
c.wg.Wait()
|
2019-02-20 14:24:25 +00:00
|
|
|
|
|
|
|
// Cleanly close the datastore
|
|
|
|
if err := c.datastore.Close(); err != nil {
|
|
|
|
logger.Errorf("error closing Datastore: %s", err)
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
2017-10-31 10:20:14 +00:00
|
|
|
c.shutdownB = true
|
2017-01-30 12:12:25 +00:00
|
|
|
close(c.doneCh)
|
2016-12-09 19:54:46 +00:00
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
2017-01-30 12:12:25 +00:00
|
|
|
// Done provides a way to learn if the Peer has been shutdown
|
|
|
|
// (for example, because it has been removed from the Cluster)
|
|
|
|
func (c *Cluster) Done() <-chan struct{} {
|
|
|
|
return c.doneCh
|
|
|
|
}
|
|
|
|
|
2017-01-25 17:07:19 +00:00
|
|
|
// ID returns information about the Cluster peer
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) ID(ctx context.Context) *api.ID {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/ID")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2017-01-26 18:59:31 +00:00
|
|
|
// ignore error since it is included in response object
|
2019-02-27 17:04:35 +00:00
|
|
|
ipfsID, err := c.ipfs.ID(ctx)
|
|
|
|
if err != nil {
|
|
|
|
ipfsID = &api.IPFSID{
|
|
|
|
Error: err.Error(),
|
|
|
|
}
|
|
|
|
}
|
2019-08-11 15:45:43 +00:00
|
|
|
|
2019-08-16 11:20:28 +00:00
|
|
|
var addrs []api.Multiaddr
|
2019-08-26 12:57:17 +00:00
|
|
|
mAddrs, err := peer.AddrInfoToP2pAddrs(&peer.AddrInfo{ID: c.id, Addrs: c.host.Addrs()})
|
|
|
|
if err == nil {
|
|
|
|
for _, mAddr := range mAddrs {
|
|
|
|
addrs = append(addrs, api.NewMultiaddrWithValue(mAddr))
|
|
|
|
}
|
2017-01-26 18:59:31 +00:00
|
|
|
}
|
|
|
|
|
2017-11-15 15:23:22 +00:00
|
|
|
peers := []peer.ID{}
|
|
|
|
// This method might get called very early by a remote peer
|
|
|
|
// and might catch us when consensus is not set
|
|
|
|
if c.consensus != nil {
|
2018-06-27 04:03:15 +00:00
|
|
|
peers, _ = c.consensus.Peers(ctx)
|
2017-11-15 15:23:22 +00:00
|
|
|
}
|
2017-11-08 19:04:04 +00:00
|
|
|
|
Fix #787: Connectivity fixes
Currently, unless doing Join() (--bootstrap), we do not connect to any peers on startup.
We however loaded up the peerstore file and Raft will automatically connect
older peers to figure out who is the leader etc. DHT bootstrap, after Raft
was working, did the rest.
For CRDTs we need to connect to people on a normal boot as otherwise, unless
bootstrapping, this does not happen, even if the peerstore contains known peers.
This introduces a number of changes:
* Move peerstore file management back inside the Cluster component, which was
already in charge of saving the peerstore file.
* We keep saving all "known addresses" but we load them with a non permanent
TTL, so that there will be clean up of peers we're not connected to for long.
* "Bootstrap" (connect) to a small number of peers during Cluster component creation.
* Bootstrap the DHT asap after this, so that other cluster components can
initialize with a working peer discovery mechanism.
* CRDT Trust() method will now:
* Protect the trusted Peer ID in the conn manager
* Give top priority in the PeerManager to that Peer (see below)
* Mark addresses as permanent in the Peerstore
The PeerManager now attaches priorities to peers when importing them and is
able to order them according to that priority. The result is that peers with
high priority are saved first in the peerstore file. When we load the peerstore
file, the first entries in it are given the highest priority.
This means that during startup we will connect to "trusted peers" first
(because they have been tagged with priority in the previous run and saved at
the top of the list). Once connected to a small number of peers, we let the
DHT bootstrap process in the background do the rest and discover the network.
All this makes the peerstore file a "bootstrap" list for CRDTs and we will attempt
to connect to peers on that list until some of those connections succeed.
2019-05-23 16:41:33 +00:00
|
|
|
clusterPeerInfos := c.peerManager.PeerInfos(peers)
|
|
|
|
addresses := []api.Multiaddr{}
|
|
|
|
for _, pinfo := range clusterPeerInfos {
|
2019-06-14 10:41:11 +00:00
|
|
|
addrs, err := peer.AddrInfoToP2pAddrs(&pinfo)
|
Fix #787: Connectivity fixes
Currently, unless doing Join() (--bootstrap), we do not connect to any peers on startup.
We however loaded up the peerstore file and Raft will automatically connect
older peers to figure out who is the leader etc. DHT bootstrap, after Raft
was working, did the rest.
For CRDTs we need to connect to people on a normal boot as otherwise, unless
bootstrapping, this does not happen, even if the peerstore contains known peers.
This introduces a number of changes:
* Move peerstore file management back inside the Cluster component, which was
already in charge of saving the peerstore file.
* We keep saving all "known addresses" but we load them with a non permanent
TTL, so that there will be clean up of peers we're not connected to for long.
* "Bootstrap" (connect) to a small number of peers during Cluster component creation.
* Bootstrap the DHT asap after this, so that other cluster components can
initialize with a working peer discovery mechanism.
* CRDT Trust() method will now:
* Protect the trusted Peer ID in the conn manager
* Give top priority in the PeerManager to that Peer (see below)
* Mark addresses as permanent in the Peerstore
The PeerManager now attaches priorities to peers when importing them and is
able to order them according to that priority. The result is that peers with
high priority are saved first in the peerstore file. When we load the peerstore
file, the first entries in it are given the highest priority.
This means that during startup we will connect to "trusted peers" first
(because they have been tagged with priority in the previous run and saved at
the top of the list). Once connected to a small number of peers, we let the
DHT bootstrap process in the background do the rest and discover the network.
All this makes the peerstore file a "bootstrap" list for CRDTs and we will attempt
to connect to peers on that list until some of those connections succeed.
2019-05-23 16:41:33 +00:00
|
|
|
if err != nil {
|
|
|
|
continue
|
|
|
|
}
|
|
|
|
for _, a := range addrs {
|
|
|
|
addresses = append(addresses, api.NewMultiaddrWithValue(a))
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-08-26 12:57:17 +00:00
|
|
|
id := &api.ID{
|
2017-02-15 12:40:08 +00:00
|
|
|
ID: c.id,
|
|
|
|
//PublicKey: c.host.Peerstore().PubKey(c.id),
|
2017-11-10 15:09:45 +00:00
|
|
|
Addresses: addrs,
|
|
|
|
ClusterPeers: peers,
|
Fix #787: Connectivity fixes
Currently, unless doing Join() (--bootstrap), we do not connect to any peers on startup.
We however loaded up the peerstore file and Raft will automatically connect
older peers to figure out who is the leader etc. DHT bootstrap, after Raft
was working, did the rest.
For CRDTs we need to connect to people on a normal boot as otherwise, unless
bootstrapping, this does not happen, even if the peerstore contains known peers.
This introduces a number of changes:
* Move peerstore file management back inside the Cluster component, which was
already in charge of saving the peerstore file.
* We keep saving all "known addresses" but we load them with a non permanent
TTL, so that there will be clean up of peers we're not connected to for long.
* "Bootstrap" (connect) to a small number of peers during Cluster component creation.
* Bootstrap the DHT asap after this, so that other cluster components can
initialize with a working peer discovery mechanism.
* CRDT Trust() method will now:
* Protect the trusted Peer ID in the conn manager
* Give top priority in the PeerManager to that Peer (see below)
* Mark addresses as permanent in the Peerstore
The PeerManager now attaches priorities to peers when importing them and is
able to order them according to that priority. The result is that peers with
high priority are saved first in the peerstore file. When we load the peerstore
file, the first entries in it are given the highest priority.
This means that during startup we will connect to "trusted peers" first
(because they have been tagged with priority in the previous run and saved at
the top of the list). Once connected to a small number of peers, we let the
DHT bootstrap process in the background do the rest and discover the network.
All this makes the peerstore file a "bootstrap" list for CRDTs and we will attempt
to connect to peers on that list until some of those connections succeed.
2019-05-23 16:41:33 +00:00
|
|
|
ClusterPeersAddresses: addresses,
|
2018-12-18 14:44:11 +00:00
|
|
|
Version: version.Version.String(),
|
|
|
|
RPCProtocolVersion: version.RPCProtocol,
|
2019-03-31 04:25:52 +00:00
|
|
|
IPFS: ipfsID,
|
2017-12-01 18:50:13 +00:00
|
|
|
Peername: c.config.Peername,
|
2017-01-24 15:19:23 +00:00
|
|
|
}
|
2019-08-26 12:57:17 +00:00
|
|
|
if err != nil {
|
|
|
|
id.Error = err.Error()
|
|
|
|
}
|
|
|
|
|
|
|
|
return id
|
2017-01-24 15:19:23 +00:00
|
|
|
}
|
|
|
|
|
2017-01-30 12:12:25 +00:00
|
|
|
// PeerAdd adds a new peer to this Cluster.
|
|
|
|
//
|
2018-07-17 10:51:31 +00:00
|
|
|
// For it to work well, the new peer should be discoverable
|
|
|
|
// (part of our peerstore or connected to one of the existing peers)
|
|
|
|
// and reachable. Since PeerAdd allows to add peers which are
|
|
|
|
// not running, or reachable, it is recommended to call Join() from the
|
|
|
|
// new peer instead.
|
|
|
|
//
|
|
|
|
// The new peer ID will be passed to the consensus
|
|
|
|
// component to be added to the peerset.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) PeerAdd(ctx context.Context, pid peer.ID) (*api.ID, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/PeerAdd")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2017-02-02 22:52:06 +00:00
|
|
|
// starting 10 nodes on the same box for testing
|
|
|
|
// causes deadlock and a global lock here
|
|
|
|
// seems to help.
|
2018-01-16 19:57:54 +00:00
|
|
|
c.paMux.Lock()
|
|
|
|
defer c.paMux.Unlock()
|
2018-07-17 10:51:31 +00:00
|
|
|
logger.Debugf("peerAdd called with %s", pid.Pretty())
|
2017-01-30 12:12:25 +00:00
|
|
|
|
2019-02-20 14:24:25 +00:00
|
|
|
// Let the consensus layer be aware of this peer
|
2018-06-27 04:03:15 +00:00
|
|
|
err := c.consensus.AddPeer(ctx, pid)
|
2017-11-14 20:04:33 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
2019-02-27 17:04:35 +00:00
|
|
|
id := &api.ID{ID: pid, Error: err.Error()}
|
2017-11-14 20:04:33 +00:00
|
|
|
return id, err
|
|
|
|
}
|
|
|
|
|
2019-05-09 12:19:07 +00:00
|
|
|
logger.Info("Peer added ", pid.Pretty())
|
|
|
|
addedID, err := c.getIDForPeer(ctx, pid)
|
2017-03-23 18:34:33 +00:00
|
|
|
if err != nil {
|
2019-05-09 12:19:07 +00:00
|
|
|
return addedID, err
|
2017-03-23 18:34:33 +00:00
|
|
|
}
|
2019-05-09 12:19:07 +00:00
|
|
|
if !containsPeer(addedID.ClusterPeers, c.id) {
|
|
|
|
addedID.ClusterPeers = append(addedID.ClusterPeers, c.id)
|
2017-11-14 22:54:23 +00:00
|
|
|
}
|
2019-05-09 12:19:07 +00:00
|
|
|
return addedID, nil
|
2017-01-30 12:12:25 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// PeerRemove removes a peer from this Cluster.
|
|
|
|
//
|
2018-07-17 10:51:31 +00:00
|
|
|
// The peer will be removed from the consensus peerset.
|
|
|
|
// This may first trigger repinnings for all content if not disabled.
|
2018-06-27 04:03:15 +00:00
|
|
|
func (c *Cluster) PeerRemove(ctx context.Context, pid peer.ID) error {
|
|
|
|
_, span := trace.StartSpan(ctx, "cluster/PeerRemove")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2017-10-31 10:20:14 +00:00
|
|
|
// We need to repin before removing the peer, otherwise, it won't
|
|
|
|
// be able to submit the pins.
|
|
|
|
logger.Infof("re-allocating all CIDs directly associated to %s", pid)
|
2019-09-06 15:56:00 +00:00
|
|
|
c.vacatePeer(ctx, pid)
|
2017-07-03 15:45:22 +00:00
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
err := c.consensus.RmPeer(ctx, pid)
|
2017-02-02 22:52:06 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
return err
|
|
|
|
}
|
2018-07-17 10:51:31 +00:00
|
|
|
logger.Info("Peer removed ", pid.Pretty())
|
2017-02-02 22:52:06 +00:00
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
2018-07-17 10:51:31 +00:00
|
|
|
// Join adds this peer to an existing cluster by bootstrapping to a
|
|
|
|
// given multiaddress. It works by calling PeerAdd on the destination
|
|
|
|
// cluster and making sure that the new peer is ready to discover and contact
|
|
|
|
// the rest.
|
2018-06-27 04:03:15 +00:00
|
|
|
func (c *Cluster) Join(ctx context.Context, addr ma.Multiaddr) error {
|
|
|
|
_, span := trace.StartSpan(ctx, "cluster/Join")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2017-02-02 22:52:06 +00:00
|
|
|
logger.Debugf("Join(%s)", addr)
|
|
|
|
|
2019-10-31 08:12:14 +00:00
|
|
|
// Add peer to peerstore so we can talk to it
|
|
|
|
pid, err := c.peerManager.ImportPeer(addr, false, peerstore.PermanentAddrTTL)
|
2017-02-02 22:52:06 +00:00
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
2017-02-15 12:40:08 +00:00
|
|
|
if pid == c.id {
|
2017-02-02 22:52:06 +00:00
|
|
|
return nil
|
2017-01-30 12:12:25 +00:00
|
|
|
}
|
2017-02-02 22:52:06 +00:00
|
|
|
|
|
|
|
// Note that PeerAdd() on the remote peer will
|
|
|
|
// figure out what our real address is (obviously not
|
2017-10-30 11:17:39 +00:00
|
|
|
// ListenAddr).
|
2019-02-27 17:04:35 +00:00
|
|
|
var myID api.ID
|
2018-06-27 04:03:15 +00:00
|
|
|
err = c.rpcClient.CallContext(
|
|
|
|
ctx,
|
|
|
|
pid,
|
2017-02-02 22:52:06 +00:00
|
|
|
"Cluster",
|
|
|
|
"PeerAdd",
|
2019-02-27 17:04:35 +00:00
|
|
|
c.id,
|
2018-06-27 04:03:15 +00:00
|
|
|
&myID,
|
|
|
|
)
|
2017-02-02 22:52:06 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
return err
|
2017-01-30 12:12:25 +00:00
|
|
|
}
|
2017-02-02 22:52:06 +00:00
|
|
|
|
2019-05-09 12:19:07 +00:00
|
|
|
// Log a fake but valid metric from the peer we are
|
|
|
|
// contacting. This will signal a CRDT component that
|
|
|
|
// we know that peer since we have metrics for it without
|
|
|
|
// having to wait for the next metric round.
|
2019-06-09 11:14:02 +00:00
|
|
|
if err := c.logPingMetric(ctx, pid); err != nil {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn(err)
|
2019-05-09 12:19:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Broadcast our metrics to the world
|
2019-12-05 14:08:43 +00:00
|
|
|
_, err = c.sendInformersMetrics(ctx)
|
2019-05-09 12:19:07 +00:00
|
|
|
if err != nil {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn(err)
|
2019-05-09 12:19:07 +00:00
|
|
|
}
|
2019-12-05 14:08:43 +00:00
|
|
|
|
2019-05-09 12:19:07 +00:00
|
|
|
_, err = c.sendPingMetric(ctx)
|
|
|
|
if err != nil {
|
2020-03-13 20:40:02 +00:00
|
|
|
logger.Warn(err)
|
2019-05-09 12:19:07 +00:00
|
|
|
}
|
|
|
|
|
2018-07-17 10:51:31 +00:00
|
|
|
// We need to trigger a DHT bootstrap asap for this peer to not be
|
|
|
|
// lost if the peer it bootstrapped to goes down. We do this manually
|
|
|
|
// by triggering 1 round of bootstrap in the background.
|
|
|
|
// Note that our regular bootstrap process is still running in the
|
|
|
|
// background since we created the cluster.
|
2020-03-13 20:40:02 +00:00
|
|
|
c.wg.Add(1)
|
2018-07-17 10:51:31 +00:00
|
|
|
go func() {
|
2020-03-13 20:40:02 +00:00
|
|
|
defer c.wg.Done()
|
|
|
|
select {
|
|
|
|
case err := <-c.dht.RefreshRoutingTable():
|
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
}
|
|
|
|
return
|
|
|
|
case <-c.ctx.Done():
|
|
|
|
return
|
|
|
|
}
|
2018-07-17 10:51:31 +00:00
|
|
|
}()
|
|
|
|
|
2019-05-09 12:19:07 +00:00
|
|
|
// ConnectSwarms in the background after a while, when we have likely
|
|
|
|
// received some metrics.
|
|
|
|
time.AfterFunc(c.config.MonitorPingInterval, func() {
|
|
|
|
c.ipfs.ConnectSwarms(ctx)
|
|
|
|
})
|
|
|
|
|
2017-02-02 22:52:06 +00:00
|
|
|
// wait for leader and for state to catch up
|
|
|
|
// then sync
|
2018-06-27 04:03:15 +00:00
|
|
|
err = c.consensus.WaitForSync(ctx)
|
2017-02-02 22:52:06 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
return err
|
|
|
|
}
|
2017-11-10 15:22:31 +00:00
|
|
|
|
2019-12-13 11:00:03 +00:00
|
|
|
// Start pinning items in the state that are not on IPFS yet.
|
|
|
|
c.RecoverAllLocal(ctx)
|
2017-02-02 22:52:06 +00:00
|
|
|
|
2017-11-08 19:04:04 +00:00
|
|
|
logger.Infof("%s: joined %s's cluster", c.id.Pretty(), pid.Pretty())
|
2017-01-30 12:12:25 +00:00
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
2019-12-12 20:22:54 +00:00
|
|
|
// StateSync performs maintenance tasks on the global state that require
|
|
|
|
// looping through all the items. It is triggered automatically on
|
|
|
|
// StateSyncInterval. Currently it:
|
|
|
|
// * Sends unpin for expired items for which this peer is "closest"
|
|
|
|
// (skipped for follower peers)
|
2018-06-27 04:03:15 +00:00
|
|
|
func (c *Cluster) StateSync(ctx context.Context) error {
|
|
|
|
_, span := trace.StartSpan(ctx, "cluster/StateSync")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2019-12-12 20:22:54 +00:00
|
|
|
if c.config.FollowerMode {
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
cState, err := c.consensus.State(ctx)
|
2016-12-09 19:54:46 +00:00
|
|
|
if err != nil {
|
2018-05-24 14:40:40 +00:00
|
|
|
return err
|
2016-12-09 19:54:46 +00:00
|
|
|
}
|
2016-12-20 18:51:13 +00:00
|
|
|
|
2019-09-18 16:05:55 +00:00
|
|
|
timeNow := time.Now()
|
2019-02-20 14:24:25 +00:00
|
|
|
clusterPins, err := cState.List(ctx)
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
2018-05-24 14:40:40 +00:00
|
|
|
|
2019-12-12 20:22:54 +00:00
|
|
|
// Only trigger pin operations if we are the closest with respect to
|
|
|
|
// other trusted peers. We cannot know if our peer ID is trusted by
|
|
|
|
// other peers in the Cluster. This assumes yes. Setting FollowerMode
|
|
|
|
// is a way to assume the opposite and skip this completely.
|
|
|
|
trustedPeers, err := c.getTrustedPeers(ctx)
|
|
|
|
if err != nil {
|
|
|
|
return nil
|
2019-09-18 16:05:55 +00:00
|
|
|
}
|
|
|
|
|
2019-12-12 20:22:54 +00:00
|
|
|
checker := distanceChecker{
|
|
|
|
local: c.id,
|
|
|
|
otherPeers: trustedPeers,
|
|
|
|
cache: make(map[peer.ID]distance, len(trustedPeers)+1),
|
2019-09-18 16:05:55 +00:00
|
|
|
}
|
|
|
|
|
2019-12-12 20:22:54 +00:00
|
|
|
// Unpin expired items when we are the closest peer to them.
|
|
|
|
for _, p := range clusterPins {
|
|
|
|
if p.ExpiredAt(timeNow) && checker.isClosest(p.Cid) {
|
|
|
|
logger.Infof("Unpinning %s: pin expired at %s", p.Cid, p.ExpireAt)
|
|
|
|
if _, err := c.Unpin(ctx, p.Cid); err != nil {
|
2019-09-18 16:05:55 +00:00
|
|
|
logger.Error(err)
|
|
|
|
}
|
2016-12-20 18:51:13 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-05-24 14:40:40 +00:00
|
|
|
return nil
|
2016-12-20 18:51:13 +00:00
|
|
|
}
|
|
|
|
|
2017-12-01 11:56:26 +00:00
|
|
|
// StatusAll returns the GlobalPinInfo for all tracked Cids in all peers.
|
|
|
|
// If an error happens, the slice will contain as much information as
|
|
|
|
// could be fetched from other peers.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) StatusAll(ctx context.Context) ([]*api.GlobalPinInfo, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/StatusAll")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2019-05-04 20:36:10 +00:00
|
|
|
return c.globalPinInfoSlice(ctx, "PinTracker", "StatusAll")
|
2016-12-23 18:35:37 +00:00
|
|
|
}
|
|
|
|
|
2017-12-01 11:56:26 +00:00
|
|
|
// StatusAllLocal returns the PinInfo for all the tracked Cids in this peer.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) StatusAllLocal(ctx context.Context) []*api.PinInfo {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/StatusAllLocal")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
return c.tracker.StatusAll(ctx)
|
2017-12-01 11:56:26 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Status returns the GlobalPinInfo for a given Cid as fetched from all
|
|
|
|
// current peers. If an error happens, the GlobalPinInfo should contain
|
|
|
|
// as much information as could be fetched from the other peers.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) Status(ctx context.Context, h cid.Cid) (*api.GlobalPinInfo, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/Status")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2019-05-04 20:36:10 +00:00
|
|
|
return c.globalPinInfoCid(ctx, "PinTracker", "Status", h)
|
2016-12-23 18:35:37 +00:00
|
|
|
}
|
|
|
|
|
2017-12-01 11:56:26 +00:00
|
|
|
// StatusLocal returns this peer's PinInfo for a given Cid.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) StatusLocal(ctx context.Context, h cid.Cid) *api.PinInfo {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/StatusLocal")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
return c.tracker.Status(ctx, h)
|
2017-12-01 11:56:26 +00:00
|
|
|
}
|
|
|
|
|
2018-08-06 12:31:45 +00:00
|
|
|
// used for RecoverLocal and SyncLocal.
|
|
|
|
func (c *Cluster) localPinInfoOp(
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx context.Context,
|
2018-09-22 01:00:10 +00:00
|
|
|
h cid.Cid,
|
2019-02-27 17:04:35 +00:00
|
|
|
f func(context.Context, cid.Cid) (*api.PinInfo, error),
|
|
|
|
) (pInfo *api.PinInfo, err error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/localPinInfoOp")
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
cids, err := c.cidsFromMetaPin(ctx, h)
|
2017-01-25 17:07:19 +00:00
|
|
|
if err != nil {
|
2019-02-27 17:04:35 +00:00
|
|
|
return nil, err
|
2016-12-15 18:08:46 +00:00
|
|
|
}
|
2018-08-06 12:31:45 +00:00
|
|
|
|
|
|
|
for _, ci := range cids {
|
2018-06-27 04:03:15 +00:00
|
|
|
pInfo, err = f(ctx, ci)
|
2018-08-06 12:31:45 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error("tracker.SyncCid() returned with error: ", err)
|
|
|
|
logger.Error("Is the ipfs daemon running?")
|
|
|
|
break
|
|
|
|
}
|
|
|
|
}
|
|
|
|
// return the last pInfo/err, should be the root Cid if everything ok
|
2017-01-25 18:38:23 +00:00
|
|
|
return pInfo, err
|
2018-08-06 12:31:45 +00:00
|
|
|
|
|
|
|
}
|
|
|
|
|
2019-12-13 11:00:03 +00:00
|
|
|
// RecoverAll triggers a RecoverAllLocal operation on all peers.
|
2019-08-21 05:49:07 +00:00
|
|
|
func (c *Cluster) RecoverAll(ctx context.Context) ([]*api.GlobalPinInfo, error) {
|
|
|
|
_, span := trace.StartSpan(ctx, "cluster/RecoverAll")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
return c.globalPinInfoSlice(ctx, "Cluster", "RecoverAllLocal")
|
|
|
|
}
|
|
|
|
|
2017-12-01 11:56:26 +00:00
|
|
|
// RecoverAllLocal triggers a RecoverLocal operation for all Cids tracked
|
2017-11-30 00:53:31 +00:00
|
|
|
// by this peer.
|
2019-09-06 15:56:00 +00:00
|
|
|
//
|
|
|
|
// Recover operations ask IPFS to pin or unpin items in error state. Recover
|
|
|
|
// is faster than calling Pin on the same CID as it avoids committing an
|
|
|
|
// identical pin to the consensus layer.
|
2019-12-13 11:00:03 +00:00
|
|
|
//
|
|
|
|
// RecoverAllLocal is called automatically every PinRecoverInterval.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) RecoverAllLocal(ctx context.Context) ([]*api.PinInfo, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/RecoverAllLocal")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
return c.tracker.RecoverAll(ctx)
|
2017-11-30 00:53:31 +00:00
|
|
|
}
|
|
|
|
|
2017-01-25 18:38:23 +00:00
|
|
|
// Recover triggers a recover operation for a given Cid in all
|
2017-01-26 18:59:31 +00:00
|
|
|
// cluster peers.
|
2019-09-06 15:56:00 +00:00
|
|
|
//
|
|
|
|
// Recover operations ask IPFS to pin or unpin items in error state. Recover
|
|
|
|
// is faster than calling Pin on the same CID as it avoids committing an
|
|
|
|
// identical pin to the consensus layer.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) Recover(ctx context.Context, h cid.Cid) (*api.GlobalPinInfo, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/Recover")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2019-05-04 20:36:10 +00:00
|
|
|
return c.globalPinInfoCid(ctx, "PinTracker", "Recover", h)
|
2016-12-15 18:08:46 +00:00
|
|
|
}
|
|
|
|
|
2017-12-01 11:56:26 +00:00
|
|
|
// RecoverLocal triggers a recover operation for a given Cid in this peer only.
|
|
|
|
// It returns the updated PinInfo, after recovery.
|
2019-09-06 15:56:00 +00:00
|
|
|
//
|
|
|
|
// Recover operations ask IPFS to pin or unpin items in error state. Recover
|
|
|
|
// is faster than calling Pin on the same CID as it avoids committing an
|
|
|
|
// identical pin to the consensus layer.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) RecoverLocal(ctx context.Context, h cid.Cid) (pInfo *api.PinInfo, err error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/RecoverLocal")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
return c.localPinInfoOp(ctx, h, c.tracker.Recover)
|
2017-12-01 11:56:26 +00:00
|
|
|
}
|
|
|
|
|
2016-12-15 18:08:46 +00:00
|
|
|
// Pins returns the list of Cids managed by Cluster and which are part
|
|
|
|
// of the current global state. This is the source of truth as to which
|
2017-04-06 02:27:02 +00:00
|
|
|
// pins are managed and their allocation, but does not indicate if
|
|
|
|
// the item is successfully pinned. For that, use StatusAll().
|
2019-02-20 14:24:25 +00:00
|
|
|
func (c *Cluster) Pins(ctx context.Context) ([]*api.Pin, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/Pins")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
cState, err := c.consensus.State(ctx)
|
2016-12-15 18:08:46 +00:00
|
|
|
if err != nil {
|
2017-02-01 17:16:09 +00:00
|
|
|
logger.Error(err)
|
2019-02-20 14:24:25 +00:00
|
|
|
return nil, err
|
2016-12-15 18:08:46 +00:00
|
|
|
}
|
2018-06-27 04:03:15 +00:00
|
|
|
return cState.List(ctx)
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
2017-04-06 02:27:02 +00:00
|
|
|
// PinGet returns information for a single Cid managed by Cluster.
|
|
|
|
// The information is obtained from the current global state. The
|
|
|
|
// returned api.Pin provides information about the allocations
|
2018-01-19 12:04:06 +00:00
|
|
|
// assigned for the requested Cid, but does not indicate if
|
2017-04-06 02:27:02 +00:00
|
|
|
// the item is successfully pinned. For that, use Status(). PinGet
|
|
|
|
// returns an error if the given Cid is not part of the global state.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) PinGet(ctx context.Context, h cid.Cid) (*api.Pin, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/PinGet")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
st, err := c.consensus.State(ctx)
|
2018-08-06 12:31:45 +00:00
|
|
|
if err != nil {
|
2019-02-27 17:04:35 +00:00
|
|
|
return nil, err
|
2018-08-06 12:31:45 +00:00
|
|
|
}
|
2019-02-20 14:24:25 +00:00
|
|
|
pin, err := st.Get(ctx, h)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
2017-04-06 02:27:02 +00:00
|
|
|
}
|
|
|
|
return pin, nil
|
|
|
|
}
|
|
|
|
|
2016-12-02 18:33:39 +00:00
|
|
|
// Pin makes the cluster Pin a Cid. This implies adding the Cid
|
|
|
|
// to the IPFS Cluster peers shared-state. Depending on the cluster
|
2016-12-16 11:40:28 +00:00
|
|
|
// pinning strategy, the PinTracker may then request the IPFS daemon
|
2017-01-23 13:01:49 +00:00
|
|
|
// to pin the Cid.
|
2016-12-02 18:33:39 +00:00
|
|
|
//
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
// Pin returns the Pin as stored in the global state (with the given
|
|
|
|
// allocations and an error if the operation could not be persisted. Pin does
|
|
|
|
// not reflect the success or failure of underlying IPFS daemon pinning
|
|
|
|
// operations which happen in async fashion.
|
2018-03-09 22:13:52 +00:00
|
|
|
//
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
// If the options UserAllocations are non-empty then these peers are pinned
|
|
|
|
// with priority over other peers in the cluster. If the max repl factor is
|
|
|
|
// less than the size of the specified peerset then peers are chosen from this
|
|
|
|
// set in allocation order. If the minimum repl factor is greater than the
|
|
|
|
// size of this set then the remaining peers are allocated in order from the
|
|
|
|
// rest of the cluster. Priority allocations are best effort. If any priority
|
|
|
|
// peers are unavailable then Pin will simply allocate from the rest of the
|
|
|
|
// cluster.
|
2019-07-12 14:40:29 +00:00
|
|
|
//
|
|
|
|
// If the Update option is set, the pin options (including allocations) will
|
|
|
|
// be copied from an existing one. This is equivalent to running PinUpdate.
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
func (c *Cluster) Pin(ctx context.Context, h cid.Cid, opts api.PinOptions) (*api.Pin, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/Pin")
|
|
|
|
defer span.End()
|
2019-07-30 17:42:26 +00:00
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
pin := api.PinWithOpts(h, opts)
|
2019-07-12 14:40:29 +00:00
|
|
|
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
result, _, err := c.pin(ctx, pin, []peer.ID{})
|
|
|
|
return result, err
|
2017-07-03 15:45:22 +00:00
|
|
|
}
|
|
|
|
|
2018-07-24 13:22:29 +00:00
|
|
|
// sets the default replication factor in a pin when it's set to 0
|
|
|
|
func (c *Cluster) setupReplicationFactor(pin *api.Pin) error {
|
2018-07-04 16:30:24 +00:00
|
|
|
rplMin := pin.ReplicationFactorMin
|
|
|
|
rplMax := pin.ReplicationFactorMax
|
|
|
|
if rplMin == 0 {
|
|
|
|
rplMin = c.config.ReplicationFactorMin
|
|
|
|
pin.ReplicationFactorMin = rplMin
|
|
|
|
}
|
|
|
|
if rplMax == 0 {
|
|
|
|
rplMax = c.config.ReplicationFactorMax
|
|
|
|
pin.ReplicationFactorMax = rplMax
|
|
|
|
}
|
|
|
|
|
2018-07-24 13:22:29 +00:00
|
|
|
return isReplicationFactorValid(rplMin, rplMax)
|
|
|
|
}
|
2018-04-30 16:00:22 +00:00
|
|
|
|
2018-07-24 13:22:29 +00:00
|
|
|
// basic checks on the pin type to check it's well-formed.
|
|
|
|
func checkPinType(pin *api.Pin) error {
|
2018-03-18 19:29:02 +00:00
|
|
|
switch pin.Type {
|
|
|
|
case api.DataType:
|
2019-02-27 17:04:35 +00:00
|
|
|
if pin.Reference != nil {
|
2018-03-18 19:29:02 +00:00
|
|
|
return errors.New("data pins should not reference other pins")
|
|
|
|
}
|
|
|
|
case api.ShardType:
|
2018-07-04 16:30:24 +00:00
|
|
|
if pin.MaxDepth != 1 {
|
|
|
|
return errors.New("must pin shards go depth 1")
|
2018-03-18 19:29:02 +00:00
|
|
|
}
|
2018-07-24 13:22:29 +00:00
|
|
|
// FIXME: indirect shard pins could have max-depth 2
|
|
|
|
// FIXME: repinning a shard type will overwrite replication
|
|
|
|
// factor from previous:
|
|
|
|
// if existing.ReplicationFactorMin != rplMin ||
|
|
|
|
// existing.ReplicationFactorMax != rplMax {
|
|
|
|
// return errors.New("shard update with wrong repl factors")
|
2018-07-23 13:56:46 +00:00
|
|
|
//}
|
2018-07-04 16:30:24 +00:00
|
|
|
case api.ClusterDAGType:
|
|
|
|
if pin.MaxDepth != 0 {
|
2018-03-18 19:29:02 +00:00
|
|
|
return errors.New("must pin roots directly")
|
|
|
|
}
|
2019-02-27 17:04:35 +00:00
|
|
|
if pin.Reference == nil {
|
2018-07-24 12:21:29 +00:00
|
|
|
return errors.New("clusterDAG pins should reference a Meta pin")
|
2018-03-18 19:29:02 +00:00
|
|
|
}
|
|
|
|
case api.MetaType:
|
2018-07-04 16:30:24 +00:00
|
|
|
if pin.Allocations != nil && len(pin.Allocations) != 0 {
|
2018-03-18 19:29:02 +00:00
|
|
|
return errors.New("meta pin should not specify allocations")
|
|
|
|
}
|
2019-02-27 17:04:35 +00:00
|
|
|
if pin.Reference == nil {
|
2018-07-24 12:21:29 +00:00
|
|
|
return errors.New("metaPins should reference a ClusterDAG")
|
2018-04-20 15:41:15 +00:00
|
|
|
}
|
|
|
|
|
2018-03-18 19:29:02 +00:00
|
|
|
default:
|
|
|
|
return errors.New("unrecognized pin type")
|
|
|
|
}
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
2018-07-24 13:22:29 +00:00
|
|
|
// setupPin ensures that the Pin object is fit for pinning. We check
|
|
|
|
// and set the replication factors and ensure that the pinType matches the
|
|
|
|
// metadata consistently.
|
2018-06-27 04:03:15 +00:00
|
|
|
func (c *Cluster) setupPin(ctx context.Context, pin *api.Pin) error {
|
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/setupPin")
|
|
|
|
defer span.End()
|
|
|
|
|
2018-07-24 13:22:29 +00:00
|
|
|
err := c.setupReplicationFactor(pin)
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
2019-11-05 09:11:50 +00:00
|
|
|
if !pin.ExpireAt.IsZero() && pin.ExpireAt.Before(time.Now()) {
|
|
|
|
return errors.New("pin.ExpireAt set before current time")
|
|
|
|
}
|
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
existing, err := c.PinGet(ctx, pin.Cid)
|
2019-02-20 14:24:25 +00:00
|
|
|
if err != nil && err != state.ErrNotFound {
|
|
|
|
return err
|
2018-07-24 13:22:29 +00:00
|
|
|
}
|
2019-02-20 14:24:25 +00:00
|
|
|
|
|
|
|
if existing != nil && existing.Type != pin.Type {
|
|
|
|
msg := "cannot repin CID with different tracking method, "
|
|
|
|
msg += "clear state with pin rm to proceed. "
|
|
|
|
msg += "New: %s. Was: %s"
|
|
|
|
return fmt.Errorf(msg, pin.Type, existing.Type)
|
|
|
|
}
|
|
|
|
|
2018-07-24 13:22:29 +00:00
|
|
|
return checkPinType(pin)
|
|
|
|
}
|
|
|
|
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
// pin performs the actual pinning and supports a blacklist to be able to
|
|
|
|
// evacuate a node and returns the pin object that it tried to pin, whether
|
|
|
|
// the pin was submitted to the consensus layer or skipped (due to error or to
|
|
|
|
// the fact that it was already valid) and error.
|
2019-07-12 14:40:29 +00:00
|
|
|
//
|
|
|
|
// This is the method called by the Cluster.Pin RPC endpoint.
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
func (c *Cluster) pin(
|
|
|
|
ctx context.Context,
|
|
|
|
pin *api.Pin,
|
|
|
|
blacklist []peer.ID,
|
|
|
|
) (*api.Pin, bool, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/pin")
|
|
|
|
defer span.End()
|
|
|
|
|
2019-07-30 17:42:26 +00:00
|
|
|
if c.config.FollowerMode {
|
|
|
|
return nil, false, errFollowerMode
|
|
|
|
}
|
|
|
|
|
2018-09-22 01:00:10 +00:00
|
|
|
if pin.Cid == cid.Undef {
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, false, errors.New("bad pin object")
|
2018-03-18 19:29:02 +00:00
|
|
|
}
|
2018-07-04 16:30:24 +00:00
|
|
|
|
2019-07-12 14:40:29 +00:00
|
|
|
// Handle pin updates when the option is set
|
|
|
|
if update := pin.PinUpdate; update != cid.Undef && !update.Equals(pin.Cid) {
|
|
|
|
pin, err := c.PinUpdate(ctx, update, pin.Cid, pin.PinOptions)
|
|
|
|
return pin, true, err
|
|
|
|
}
|
|
|
|
|
2018-07-04 16:30:24 +00:00
|
|
|
// setup pin might produce some side-effects to our pin
|
2019-02-27 17:04:35 +00:00
|
|
|
err := c.setupPin(ctx, pin)
|
2018-03-18 19:29:02 +00:00
|
|
|
if err != nil {
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, false, err
|
2018-03-18 19:29:02 +00:00
|
|
|
}
|
|
|
|
if pin.Type == api.MetaType {
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, true, c.consensus.LogPin(ctx, pin)
|
2018-03-18 19:29:02 +00:00
|
|
|
}
|
|
|
|
|
2019-09-06 15:56:00 +00:00
|
|
|
// We did not change ANY options and the pin exists so we just repin
|
|
|
|
// what there is without doing new allocations. While this submits
|
|
|
|
// pins to the consensus layer even if they are, this should trigger the
|
|
|
|
// pin tracker and allows users to get re-pin operations by re-adding
|
|
|
|
// without having to use recover, which is naturally expected.
|
|
|
|
existing, err := c.PinGet(ctx, pin.Cid)
|
|
|
|
if err == nil &&
|
|
|
|
pin.PinOptions.Equals(&existing.PinOptions) &&
|
|
|
|
len(blacklist) == 0 {
|
|
|
|
pin = existing
|
2018-03-18 19:29:02 +00:00
|
|
|
}
|
2017-02-13 15:46:53 +00:00
|
|
|
|
2019-09-06 15:56:00 +00:00
|
|
|
// Usually allocations are unset when pinning normally, however, the
|
|
|
|
// allocations may have been preset by the adder in which case they
|
|
|
|
// need to be respected. Whenever allocations are set. We don't
|
2019-12-16 12:38:17 +00:00
|
|
|
// re-allocate. repinFromPeer() unsets allocations for this reason.
|
|
|
|
// allocate() will check which peers are currently allocated
|
|
|
|
// and try to respect them.
|
2019-09-06 15:56:00 +00:00
|
|
|
if len(pin.Allocations) == 0 {
|
|
|
|
allocs, err := c.allocate(
|
|
|
|
ctx,
|
|
|
|
pin.Cid,
|
|
|
|
pin.ReplicationFactorMin,
|
|
|
|
pin.ReplicationFactorMax,
|
|
|
|
blacklist,
|
|
|
|
pin.UserAllocations,
|
|
|
|
)
|
|
|
|
if err != nil {
|
|
|
|
return pin, false, err
|
|
|
|
}
|
|
|
|
pin.Allocations = allocs
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
2018-01-16 10:21:34 +00:00
|
|
|
|
|
|
|
if len(pin.Allocations) == 0 {
|
2019-02-20 14:24:25 +00:00
|
|
|
logger.Infof("pinning %s everywhere:", pin.Cid)
|
2018-01-16 10:21:34 +00:00
|
|
|
} else {
|
2019-02-20 14:24:25 +00:00
|
|
|
logger.Infof("pinning %s on %s:", pin.Cid, pin.Allocations)
|
2018-01-16 10:21:34 +00:00
|
|
|
}
|
|
|
|
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, true, c.consensus.LogPin(ctx, pin)
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
// Unpin removes a previously pinned Cid from Cluster. It returns
|
|
|
|
// the global state Pin object as it was stored before removal, or
|
|
|
|
// an error if it was not possible to update the global state.
|
|
|
|
//
|
|
|
|
// Unpin does not reflect the success or failure of underlying IPFS daemon
|
|
|
|
// unpinning operations, which happen in async fashion.
|
|
|
|
func (c *Cluster) Unpin(ctx context.Context, h cid.Cid) (*api.Pin, error) {
|
|
|
|
_, span := trace.StartSpan(ctx, "cluster/Unpin")
|
2018-06-27 04:03:15 +00:00
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
2019-07-30 17:42:26 +00:00
|
|
|
if c.config.FollowerMode {
|
|
|
|
return nil, errFollowerMode
|
|
|
|
}
|
|
|
|
|
2017-02-28 15:01:26 +00:00
|
|
|
logger.Info("IPFS cluster unpinning:", h)
|
2018-06-27 04:03:15 +00:00
|
|
|
pin, err := c.PinGet(ctx, h)
|
2018-03-18 19:29:02 +00:00
|
|
|
if err != nil {
|
2019-02-20 14:24:25 +00:00
|
|
|
return nil, err
|
2018-03-18 19:29:02 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
switch pin.Type {
|
|
|
|
case api.DataType:
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, c.consensus.LogUnpin(ctx, pin)
|
2018-03-18 19:29:02 +00:00
|
|
|
case api.ShardType:
|
2020-02-03 09:30:04 +00:00
|
|
|
err := "cannot unpin a shard directly. Unpin content root CID instead."
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, errors.New(err)
|
2018-03-18 19:29:02 +00:00
|
|
|
case api.MetaType:
|
|
|
|
// Unpin cluster dag and referenced shards
|
|
|
|
err := c.unpinClusterDag(pin)
|
|
|
|
if err != nil {
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, err
|
2018-03-18 19:29:02 +00:00
|
|
|
}
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, c.consensus.LogUnpin(ctx, pin)
|
2018-07-04 16:30:24 +00:00
|
|
|
case api.ClusterDAGType:
|
2018-07-23 13:56:46 +00:00
|
|
|
err := "cannot unpin a Cluster DAG directly. Unpin content root CID instead."
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, errors.New(err)
|
2018-03-18 19:29:02 +00:00
|
|
|
default:
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
return pin, errors.New("unrecognized pin type")
|
2018-03-18 19:29:02 +00:00
|
|
|
}
|
|
|
|
}
|
2017-02-13 15:46:53 +00:00
|
|
|
|
2018-03-18 19:29:02 +00:00
|
|
|
// unpinClusterDag unpins the clusterDAG metadata node and the shard metadata
|
|
|
|
// nodes that it references. It handles the case where multiple parents
|
|
|
|
// reference the same metadata node, only unpinning those nodes without
|
|
|
|
// existing references
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) unpinClusterDag(metaPin *api.Pin) error {
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx, span := trace.StartSpan(c.ctx, "cluster/unpinClusterDag")
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
cids, err := c.cidsFromMetaPin(ctx, metaPin.Cid)
|
2018-03-18 19:29:02 +00:00
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
2018-08-06 12:31:45 +00:00
|
|
|
// TODO: FIXME: potentially unpinning shards which are referenced
|
|
|
|
// by other clusterDAGs.
|
|
|
|
for _, ci := range cids {
|
2018-06-27 04:03:15 +00:00
|
|
|
err = c.consensus.LogUnpin(ctx, api.PinCid(ci))
|
2018-03-18 19:29:02 +00:00
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
}
|
2018-08-06 12:31:45 +00:00
|
|
|
return nil
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
2019-07-12 14:40:29 +00:00
|
|
|
// PinUpdate pins a new CID based on an existing cluster Pin. The allocations
|
|
|
|
// and most pin options (replication factors) are copied from the existing
|
|
|
|
// Pin. The options object can be used to set the Name for the new pin and
|
|
|
|
// might support additional options in the future.
|
|
|
|
//
|
|
|
|
// The from pin is NOT unpinned upon completion. The new pin might take
|
|
|
|
// advantage of efficient pin/update operation on IPFS-side (if the
|
|
|
|
// IPFSConnector supports it - the default one does). This may offer
|
|
|
|
// significant speed when pinning items which are similar to previously pinned
|
|
|
|
// content.
|
|
|
|
func (c *Cluster) PinUpdate(ctx context.Context, from cid.Cid, to cid.Cid, opts api.PinOptions) (*api.Pin, error) {
|
|
|
|
existing, err := c.PinGet(ctx, from)
|
|
|
|
if err != nil { // including when the existing pin is not found
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
// Hector: I am not sure whether it has any point to update something
|
|
|
|
// like a MetaType.
|
|
|
|
if existing.Type != api.DataType {
|
|
|
|
return nil, errors.New("this pin type cannot be updated")
|
|
|
|
}
|
|
|
|
|
|
|
|
existing.Cid = to
|
|
|
|
existing.PinUpdate = from
|
|
|
|
if opts.Name != "" {
|
|
|
|
existing.Name = opts.Name
|
|
|
|
}
|
|
|
|
|
|
|
|
return existing, c.consensus.LogPin(ctx, existing)
|
|
|
|
}
|
|
|
|
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
// PinPath pins an CID resolved from its IPFS Path. It returns the resolved
|
|
|
|
// Pin object.
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
func (c *Cluster) PinPath(ctx context.Context, path string, opts api.PinOptions) (*api.Pin, error) {
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/PinPath")
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
ci, err := c.ipfs.Resolve(ctx, path)
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
if err != nil {
|
2019-02-27 17:04:35 +00:00
|
|
|
return nil, err
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
}
|
|
|
|
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
return c.Pin(ctx, ci, opts)
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// UnpinPath unpins a CID resolved from its IPFS Path. If returns the
|
|
|
|
// previously pinned Pin object.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) UnpinPath(ctx context.Context, path string) (*api.Pin, error) {
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/UnpinPath")
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
ci, err := c.ipfs.Resolve(ctx, path)
|
|
|
|
if err != nil {
|
2019-02-27 17:04:35 +00:00
|
|
|
return nil, err
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
}
|
|
|
|
|
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:
These changes the following Cluster Go API methods:
* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)
Pin and Unpin now return the pinned object.
The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.
The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.
This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.
Most of the changes here are updating tests to the new signatures
* Adapt load-balancing client to new Pin/Unpin signatures
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
* cluster.go: Fix typo
Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 13:39:11 +00:00
|
|
|
return c.Unpin(ctx, ci)
|
Add PinPath/UnpinPath support.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes #450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:07:50 +00:00
|
|
|
}
|
|
|
|
|
2018-04-25 02:12:30 +00:00
|
|
|
// AddFile adds a file to the ipfs daemons of the cluster. The ipfs importer
|
|
|
|
// pipeline is used to DAGify the file. Depending on input parameters this
|
|
|
|
// DAG can be added locally to the calling cluster peer's ipfs repo, or
|
|
|
|
// sharded across the entire cluster.
|
2018-09-22 01:00:10 +00:00
|
|
|
func (c *Cluster) AddFile(reader *multipart.Reader, params *api.AddParams) (cid.Cid, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
// TODO: add context param and tracing
|
2019-07-30 17:42:26 +00:00
|
|
|
|
2018-08-07 18:01:02 +00:00
|
|
|
var dags adder.ClusterDAGService
|
2018-07-04 16:30:24 +00:00
|
|
|
if params.Shard {
|
2018-08-07 18:01:02 +00:00
|
|
|
dags = sharding.New(c.rpcClient, params.PinOptions, nil)
|
2018-07-04 16:30:24 +00:00
|
|
|
} else {
|
2019-09-10 08:56:35 +00:00
|
|
|
dags = single.New(c.rpcClient, params.PinOptions, params.Local)
|
2018-07-04 16:30:24 +00:00
|
|
|
}
|
2018-08-08 19:11:26 +00:00
|
|
|
add := adder.New(dags, params, nil)
|
|
|
|
return add.FromMultipart(c.ctx, reader)
|
2018-04-25 02:12:30 +00:00
|
|
|
}
|
|
|
|
|
2017-11-08 19:04:04 +00:00
|
|
|
// Version returns the current IPFS Cluster version.
|
2016-12-02 18:33:39 +00:00
|
|
|
func (c *Cluster) Version() string {
|
2018-12-18 14:44:11 +00:00
|
|
|
return version.Version.String()
|
2016-12-02 18:33:39 +00:00
|
|
|
}
|
|
|
|
|
2017-11-08 19:04:04 +00:00
|
|
|
// Peers returns the IDs of the members of this Cluster.
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) Peers(ctx context.Context) []*api.ID {
|
2018-06-27 04:03:15 +00:00
|
|
|
_, span := trace.StartSpan(ctx, "cluster/Peers")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
members, err := c.consensus.Peers(ctx)
|
2017-11-08 19:04:04 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
logger.Error("an empty list of peers will be returned")
|
2019-02-27 17:04:35 +00:00
|
|
|
return []*api.ID{}
|
2017-11-08 19:04:04 +00:00
|
|
|
}
|
2018-10-26 23:54:07 +00:00
|
|
|
lenMembers := len(members)
|
2017-11-08 19:04:04 +00:00
|
|
|
|
2019-02-27 17:04:35 +00:00
|
|
|
peers := make([]*api.ID, lenMembers, lenMembers)
|
2017-01-26 18:59:31 +00:00
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
ctxs, cancels := rpcutil.CtxsWithCancel(ctx, lenMembers)
|
2018-05-08 11:10:38 +00:00
|
|
|
defer rpcutil.MultiCancel(cancels)
|
2018-05-01 16:01:59 +00:00
|
|
|
|
|
|
|
errs := c.rpcClient.MultiCall(
|
|
|
|
ctxs,
|
|
|
|
members,
|
|
|
|
"Cluster",
|
|
|
|
"ID",
|
|
|
|
struct{}{},
|
2019-02-27 17:04:35 +00:00
|
|
|
rpcutil.CopyIDsToIfaces(peers),
|
2018-05-01 16:01:59 +00:00
|
|
|
)
|
2017-01-26 18:59:31 +00:00
|
|
|
|
2019-05-09 19:23:49 +00:00
|
|
|
finalPeers := []*api.ID{}
|
|
|
|
|
2017-01-26 18:59:31 +00:00
|
|
|
for i, err := range errs {
|
2019-05-09 19:23:49 +00:00
|
|
|
if err == nil {
|
|
|
|
finalPeers = append(finalPeers, peers[i])
|
|
|
|
continue
|
|
|
|
}
|
|
|
|
|
|
|
|
if rpc.IsAuthorizationError(err) {
|
|
|
|
continue
|
2017-01-26 18:59:31 +00:00
|
|
|
}
|
2019-05-09 19:23:49 +00:00
|
|
|
|
|
|
|
peers[i] = &api.ID{}
|
|
|
|
peers[i].ID = members[i]
|
|
|
|
peers[i].Error = err.Error()
|
2017-01-26 18:59:31 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return peers
|
|
|
|
}
|
|
|
|
|
2019-09-18 16:05:55 +00:00
|
|
|
// getTrustedPeers gives listed of trusted peers except the current peer.
|
|
|
|
func (c *Cluster) getTrustedPeers(ctx context.Context) ([]peer.ID, error) {
|
|
|
|
peers, err := c.consensus.Peers(ctx)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
trustedPeers := make([]peer.ID, 0, len(peers))
|
|
|
|
|
|
|
|
for _, p := range peers {
|
|
|
|
if p == c.id || !c.consensus.IsTrustedPeer(ctx, p) {
|
|
|
|
continue
|
|
|
|
}
|
|
|
|
trustedPeers = append(trustedPeers, p)
|
|
|
|
}
|
|
|
|
|
|
|
|
return trustedPeers, nil
|
|
|
|
}
|
|
|
|
|
2019-11-08 14:15:01 +00:00
|
|
|
func setTrackerStatus(gpin *api.GlobalPinInfo, h cid.Cid, peers []peer.ID, status api.TrackerStatus, t time.Time) {
|
|
|
|
for _, p := range peers {
|
|
|
|
gpin.PeerMap[peer.IDB58Encode(p)] = &api.PinInfo{
|
|
|
|
Cid: h,
|
|
|
|
Peer: p,
|
|
|
|
PeerName: p.String(),
|
|
|
|
Status: status,
|
|
|
|
TS: t,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-05-04 20:36:10 +00:00
|
|
|
func (c *Cluster) globalPinInfoCid(ctx context.Context, comp, method string, h cid.Cid) (*api.GlobalPinInfo, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/globalPinInfoCid")
|
|
|
|
defer span.End()
|
|
|
|
|
2019-11-06 04:44:59 +00:00
|
|
|
gpin := &api.GlobalPinInfo{
|
|
|
|
Cid: h,
|
|
|
|
PeerMap: make(map[string]*api.PinInfo),
|
|
|
|
}
|
2019-11-08 15:19:06 +00:00
|
|
|
// allocated peers, we will contact them through rpc
|
2019-11-05 13:39:15 +00:00
|
|
|
var dests []peer.ID
|
2019-11-08 15:19:06 +00:00
|
|
|
// un-allocated peers, we will set remote status
|
2019-11-08 14:15:01 +00:00
|
|
|
var remote []peer.ID
|
2019-11-08 07:13:01 +00:00
|
|
|
timeNow := time.Now()
|
2019-11-06 04:44:59 +00:00
|
|
|
|
2019-11-08 15:19:06 +00:00
|
|
|
// set dests and remote
|
2019-11-05 13:39:15 +00:00
|
|
|
if c.config.FollowerMode {
|
|
|
|
// during follower mode return status only on self peer
|
|
|
|
dests = []peer.ID{c.host.ID()}
|
2019-11-08 14:15:01 +00:00
|
|
|
remote = []peer.ID{}
|
2019-11-05 13:39:15 +00:00
|
|
|
} else {
|
2019-11-08 14:15:01 +00:00
|
|
|
members, err := c.consensus.Peers(ctx)
|
2019-11-08 07:13:01 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
// If pin is not part of the pinset, mark it unpinned
|
2019-11-05 13:39:15 +00:00
|
|
|
pin, err := c.PinGet(ctx, h)
|
2019-11-08 14:15:01 +00:00
|
|
|
if err == state.ErrNotFound {
|
|
|
|
setTrackerStatus(gpin, h, members, api.TrackerStatusUnpinned, timeNow)
|
2019-11-06 04:44:59 +00:00
|
|
|
return gpin, nil
|
2019-11-05 13:39:15 +00:00
|
|
|
}
|
2019-11-08 14:15:01 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
return nil, err
|
|
|
|
}
|
2019-11-06 04:44:59 +00:00
|
|
|
|
2019-11-05 13:39:15 +00:00
|
|
|
if len(pin.Allocations) > 0 {
|
|
|
|
dests = pin.Allocations
|
2019-11-08 14:15:01 +00:00
|
|
|
remote = peersSubtract(members, dests)
|
2019-11-05 13:39:15 +00:00
|
|
|
} else {
|
2019-11-08 07:13:01 +00:00
|
|
|
dests = members
|
2019-11-08 14:15:01 +00:00
|
|
|
remote = []peer.ID{}
|
2019-11-05 13:39:15 +00:00
|
|
|
}
|
2016-12-20 18:51:13 +00:00
|
|
|
}
|
|
|
|
|
2019-11-08 14:15:01 +00:00
|
|
|
// set status remote on un-allocated peers
|
|
|
|
setTrackerStatus(gpin, h, remote, api.TrackerStatusRemote, timeNow)
|
|
|
|
|
2019-11-05 13:39:15 +00:00
|
|
|
lenDests := len(dests)
|
|
|
|
replies := make([]*api.PinInfo, lenDests, lenDests)
|
|
|
|
ctxs, cancels := rpcutil.CtxsWithCancel(ctx, lenDests)
|
2018-05-08 11:10:38 +00:00
|
|
|
defer rpcutil.MultiCancel(cancels)
|
2018-05-01 16:01:59 +00:00
|
|
|
|
|
|
|
errs := c.rpcClient.MultiCall(
|
|
|
|
ctxs,
|
2019-11-05 13:39:15 +00:00
|
|
|
dests,
|
2019-05-04 20:36:10 +00:00
|
|
|
comp,
|
2018-05-01 16:01:59 +00:00
|
|
|
method,
|
2019-02-27 17:04:35 +00:00
|
|
|
h,
|
|
|
|
rpcutil.CopyPinInfoToIfaces(replies),
|
2018-05-01 16:01:59 +00:00
|
|
|
)
|
2017-02-08 17:04:08 +00:00
|
|
|
|
2019-02-27 17:04:35 +00:00
|
|
|
for i, r := range replies {
|
2017-06-21 12:16:28 +00:00
|
|
|
e := errs[i]
|
|
|
|
|
|
|
|
// No error. Parse and continue
|
|
|
|
if e == nil {
|
2019-11-05 13:39:15 +00:00
|
|
|
gpin.PeerMap[peer.IDB58Encode(dests[i])] = r
|
2017-06-21 12:16:28 +00:00
|
|
|
continue
|
|
|
|
}
|
|
|
|
|
2019-05-09 19:23:49 +00:00
|
|
|
if rpc.IsAuthorizationError(e) {
|
|
|
|
logger.Debug("rpc auth error:", e)
|
|
|
|
continue
|
|
|
|
}
|
|
|
|
|
2017-06-21 12:16:28 +00:00
|
|
|
// Deal with error cases (err != nil): wrap errors in PinInfo
|
2019-11-05 13:39:15 +00:00
|
|
|
logger.Errorf("%s: error in broadcast response from %s: %s ", c.id, dests[i], e)
|
|
|
|
gpin.PeerMap[peer.IDB58Encode(dests[i])] = &api.PinInfo{
|
2019-02-27 17:04:35 +00:00
|
|
|
Cid: h,
|
2019-11-05 13:39:15 +00:00
|
|
|
Peer: dests[i],
|
|
|
|
PeerName: dests[i].String(),
|
2019-02-27 17:04:35 +00:00
|
|
|
Status: api.TrackerStatusClusterError,
|
2019-11-08 07:13:01 +00:00
|
|
|
TS: timeNow,
|
2019-02-27 17:04:35 +00:00
|
|
|
Error: e.Error(),
|
2016-12-20 18:51:13 +00:00
|
|
|
}
|
|
|
|
}
|
2016-12-28 15:25:24 +00:00
|
|
|
|
2019-10-29 14:48:16 +00:00
|
|
|
return gpin, nil
|
2016-12-20 18:51:13 +00:00
|
|
|
}
|
|
|
|
|
2019-05-04 20:36:10 +00:00
|
|
|
func (c *Cluster) globalPinInfoSlice(ctx context.Context, comp, method string) ([]*api.GlobalPinInfo, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/globalPinInfoSlice")
|
|
|
|
defer span.End()
|
|
|
|
|
2019-02-27 17:04:35 +00:00
|
|
|
infos := make([]*api.GlobalPinInfo, 0)
|
|
|
|
fullMap := make(map[cid.Cid]*api.GlobalPinInfo)
|
2016-12-20 18:51:13 +00:00
|
|
|
|
2019-07-30 17:42:26 +00:00
|
|
|
var members []peer.ID
|
|
|
|
var err error
|
|
|
|
if c.config.FollowerMode {
|
|
|
|
members = []peer.ID{c.host.ID()}
|
|
|
|
} else {
|
|
|
|
members, err = c.consensus.Peers(ctx)
|
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
return nil, err
|
|
|
|
}
|
2017-11-08 19:04:04 +00:00
|
|
|
}
|
2018-10-26 23:54:07 +00:00
|
|
|
lenMembers := len(members)
|
2017-11-08 19:04:04 +00:00
|
|
|
|
2019-02-27 17:04:35 +00:00
|
|
|
replies := make([][]*api.PinInfo, lenMembers, lenMembers)
|
2018-05-01 16:01:59 +00:00
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
ctxs, cancels := rpcutil.CtxsWithCancel(ctx, lenMembers)
|
2018-05-08 11:10:38 +00:00
|
|
|
defer rpcutil.MultiCancel(cancels)
|
2018-05-01 16:01:59 +00:00
|
|
|
|
|
|
|
errs := c.rpcClient.MultiCall(
|
|
|
|
ctxs,
|
|
|
|
members,
|
2019-05-04 20:36:10 +00:00
|
|
|
comp,
|
2018-05-01 16:01:59 +00:00
|
|
|
method,
|
|
|
|
struct{}{},
|
2019-02-27 17:04:35 +00:00
|
|
|
rpcutil.CopyPinInfoSliceToIfaces(replies),
|
2018-05-01 16:01:59 +00:00
|
|
|
)
|
2016-12-20 18:51:13 +00:00
|
|
|
|
2019-02-27 17:04:35 +00:00
|
|
|
mergePins := func(pins []*api.PinInfo) {
|
|
|
|
for _, p := range pins {
|
2019-02-27 18:43:29 +00:00
|
|
|
if p == nil {
|
|
|
|
continue
|
|
|
|
}
|
2019-02-27 17:04:35 +00:00
|
|
|
item, ok := fullMap[p.Cid]
|
2016-12-20 18:51:13 +00:00
|
|
|
if !ok {
|
2019-02-27 17:04:35 +00:00
|
|
|
fullMap[p.Cid] = &api.GlobalPinInfo{
|
2017-02-08 17:04:08 +00:00
|
|
|
Cid: p.Cid,
|
2019-02-27 17:04:35 +00:00
|
|
|
PeerMap: map[string]*api.PinInfo{
|
|
|
|
peer.IDB58Encode(p.Peer): p,
|
2016-12-20 18:51:13 +00:00
|
|
|
},
|
|
|
|
}
|
|
|
|
} else {
|
2019-02-27 17:04:35 +00:00
|
|
|
item.PeerMap[peer.IDB58Encode(p.Peer)] = p
|
2016-12-20 18:51:13 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-01-24 00:09:27 +00:00
|
|
|
erroredPeers := make(map[peer.ID]string)
|
2016-12-23 18:35:37 +00:00
|
|
|
for i, r := range replies {
|
2017-01-25 17:07:19 +00:00
|
|
|
if e := errs[i]; e != nil { // This error must come from not being able to contact that cluster member
|
2019-05-09 19:23:49 +00:00
|
|
|
if rpc.IsAuthorizationError(e) {
|
|
|
|
logger.Debug("rpc auth error", e)
|
|
|
|
continue
|
|
|
|
}
|
2017-02-15 12:40:08 +00:00
|
|
|
logger.Errorf("%s: error in broadcast response from %s: %s ", c.id, members[i], e)
|
2017-01-24 00:09:27 +00:00
|
|
|
erroredPeers[members[i]] = e.Error()
|
2017-01-23 23:52:42 +00:00
|
|
|
} else {
|
|
|
|
mergePins(r)
|
2016-12-20 18:51:13 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-01-24 00:09:27 +00:00
|
|
|
// Merge any errors
|
|
|
|
for p, msg := range erroredPeers {
|
2019-02-27 17:04:35 +00:00
|
|
|
for c := range fullMap {
|
|
|
|
fullMap[c].PeerMap[peer.IDB58Encode(p)] = &api.PinInfo{
|
2017-02-08 17:04:08 +00:00
|
|
|
Cid: c,
|
2017-01-24 00:09:27 +00:00
|
|
|
Peer: p,
|
2017-02-08 17:04:08 +00:00
|
|
|
Status: api.TrackerStatusClusterError,
|
2017-01-24 00:09:27 +00:00
|
|
|
TS: time.Now(),
|
|
|
|
Error: msg,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-12-20 18:51:13 +00:00
|
|
|
for _, v := range fullMap {
|
|
|
|
infos = append(infos, v)
|
|
|
|
}
|
|
|
|
|
2017-01-23 23:52:42 +00:00
|
|
|
return infos, nil
|
2016-12-20 18:51:13 +00:00
|
|
|
}
|
2017-01-23 19:29:05 +00:00
|
|
|
|
2019-02-27 17:04:35 +00:00
|
|
|
func (c *Cluster) getIDForPeer(ctx context.Context, pid peer.ID) (*api.ID, error) {
|
2018-06-27 04:03:15 +00:00
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/getIDForPeer")
|
|
|
|
defer span.End()
|
|
|
|
|
2019-02-27 17:04:35 +00:00
|
|
|
var id api.ID
|
2018-06-27 04:03:15 +00:00
|
|
|
err := c.rpcClient.CallContext(
|
|
|
|
ctx,
|
|
|
|
pid,
|
|
|
|
"Cluster",
|
|
|
|
"ID",
|
|
|
|
struct{}{},
|
2019-02-27 17:04:35 +00:00
|
|
|
&id,
|
2018-06-27 04:03:15 +00:00
|
|
|
)
|
2017-02-02 22:52:06 +00:00
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
2019-02-27 17:04:35 +00:00
|
|
|
id.ID = pid
|
2017-02-02 22:52:06 +00:00
|
|
|
id.Error = err.Error()
|
2017-01-23 19:29:05 +00:00
|
|
|
}
|
2019-02-27 17:04:35 +00:00
|
|
|
return &id, err
|
2017-01-23 19:29:05 +00:00
|
|
|
}
|
2017-02-13 15:46:53 +00:00
|
|
|
|
2018-08-06 12:31:45 +00:00
|
|
|
// cidsFromMetaPin expands a meta-pin and returns a list of Cids that
|
|
|
|
// Cluster handles for it: the ShardPins, the ClusterDAG and the MetaPin, in
|
|
|
|
// that order (the MetaPin is the last element).
|
|
|
|
// It returns a slice with only the given Cid if it's not a known Cid or not a
|
|
|
|
// MetaPin.
|
2018-06-27 04:03:15 +00:00
|
|
|
func (c *Cluster) cidsFromMetaPin(ctx context.Context, h cid.Cid) ([]cid.Cid, error) {
|
|
|
|
ctx, span := trace.StartSpan(ctx, "cluster/cidsFromMetaPin")
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
cState, err := c.consensus.State(ctx)
|
2018-08-06 12:31:45 +00:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2018-09-22 01:00:10 +00:00
|
|
|
list := []cid.Cid{h}
|
2018-08-06 12:31:45 +00:00
|
|
|
|
2019-02-20 14:24:25 +00:00
|
|
|
pin, err := cState.Get(ctx, h)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
if pin == nil {
|
2018-08-06 12:31:45 +00:00
|
|
|
return list, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
if pin.Type != api.MetaType {
|
|
|
|
return list, nil
|
|
|
|
}
|
|
|
|
|
2019-02-27 17:04:35 +00:00
|
|
|
if pin.Reference == nil {
|
2019-02-27 20:22:36 +00:00
|
|
|
return nil, errors.New("metaPin.Reference is unset")
|
2019-02-27 17:04:35 +00:00
|
|
|
}
|
|
|
|
list = append([]cid.Cid{*pin.Reference}, list...)
|
|
|
|
clusterDagPin, err := c.PinGet(ctx, *pin.Reference)
|
2018-08-06 12:31:45 +00:00
|
|
|
if err != nil {
|
|
|
|
return list, fmt.Errorf("could not get clusterDAG pin from state. Malformed pin?: %s", err)
|
|
|
|
}
|
|
|
|
|
2018-06-27 04:03:15 +00:00
|
|
|
clusterDagBlock, err := c.ipfs.BlockGet(ctx, clusterDagPin.Cid)
|
2018-08-06 12:31:45 +00:00
|
|
|
if err != nil {
|
|
|
|
return list, fmt.Errorf("error reading clusterDAG block from ipfs: %s", err)
|
|
|
|
}
|
|
|
|
|
|
|
|
clusterDagNode, err := sharding.CborDataToNode(clusterDagBlock, "cbor")
|
|
|
|
if err != nil {
|
|
|
|
return list, fmt.Errorf("error parsing clusterDAG block: %s", err)
|
|
|
|
}
|
|
|
|
for _, l := range clusterDagNode.Links() {
|
2018-09-22 01:00:10 +00:00
|
|
|
list = append([]cid.Cid{l.Cid}, list...)
|
2018-08-06 12:31:45 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return list, nil
|
|
|
|
}
|
|
|
|
|
2017-11-14 22:54:23 +00:00
|
|
|
// diffPeers returns the peerIDs added and removed from peers2 in relation to
|
|
|
|
// peers1
|
|
|
|
func diffPeers(peers1, peers2 []peer.ID) (added, removed []peer.ID) {
|
|
|
|
m1 := make(map[peer.ID]struct{})
|
|
|
|
m2 := make(map[peer.ID]struct{})
|
|
|
|
added = make([]peer.ID, 0)
|
|
|
|
removed = make([]peer.ID, 0)
|
|
|
|
if peers1 == nil && peers2 == nil {
|
|
|
|
return
|
|
|
|
}
|
|
|
|
if peers1 == nil {
|
|
|
|
added = peers2
|
|
|
|
return
|
|
|
|
}
|
|
|
|
if peers2 == nil {
|
|
|
|
removed = peers1
|
|
|
|
return
|
|
|
|
}
|
|
|
|
|
|
|
|
for _, p := range peers1 {
|
|
|
|
m1[p] = struct{}{}
|
|
|
|
}
|
|
|
|
for _, p := range peers2 {
|
|
|
|
m2[p] = struct{}{}
|
|
|
|
}
|
2017-12-06 12:45:35 +00:00
|
|
|
for k := range m1 {
|
2017-11-14 22:54:23 +00:00
|
|
|
_, ok := m2[k]
|
|
|
|
if !ok {
|
|
|
|
removed = append(removed, k)
|
|
|
|
}
|
|
|
|
}
|
2017-12-06 12:45:35 +00:00
|
|
|
for k := range m2 {
|
2017-11-14 22:54:23 +00:00
|
|
|
_, ok := m1[k]
|
|
|
|
if !ok {
|
|
|
|
added = append(added, k)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return
|
|
|
|
}
|
2019-10-22 05:40:32 +00:00
|
|
|
|
|
|
|
// RepoGC performs garbage collection sweep on all peers' IPFS repo.
|
|
|
|
func (c *Cluster) RepoGC(ctx context.Context) (*api.GlobalRepoGC, error) {
|
|
|
|
_, span := trace.StartSpan(ctx, "cluster/RepoGC")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
members, err := c.consensus.Peers(ctx)
|
|
|
|
if err != nil {
|
|
|
|
logger.Error(err)
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2019-10-22 09:04:11 +00:00
|
|
|
// to club `RepoGCLocal` responses of all peers into one
|
2019-10-22 05:40:32 +00:00
|
|
|
globalRepoGC := api.GlobalRepoGC{PeerMap: make(map[string]*api.RepoGC)}
|
2019-10-22 09:04:11 +00:00
|
|
|
for _, member := range members {
|
|
|
|
var repoGC api.RepoGC
|
|
|
|
err = c.rpcClient.CallContext(
|
|
|
|
ctx,
|
|
|
|
member,
|
|
|
|
"Cluster",
|
|
|
|
"RepoGCLocal",
|
|
|
|
struct{}{},
|
|
|
|
&repoGC,
|
|
|
|
)
|
|
|
|
if err == nil {
|
|
|
|
globalRepoGC.PeerMap[peer.IDB58Encode(member)] = &repoGC
|
2019-10-22 05:40:32 +00:00
|
|
|
continue
|
|
|
|
}
|
|
|
|
|
2019-10-22 09:04:11 +00:00
|
|
|
if rpc.IsAuthorizationError(err) {
|
|
|
|
logger.Debug("rpc auth error:", err)
|
2019-10-22 05:40:32 +00:00
|
|
|
continue
|
|
|
|
}
|
|
|
|
|
2019-10-22 09:04:11 +00:00
|
|
|
logger.Errorf("%s: error in broadcast response from %s: %s ", c.id, member, err)
|
2019-10-22 05:40:32 +00:00
|
|
|
|
2019-10-22 09:04:11 +00:00
|
|
|
globalRepoGC.PeerMap[peer.IDB58Encode(member)] = &api.RepoGC{
|
|
|
|
Peer: member,
|
|
|
|
Peername: peer.IDB58Encode(member),
|
2019-10-22 05:40:32 +00:00
|
|
|
Keys: []api.IPFSRepoGC{},
|
|
|
|
Error: err.Error(),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return &globalRepoGC, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
// RepoGCLocal performs garbage collection only on the local IPFS deamon.
|
|
|
|
func (c *Cluster) RepoGCLocal(ctx context.Context) (*api.RepoGC, error) {
|
|
|
|
_, span := trace.StartSpan(ctx, "cluster/RepoGCLocal")
|
|
|
|
defer span.End()
|
|
|
|
ctx = trace.NewContext(c.ctx, span)
|
|
|
|
|
|
|
|
resp, err := c.ipfs.RepoGC(ctx)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
resp.Peer = c.id
|
|
|
|
resp.Peername = c.config.Peername
|
|
|
|
return resp, nil
|
|
|
|
}
|