api client test
api test
one sharness test
refactoring of testingData for access from other packages
always wrap files added by cluster
remove unused flag 'progress'
changes to support hidden flag
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
addFile function is now a Cluster method accessed by RPC
residue from attempting to stream responses removed
ipfs-cluster-ctl ls bug fixed
problem with importer/add not printing resolved
new test now checks for this
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
fixed obo error
fix parent cid to be a set per new pin format
finalize now works if there is not data to flush
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
non-recursive cdag pins
priority pins for shards to allocated peer
cdag, meta and shard pin types used
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
1. Refactored importer endpoint, including writing cluster-specific
file adder, to get print info from importer
2. Refactored importer consumption to select equally from
channels of different output signals and manage context
timeouts correctly (only in local add here, sharding to follow)
3. Added output streaming and an error/termination handling protocol
4. Discovered that naive eager response streaming cuts off
reads from request data stream and breaks behavior, for
now all responses come after file ingestion.
5. Added ipfs add style flags (trickle, rawleaves etc.) and
refactored importer endpoint to take in these parameters
to provide identicle behavior to ipfs
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
sharding passes manual tests on single node cluster,
adding the shards of a directory and pinning the
clusterDAG to cluster/ipfs state
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
A sharder config provides a default shard size
getAllocations calls newly exposed rpcs
ipld cbor cluster dag node construction
logic to handle Flushing final shard
logic to initialize and finalize shard sessions
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
Write basic scaffolding to include a sharding component in cluster
Sketch out a high level implementation in pseudo code
Share thoughts on upcoming design challenges
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
RPC call to put a block in ipfs
IPFSConnector method to implement the RPC call
cluster restapi reads from channel and puts
blocks into IPFS via RPC in ctl add handler
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
dependence on dex replaced with ipld-importer subpackage
slight improvement to importer tests and formatting
gx deps update: new raft, ipld format and ipfs added
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
dummy echo server implemented on cluster service api
manual tests with files work
wireshark examination shows transfer-encoding = chunked
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
parse file path arguments to cmdkit.File objects
no support for directories, symlinks or recursive add
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
This uses go-libp2p-kad-dht as routing provider for the Cluster Peers.
This means that:
* A cluster peer can discover other Cluster peers even if they are
not in their peerstore file.
* We remove a bunch of code sending and receiving peers multiaddresses
when a new peer was added to the Cluster.
* PeerAdd now takes an ID and not a multiaddress. We do not need to
ask the new peer which is our external multiaddress nor broadcast
the new multiaddress to everyone. This will fix problems when bootstrapping
a new peer to the Cluster while not all the other peers are online.
* Adding a new peer does not mean to open connections to all peers
anymore. The number of connections will be made according to the DHT
parameters (this is good to have for future work)
The that detecting a peer addition in the watchPeers() function does
no longer mean that we have connected to it or that we know its
multiaddresses. Therefore it's no point to save the peerstore in these
events anymore.
Here a question opens, should we save the peerstore at all, and should we
save multiaddresses only for cluster peers, or for everyone known?
Currently, the peerstore is only updated on clean shutdown,
and it is updated with all the multiaddresses known, and not limited to
peer IDs in the cluster, (because, why not).
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
