1d98538411
This commit fixes #810 and adds block streaming to the final destinations when adding. This should add major performance gains when adding data to clusters. Before, everytime cluster issued a block, it was broadcasted individually to all destinations (new libp2p stream), where it was block/put to IPFS (a single block/put http roundtrip per block). Now, blocks are streamed all the way from the adder module to the ipfs daemon, by making every block as it arrives a single part in a multipart block/put request. Before, block-broadcast needed to wait for all destinations to finish in order to process the next block. Now, buffers allow some destinations to be faster than others while sending and receiving blocks. Before, if a block put request failed to be broadcasted everywhere, an error would happen at that moment. Now, we keep streaming until the end and only then report any errors. The operation succeeds as long as at least one stream finished successfully. Errors block/putting to IPFS will not abort streams. Instead, subsequent blocks are retried with a new request, although the method will return an error when the stream finishes if there were errors at any point. |
||
---|---|---|
.github | ||
adder | ||
allocator/balanced | ||
api | ||
cmd | ||
cmdutils | ||
config | ||
consensus | ||
datastore | ||
docker | ||
informer | ||
ipfsconn/ipfshttp | ||
monitor | ||
observations | ||
pintracker | ||
pstoremgr | ||
rpcutil | ||
sharness | ||
state | ||
test | ||
version | ||
.codeclimate.yml | ||
.codecov.yml | ||
.dockerignore | ||
.gitignore | ||
add_test.go | ||
allocate.go | ||
CHANGELOG.md | ||
cluster_config_test.go | ||
cluster_config.go | ||
cluster_test.go | ||
cluster.go | ||
clusterhost.go | ||
config_test.go | ||
connect_graph.go | ||
CONTRIBUTING.md | ||
COPYRIGHT | ||
docker-compose.yml | ||
Dockerfile | ||
Dockerfile-bundle | ||
Dockerfile-test | ||
go.mod | ||
go.sum | ||
ipfscluster_test.go | ||
ipfscluster.go | ||
LICENSE | ||
LICENSE-APACHE | ||
LICENSE-MIT | ||
logging.go | ||
Makefile | ||
peer_manager_test.go | ||
pnet_test.go | ||
README.md | ||
release.sh | ||
rpc_api.go | ||
rpc_policy.go | ||
util.go |
IPFS Cluster
Automated data availability and redundancy on IPFS
IPFS Cluster provides data orchestration across a swarm of IPFS daemons by allocating, replicating and tracking a global pinset distributed among multiple peers.
It provides:
- A cluster peer application:
ipfs-cluster-service
, to be run along withgo-ipfs
. - A client CLI application:
ipfs-cluster-ctl
, which allows easily interacting with the peer's HTTP API. - An additional "follower" peer application:
ipfs-cluster-follow
, focused on simplifying the process of configuring and running follower peers.
Are you using IPFS Cluster?
Please participate in the IPFS Cluster user registry.
Table of Contents
Documentation
Please visit https://cluster.ipfs.io/documentation/ to access user documentation, guides and any other resources, including detailed download and usage instructions.
News & Roadmap
We regularly post project updates to https://cluster.ipfs.io/news/ .
The most up-to-date Roadmap is available at https://cluster.ipfs.io/roadmap/ .
Install
Instructions for different installation methods (including from source) are available at https://cluster.ipfs.io/download .
Usage
Extensive usage information is provided at https://cluster.ipfs.io/documentation/ , including:
Contribute
PRs accepted. As part of the IPFS project, we have some contribution guidelines.
License
This library is dual-licensed under Apache 2.0 and MIT terms.
© 2020. Protocol Labs, Inc.