592d61b228
When IPFS starts failing or doesn't respond (i.e. during a restart), cluster is likely to start sending requests at very fast rates. i.e. if there are 100k items to be pinned, and pins start failing immediately, cluster will consume the pin queue really fast and it will all be failures. At the same time, ipfs is hammered non-stop until recover, which may make it harder. This commits introduces a rate-limit when requests to IPFS fail. After 10 failed requests, requests will be sent at most at 1req/s rate. Once a requests succeeds, the rate-limit is raised. This should prevent hammering the IPFS daemon, but also increased CPU in cluster as it burns through pinning queues when IPFS is offline, making the situation in machines worse (and emitting way more logs). |
||
---|---|---|
.github | ||
adder | ||
allocator/balanced | ||
api | ||
cmd | ||
cmdutils | ||
config | ||
consensus | ||
datastore | ||
docker | ||
informer | ||
ipfsconn/ipfshttp | ||
monitor | ||
observations | ||
pintracker | ||
pstoremgr | ||
rpcutil | ||
sharness | ||
state | ||
test | ||
version | ||
.codeclimate.yml | ||
.codecov.yml | ||
.dockerignore | ||
.gitignore | ||
add_test.go | ||
allocate.go | ||
CHANGELOG.md | ||
cluster_config_test.go | ||
cluster_config.go | ||
cluster_test.go | ||
cluster.go | ||
clusterhost.go | ||
config_test.go | ||
connect_graph.go | ||
CONTRIBUTING.md | ||
COPYRIGHT | ||
docker-compose.yml | ||
Dockerfile | ||
Dockerfile-bundle | ||
Dockerfile-test | ||
go.mod | ||
go.sum | ||
ipfscluster_test.go | ||
ipfscluster.go | ||
LICENSE | ||
LICENSE-APACHE | ||
LICENSE-MIT | ||
logging.go | ||
Makefile | ||
peer_manager_test.go | ||
pnet_test.go | ||
README.md | ||
release.sh | ||
rpc_api.go | ||
rpc_policy.go | ||
util.go |
IPFS Cluster
Pinset orchestration for IPFS
IPFS Cluster provides data orchestration across a swarm of IPFS daemons by allocating, replicating and tracking a global pinset distributed among multiple peers.
There are 3 different applications:
- A cluster peer application:
ipfs-cluster-service
, to be run along withgo-ipfs
as a sidecar. - A client CLI application:
ipfs-cluster-ctl
, which allows easily interacting with the peer's HTTP API. - An additional "follower" peer application:
ipfs-cluster-follow
, focused on simplifying the process of configuring and running follower peers.
Are you using IPFS Cluster?
Please participate in the IPFS Cluster user registry.
Table of Contents
Documentation
Please visit https://ipfscluster.io/documentation/ to access user documentation, guides and any other resources, including detailed download and usage instructions.
News & Roadmap
We regularly post project updates to https://ipfscluster.io/news/ .
The most up-to-date Roadmap is available at https://ipfscluster.io/roadmap/ .
Install
Instructions for different installation methods (including from source) are available at https://ipfscluster.io/download .
Usage
Extensive usage information is provided at https://ipfscluster.io/documentation/ , including:
Contribute
PRs accepted. As part of the IPFS project, we have some contribution guidelines.
License
This library is dual-licensed under Apache 2.0 and MIT terms.
© 2022. Protocol Labs, Inc.