cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,839 Commits 3 Branches 70 Tags 15 MiB
ba1ba989ca
Commit Graph

2839 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Hector Sanjuan
ba1ba989ca Fix #1796: Implement basic DHT-peer source for libp2p AutoRelay. 
At some point libp2p decided that it would not automatically lookup peers on
the DHT when trying to find relays, and instead silently introduced options
for the EnableAutoRelay option, in a way that it randomly panicked first, or,
in later versions, consistently panics.

This fixes that by providing a PeerSource option for AutoRelay. We auto-relay
decides that it needs peers, we will perform a cluster-DHT lookup and send
those. Hopefully this is similar to the previous behaviour.

Since all cluster peers are relays, that should work, hopefully.  In general,
this means NAT'ed peers should be able to find relays to perform hole-punching
when connecting to other NAT'ed peers etc. In practice this is not a scenario
we see a lot with clusters so it is not very well tested.

At least things are not going to panic.
 2023-01-27 18:12:48 +01:00
Hector Sanjuan
b29f8b807a Fix #1796: Disable AutoRelay 
Per https://github.com/libp2p/go-libp2p/issues/1852, the AutoRelay subsystem
is now panicking on users. EnableAutoRelay must now be called with options,
otherwise it seems to panic for some people.

Disabling it is the best for now, given relays are enabled and a node must be
able to connect to others on bootstrap, perhaps it does not need to
re-discover new relays (every other node should be a relay).

In any case we should revisit relay support and related services in Cluster,
since semantics have changed a lot in libp2p, relayV2 is a thing,
hole-punching is a thing etc. etc.
 2023-01-27 14:53:08 +01:00
Hector Sanjuan
fd7e4ea336
ipfsconn: do not error when we cannot parse multiaddresses from ipfs. (#1841) 
Fixes: #1835.

If IPFS introduces a new multiaddress type/string that we have not compiled,
we error. This caused issues with latest ipfs version (which we fixed by
upgrading libraries too). This makes cluster a bit more future proof with
upcoming ipfs versions.
 2023-01-27 14:52:21 +01:00
Hector Sanjuan
7cc91cae4f
Dependency upgrades (#1840) 
* Bump github.com/libp2p/go-libp2p-kad-dht from 0.18.0 to 0.20.0 (#1821)

Bumps [github.com/libp2p/go-libp2p-kad-dht](https://github.com/libp2p/go-libp2p-kad-dht) from 0.18.0 to 0.20.0.
- [Release notes](https://github.com/libp2p/go-libp2p-kad-dht/releases)
- [Commits](https://github.com/libp2p/go-libp2p-kad-dht/compare/v0.18.0...v0.20.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-kad-dht
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/dgraph-io/badger/v3 from 3.2103.4 to 3.2103.5 (#1822)

Bumps [github.com/dgraph-io/badger/v3](https://github.com/dgraph-io/badger) from 3.2103.4 to 3.2103.5.
- [Release notes](https://github.com/dgraph-io/badger/releases)
- [Changelog](https://github.com/dgraph-io/badger/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dgraph-io/badger/compare/v3.2103.4...v3.2103.5)

---
updated-dependencies:
- dependency-name: github.com/dgraph-io/badger/v3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump go.uber.org/multierr from 1.8.0 to 1.9.0 (#1823)

Bumps [go.uber.org/multierr](https://github.com/uber-go/multierr) from 1.8.0 to 1.9.0.
- [Release notes](https://github.com/uber-go/multierr/releases)
- [Changelog](https://github.com/uber-go/multierr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/multierr/compare/v1.8.0...v1.9.0)

---
updated-dependencies:
- dependency-name: go.uber.org/multierr
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/rs/cors from 1.8.2 to 1.8.3 (#1826)

Bumps [github.com/rs/cors](https://github.com/rs/cors) from 1.8.2 to 1.8.3.
- [Release notes](https://github.com/rs/cors/releases)
- [Commits](https://github.com/rs/cors/compare/v1.8.2...v1.8.3)

---
updated-dependencies:
- dependency-name: github.com/rs/cors
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/ipfs/go-merkledag from 0.8.1 to 0.9.0 (#1830)

Bumps [github.com/ipfs/go-merkledag](https://github.com/ipfs/go-merkledag) from 0.8.1 to 0.9.0.
- [Release notes](https://github.com/ipfs/go-merkledag/releases)
- [Commits](https://github.com/ipfs/go-merkledag/compare/v0.8.1...v0.9.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-merkledag
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/dustin/go-humanize from 1.0.0 to 1.0.1 (#1831)

Bumps [github.com/dustin/go-humanize](https://github.com/dustin/go-humanize) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/dustin/go-humanize/releases)
- [Commits](https://github.com/dustin/go-humanize/compare/v1.0.0...v1.0.1)

---
updated-dependencies:
- dependency-name: github.com/dustin/go-humanize
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/urfave/cli from 1.22.10 to 1.22.12 (#1834)

Bumps [github.com/urfave/cli](https://github.com/urfave/cli) from 1.22.10 to 1.22.12.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v1.22.10...v1.22.12)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/ipfs/go-unixfs from 0.4.1 to 0.4.2 (#1829)

Bumps [github.com/ipfs/go-unixfs](https://github.com/ipfs/go-unixfs) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/ipfs/go-unixfs/releases)
- [Commits](https://github.com/ipfs/go-unixfs/compare/v0.4.1...v0.4.2)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-unixfs
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump go.opencensus.io from 0.23.0 to 0.24.0 (#1794)

Dependabot couldn't find the original pull request head commit, 9dedea6eeef2c8b5f7358cc29b5b088ac6cf7aba.

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/ugorji/go/codec from 1.2.7 to 1.2.8 (#1827)

Bumps [github.com/ugorji/go/codec](https://github.com/ugorji/go) from 1.2.7 to 1.2.8.
- [Release notes](https://github.com/ugorji/go/releases)
- [Commits](https://github.com/ugorji/go/compare/v1.2.7...v1.2.8)

---
updated-dependencies:
- dependency-name: github.com/ugorji/go/codec
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/libp2p/go-libp2p-pubsub from 0.8.2 to 0.8.3 (#1838)

Bumps [github.com/libp2p/go-libp2p-pubsub](https://github.com/libp2p/go-libp2p-pubsub) from 0.8.2 to 0.8.3.
- [Release notes](https://github.com/libp2p/go-libp2p-pubsub/releases)
- [Commits](https://github.com/libp2p/go-libp2p-pubsub/compare/v0.8.2...v0.8.3)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-pubsub
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update go-libp2p

* Bump github.com/ipfs/go-block-format from 0.0.3 to 0.1.1 (#1837)

Bumps [github.com/ipfs/go-block-format](https://github.com/ipfs/go-block-format) from 0.0.3 to 0.1.1.
- [Release notes](https://github.com/ipfs/go-block-format/releases)
- [Commits](https://github.com/ipfs/go-block-format/compare/v0.0.3...v0.1.1)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-block-format
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update go-mfs (#1818)

* Fix docker build failure (git safe directory) (#1836)

Git refuses to run `git rev-parse HEAD` now, in docker.

* Remove go-block-format and use go-libipfs (deprecation)

* Update ipfs-lite

* Bump golang.org/x/crypto from 0.3.0 to 0.5.0 (#1839)

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.3.0 to 0.5.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/compare/v0.3.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-27 14:34:53 +01:00
Hector Sanjuan
fc6ce30ac1
Fix docker build failure (git safe directory) (#1836) 
Git refuses to run `git rev-parse HEAD` now, in docker.
 2023-01-27 13:55:38 +01:00
Hector Sanjuan
38b463d658
Update go-mfs (#1818) 2022-12-09 19:41:02 +01:00
Hector Sanjuan
b93bd66169
Dependency upgrades (#1813) 
* build(deps): bump github.com/multiformats/go-multiaddr

Bumps [github.com/multiformats/go-multiaddr](https://github.com/multiformats/go-multiaddr) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/multiformats/go-multiaddr/releases)
- [Commits](https://github.com/multiformats/go-multiaddr/compare/v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multiaddr
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli/v2 from 2.14.1 to 2.15.0

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.14.1 to 2.15.0.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.14.1...v2.15.0)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli/v2 from 2.15.0 to 2.16.2

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.15.0 to 2.16.2.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.15.0...v2.16.2)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-pubsub from 0.8.0 to 0.8.1

Bumps [github.com/libp2p/go-libp2p-pubsub](https://github.com/libp2p/go-libp2p-pubsub) from 0.8.0 to 0.8.1.
- [Release notes](https://github.com/libp2p/go-libp2p-pubsub/releases)
- [Commits](https://github.com/libp2p/go-libp2p-pubsub/compare/v0.8.0...v0.8.1)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-pubsub
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli/v2 from 2.16.2 to 2.16.3

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.16.2 to 2.16.3.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.16.2...v2.16.3)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-ipns from 0.2.0 to 0.3.0

Bumps [github.com/ipfs/go-ipns](https://github.com/ipfs/go-ipns) from 0.2.0 to 0.3.0.
- [Release notes](https://github.com/ipfs/go-ipns/releases)
- [Commits](https://github.com/ipfs/go-ipns/compare/v0.2.0...v0.3.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipns
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/hashicorp/raft from 1.3.10 to 1.3.11

Bumps [github.com/hashicorp/raft](https://github.com/hashicorp/raft) from 1.3.10 to 1.3.11.
- [Release notes](https://github.com/hashicorp/raft/releases)
- [Changelog](https://github.com/hashicorp/raft/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hashicorp/raft/compare/v1.3.10...v1.3.11)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/raft
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/prometheus/client_golang from 1.13.0 to 1.14.0

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.13.0 to 1.14.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.13.0...v1.14.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/multiformats/go-multiaddr from 0.7.0 to 0.8.0

Bumps [github.com/multiformats/go-multiaddr](https://github.com/multiformats/go-multiaddr) from 0.7.0 to 0.8.0.
- [Release notes](https://github.com/multiformats/go-multiaddr/releases)
- [Commits](https://github.com/multiformats/go-multiaddr/compare/v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multiaddr
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-ipfs-files from 0.1.1 to 0.2.0

Bumps [github.com/ipfs/go-ipfs-files](https://github.com/ipfs/go-ipfs-files) from 0.1.1 to 0.2.0.
- [Release notes](https://github.com/ipfs/go-ipfs-files/releases)
- [Commits](https://github.com/ipfs/go-ipfs-files/compare/v0.1.1...v0.2.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipfs-files
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-merkledag from 0.7.0 to 0.8.1

Bumps [github.com/ipfs/go-merkledag](https://github.com/ipfs/go-merkledag) from 0.7.0 to 0.8.1.
- [Release notes](https://github.com/ipfs/go-merkledag/releases)
- [Commits](https://github.com/ipfs/go-merkledag/compare/v0.7.0...v0.8.1)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-merkledag
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.4.3

Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.4.2 to 4.4.3.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.4.2...v4.4.3)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-pubsub from 0.8.1 to 0.8.2

Bumps [github.com/libp2p/go-libp2p-pubsub](https://github.com/libp2p/go-libp2p-pubsub) from 0.8.1 to 0.8.2.
- [Release notes](https://github.com/libp2p/go-libp2p-pubsub/releases)
- [Commits](https://github.com/libp2p/go-libp2p-pubsub/compare/v0.8.1...v0.8.2)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-pubsub
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/multiformats/go-multicodec from 0.6.0 to 0.7.0

Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/multiformats/go-multicodec/releases)
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-unixfs from 0.4.0 to 0.4.1

Bumps [github.com/ipfs/go-unixfs](https://github.com/ipfs/go-unixfs) from 0.4.0 to 0.4.1.
- [Release notes](https://github.com/ipfs/go-unixfs/releases)
- [Commits](https://github.com/ipfs/go-unixfs/compare/v0.4.0...v0.4.1)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-unixfs
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/urfave/cli/v2 from 2.16.3 to 2.23.5

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.16.3 to 2.23.5.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.16.3...v2.23.5)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Revert "Bump github.com/urfave/cli/v2 from 2.16.3 to 2.23.5"

This reverts commit 094760a923dfcb32d6a4a5ab274f4cf93ee8a8d5.

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-02 18:32:08 +01:00
Hector Sanjuan
3368c858f3
Merge pull request #1809 from ipfs-cluster/perf-storage 
Add support for additional datastores
 2022-12-02 10:49:41 +00:00
Hector Sanjuan
58cad80157 Enable tests using the new datastores (badger3, pebble) 2022-12-02 11:29:33 +01:00
Hector Sanjuan
f496e522cd Add Pebble support as datastore backend 
Pebble is a modern datastore based on RocksDB. This commit adds support to use
it as datastore backend for IPFS-Cluster.
 2022-12-02 11:29:33 +01:00
Hector Sanjuan
970bdc8cb0 Add Badger v3 support 
This commit adds badger v3 support as datastore backend for IPFS Cluster.

Badgerv3 has a number of improvements over v1, particularly around
compaction. It is significantly more modern and should behave better in
general (some testing is pending).
 2022-12-02 11:29:33 +01:00
Hector Sanjuan
c10c37f25b Adjust CRDT consensus timeout to be infinite (1 year) 
This allows for crdt migrations to finish correctly.
 2022-11-25 12:56:01 +01:00
Hector Sanjuan
36f516696f
Release v1.0.4 2022-09-27 12:44:43 +02:00
Hector Sanjuan
f376cf5106
Merge pull request #1772 from ipfs-cluster/v1.0.4/changelog 
Changelog for v1.0.4
 2022-09-27 12:44:18 +02:00
Hector Sanjuan
9cc97690bf
Merge pull request #1774 from ipfs-cluster/fix/raft-test 
Fix: repinning does not re-allocate as needed
 2022-09-27 12:44:05 +02:00
Hector Sanjuan
c23cff20a4 Changelog for v1.0.4 2022-09-27 12:43:38 +02:00
Hector Sanjuan
11124ee224 Fix: repinning does not re-allocate as needed 
Long story: Since #1768 there has been a recurring repinning test failure with
Raft consensus.

Per the test, if a pin is allocated to a peer that has been shutdown,
submitting the pin again should re-allocate it to a peer that is still
running.

Investigation on why this test fails and why it fails only in Raft lead to
realizing that this and other similar tests, were passing by chance. The
needed re-allocations were made not by the new submission of the pin, but by
the automatic-repinning feature. The actual resubmitted pin was carrying the
same allocations (one of them being the peer that was down), but it was
silently failing because the RedirectToLeader() code path was using
cc.ctx and hitting the peer that had been shutdown, which caused it to error.

Fixing the context propagation, meant that we would re-overwrite the pin with
the old allocations, thus the actual behaviour did not pass the test.

So, on one side, this fix an number of tests that had not disabled automatic
repinning and was probably getting in the way of things. On the other side,
this removes a condition that prevents re-allocation of pins if they exists
and options have not changed.

I don't fully understand why this was there though, since the Allocate() code
does return the old allocations anyways when they are enough, so it should not
re-allocate randomly. I suspect this was preventing some misbehaviour in the
Allocate() code from the time before it was improved with multiple allocators
etc.
 2022-09-27 12:31:24 +02:00
Hector Sanjuan
53bec20807
Merge pull request #1771 from ipfs-cluster/crdt-mark-dirty 
service: add crdt info, mark-dirty, mark-clean commands
 2022-09-26 21:25:17 +02:00
Hector Sanjuan
ffef2cea0a service: add crdt info, mark-dirty, mark-clean commands 
Provide access to some crdt-internal operations
 2022-09-26 19:39:04 +02:00
Hector Sanjuan
048d168126
Merge pull request #1768 from ipfs-cluster/fix/bad-context-propagation 
Fix bad context propagation / deadlocks
 2022-09-26 19:36:13 +02:00
Hector Sanjuan
21855c3130 Fix bad context propagation / deadlocks 
We are propagating the wrong context (mostly from the Cluster top-level
methods). This makes that request cancellations (and cancellations of the
associated contexts) are not propagated to many methods, and can result in
deadlocks when an operation that is holding a lock is not aborted.

This affects for example the operation tracker. Getting all operations from
the tracker relies on someone reading from the out channel, or on the context
being cancelled. When a request is aborted in the middle of the response, and
the context is not cancelled, everything that wants to list operations would
become deadlocked, including operations that need write locks like
TrackNewOperation.

This fixes it.
 2022-09-26 19:35:55 +02:00
Hector Sanjuan
328f2388d0
Merge pull request #1770 from ipfs-cluster/fix/ipfshttp-panic-return 
ipfshttp: fix return with nil error
 2022-09-26 19:34:56 +02:00
Hector Sanjuan
1efd6a7bd1 ipfshttp: fix return with nil error 2022-09-26 18:50:33 +02:00
Hector Sanjuan
003da51c7b
Release v1.0.3 2022-09-16 12:01:03 +02:00
Hector Sanjuan
b9debdb79c
Merge pull request #1763 from ipfs-cluster/v1.0.3/changelog 
Changelog updates for v1.0.3
 2022-09-15 19:10:51 +02:00
Hector Sanjuan
cb6cdc8ad6 Changelog updates for v1.0.3 2022-09-15 18:29:54 +02:00
Hector Sanjuan
af1f69a6f0 ipfshttp: remove leftover Println 2022-09-15 18:02:04 +02:00
Hector Sanjuan
c9895bf607
Merge pull request #1762 from ipfs-cluster/fix/1733-ipfs-error-handling 
Behaviour improvements when the ipfs daemon is unavailable
 2022-09-15 17:59:04 +02:00
Hector Sanjuan
592d61b228 ipfshttp: rate limit requests when failures happen 
When IPFS starts failing or doesn't respond (i.e. during a restart), cluster
is likely to start sending requests at very fast rates. i.e. if there are 100k
items to be pinned, and pins start failing immediately, cluster will consume
the pin queue really fast and it will all be failures. At the same time, ipfs
is hammered non-stop until recover, which may make it harder.

This commits introduces a rate-limit when requests to IPFS fail. After 10
failed requests, requests will be sent at most at 1req/s rate. Once a requests
succeeds, the rate-limit is raised.

This should prevent hammering the IPFS daemon, but also increased CPU in
cluster as it burns through pinning queues when IPFS is offline, making the
situation in machines worse (and emitting way more logs).
 2022-09-15 17:37:26 +02:00
Hector Sanjuan
2286ee73f8 api: return errors on stream response requests with 0 items 
This fixes a bug in API code that made it return 204-No content when the RPC
methods failed with an error before any items were returned on the channel.
 2022-09-15 16:40:34 +02:00
Hector Sanjuan
12b8ce63ce stateless: abort when ipfs PinLs errors 
Unfortunately we were not paying attentions to errors while rpc-streaming pins
in the pintracker. The result is that the StatusAll operation would list all
the pins as unexpectedly unpinned when ipfs is offline, and this would result in
recover/requeing operations for all pins when ipfs is offline.

This commits changes the behaviour so that if IPFS Pin/ls has resulted in an
error, then the StatusAll operation cannot complete at all.
 2022-09-15 16:40:34 +02:00
Hector Sanjuan
a60a835e36 Wait for IPFS to be ready during start 
This commit introduces unlimited waiting on start until a request to `ipfs id`
succeeds.

Waiting has some consequences:

* State watching (recover/sync) and metrics publishing does not start until ipfs is ready
* swarm/connect is not triggered until ipfs is ready.

Once the first request to ipfs succeeds everything goes to what it was before.

This alleviates trying operations like sending our IDs in metrics when IPFS is
simply not there.
 2022-09-15 16:40:34 +02:00
Hector Sanjuan
b80f89dd01
Merge pull request #1516 from ipfs-cluster/crdtdot 
Allow exporting CRDT dag to dot files with: "ipfs-cluster-service state crdt dot"
 2022-09-14 10:58:58 +02:00
Hector Sanjuan
29d6f69819 Add "service state crdt dot" command 
This subcommand allows to export the peer's CRDT DAG as dot file.

It also opens the door to have more crdt-specific subcommands, i.e. CAR export
etc.
 2022-09-13 16:36:10 +02:00
Hector Sanjuan
fe11730e58
Merge pull request #1757 from ipfs-cluster/update-raft 
Update raft libraries
 2022-09-09 17:33:07 +02:00
Hector Sanjuan
b81379e383 Update raft libraries 
This updates Raft to v1.3.0. We can also remove some logging glue that is no
longer necessary in here.
 2022-09-09 17:17:45 +02:00
Hector Sanjuan
920ba03b1e
Merge pull request #1756 from ipfs-cluster/fix/1738-proxy-block-dag-put-intercept 
ipfsproxy: intercept block/put and dag/put and pin to cluster on pin=true
 2022-09-09 16:44:22 +02:00
Hector Sanjuan
7af52bdb4e ipfsproxy: forward block/put and dag/put with pin=false 
Since the pin argument is interpreted and performed by cluster. Pins may be allocated to other nodes.
 2022-09-09 16:41:58 +02:00
Hector Sanjuan
b2b33e8668 ipfsproxy: add missing heads to dag/put, block/put handlers 2022-09-09 16:35:55 +02:00
Hector Sanjuan
06f0cac9c0
Merge pull request #1753 from ipfs-cluster/fix/1706-fix-block-warning 
ipfshttp: Fix "seen blocks" tracking and blockPut metrics
 2022-09-09 16:32:09 +02:00
Hector Sanjuan
573d7e8916 ipfsproxy: add tests for block/put and dag/put intercepts 2022-09-09 16:27:09 +02:00
Hector Sanjuan
6ce90dfe47 ipfsproxy: intercept block/put and dag/put and pin to cluster on pin=true 
This fixes #1738. Tests still missing
 2022-09-09 00:49:50 +02:00
Hector Sanjuan
5452b59a2e
Dependency upgrades (#1755) 
* Update go-libp2p to v0.22.0

* Testing with go1.19

* build(deps): bump github.com/multiformats/go-multicodec

Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/multiformats/go-multicodec/releases)
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipld/go-car from 0.4.0 to 0.5.0

Bumps [github.com/ipld/go-car](https://github.com/ipld/go-car) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/ipld/go-car/releases)
- [Commits](https://github.com/ipld/go-car/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/ipld/go-car
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/prometheus/client_golang

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.2 to 1.13.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.2...v1.13.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/hashicorp/go-hclog from 1.2.1 to 1.3.0

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 1.2.1 to 1.3.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v1.2.1...v1.3.0)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-ds-crdt from 0.3.6 to 0.3.7

Bumps [github.com/ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/ipfs/go-ds-crdt/releases)
- [Commits](https://github.com/ipfs/go-ds-crdt/compare/v0.3.6...v0.3.7)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-crdt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli/v2 from 2.10.2 to 2.14.1

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.10.2 to 2.14.1.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.10.2...v2.14.1)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-http from 0.3.0 to 0.4.0

Bumps [github.com/libp2p/go-libp2p-http](https://github.com/libp2p/go-libp2p-http) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/libp2p/go-libp2p-http/releases)
- [Commits](https://github.com/libp2p/go-libp2p-http/compare/v0.3.0...v0.4.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-gorpc from 0.4.0 to 0.5.0

Bumps [github.com/libp2p/go-libp2p-gorpc](https://github.com/libp2p/go-libp2p-gorpc) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/libp2p/go-libp2p-gorpc/releases)
- [Commits](https://github.com/libp2p/go-libp2p-gorpc/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-gorpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump contrib.go.opencensus.io/exporter/prometheus

Bumps [contrib.go.opencensus.io/exporter/prometheus](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/releases)
- [Commits](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/compare/v0.4.1...v0.4.2)

---
updated-dependencies:
- dependency-name: contrib.go.opencensus.io/exporter/prometheus
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-raft from 0.1.8 to 0.2.0

Bumps [github.com/libp2p/go-libp2p-raft](https://github.com/libp2p/go-libp2p-raft) from 0.1.8 to 0.2.0.
- [Release notes](https://github.com/libp2p/go-libp2p-raft/releases)
- [Commits](https://github.com/libp2p/go-libp2p-raft/compare/v0.1.8...v0.2.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-raft
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli from 1.22.9 to 1.22.10

Bumps [github.com/urfave/cli](https://github.com/urfave/cli) from 1.22.9 to 1.22.10.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v1.22.9...v1.22.10)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix checker/linter/staticcheck warnings

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-09-06 16:57:17 +02:00
Hector Sanjuan
8c93d4cb81
Merge pull request #1754 from ipfs-cluster/struct-align-comment 
Add comments to struct fields that must be aligned.
 2022-09-06 16:23:24 +02:00
Hector Sanjuan
ed54c665b8 ipfshttp: Fix "seen blocks" tracking and blockPut metrics 
This fixes two bugs. First, the "blockPut response CID does not match the
multihash" warning was coming up when it shouldn't. Particularly, the
multipart reader called Node() several times for the same block, resulting in
CIDs been removed from the Seen set, and causing the warning when there were
several blocks (usually the empty dir block).

This also means we were counting Blockputs (and total data added) wrong in the
metrics, double-counting some blocks as these were recorded in Node() calls.

The fix makes the tracking in Next(), which is only called once for each
block. To avoid timing issues between Block reads from the channel and
blockput responses, the Seen set now stores how many times we have seen a
block. Thus a duplicated block that will get two BlockPut responses will not
trigger a warning regardless of the time when those responses arrive.

Fixes #1706.
 2022-09-05 18:10:36 +02:00
Hector Sanjuan
a84869d3db Add comments to struct fields that must be aligned. 2022-09-05 17:03:30 +02:00
Hector Sanjuan
71bda2d658
Merge pull request #1736 from cldy309/fix/1735-panic 
fix panic: unaligned 64-bit atomic operation on Linux@armv7
 2022-09-05 16:44:31 +02:00
chenlong348
bd3b88b933 fix panic: unaligned 64-bit atomic operation on Linux@armv7 2022-08-15 10:13:37 +08:00
Hector Sanjuan
c4d78d52f8
Merge pull request #1732 from ipfs-cluster/fix/goroutine-leak-adder 
Fix: leaking goroutines on aborted /add requests
 2022-07-08 17:40:24 +02:00
Hector Sanjuan
d19c7facff Fix: leaking goroutines on aborted /add requests 
It has been observed that some peers have a growing number of goroutines,
usually stuck in go-libp2p-gorpc.MultiStream() function, which is waiting to
read items from the arguments channel.

We suspect this is due to aborted /add requests. In situations when the add
request is aborted or fails, Finalize() is never called and the blocks channel
stays open, so MultiStream() can never exit, and the BlockStreamer can never
stop streaming etc.

As a fix, we added the requirement to call Close() when we stop using a
ClusterDAGService (error or not). This should ensure that the blocks channel
is always closed and not just on Finalize().
 2022-07-08 17:39:59 +02:00