cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,406 Commits 3 Branches 70 Tags 15 MiB
ea4dd7ed98
Commit Graph

2406 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Hector Sanjuan
ea4dd7ed98
Set development version v0.14.0-next 2021-07-09 00:25:18 +02:00
Hector Sanjuan
f7e6f4a26c
Release v0.14.0 2021-07-09 00:24:00 +02:00
Hector Sanjuan
78aa96449a Final update to changelog for v0.14.0 2021-07-09 00:23:30 +02:00
Hector Sanjuan
d3e5d87ab5
Merge pull request #1400 from ipfs/v0.14.0/changelog 
Changelog for v0.14.0
 2021-07-08 23:57:13 +02:00
Hector Sanjuan
7a954e4fd2
Merge pull request #1405 from ipfs/feat/speed-up-pin-ls 
restapi: allocationsHandler: skip filtering pins when not needed.
 2021-07-08 23:56:07 +02:00
Hector Sanjuan
b2b42c29ae
Merge pull request #1404 from ipfs/fix/1403-ctl-https 
Fix #1403: ctl: ssl errors when talking to https remote peers
 2021-07-08 23:55:55 +02:00
Hector Sanjuan
dc6c05ae51 Changelog for v0.14.0 2021-07-08 23:43:59 +02:00
Hector Sanjuan
3ba7d79d0a restapi: allocationsHandler: skip filtering pins when not needed. 
The restapi component supports filters for the pinset. This was done to keep
expected output when sharding was fully supported by filtering out "internal"
pins.

However this filter requires looping on the full pinset and re-allocating  and
usually does nothing. The useless copy is significant for really big pinsets.

Additionally, ipfs-cluster-ctl set the filter by default to "pins". By setting
it to "all" instead we can skip the whole filtering step and, in practice, get the
same results.
 2021-07-08 23:27:55 +02:00
Hector Sanjuan
33c7b5d81a Fix #1403: ctl: ssl errors when talking to https remote peers 
When using SSL and not talking to libp2p-http endpoints, we should not
resolve the dns names in the multiaddresses as otherwise we cannot
verify the https certificates used by the remote endpoint.
 2021-07-08 23:21:36 +02:00
Hector Sanjuan
3ba27bebb4
Release candidate v0.14.0-rc3 2021-07-08 12:48:30 +02:00
Hector Sanjuan
164e674fcd
Merge pull request #1399 from ipfs/fix/1360-efficient-pin-status-followup 
pintracker: avoid listing the state unless necessary
 2021-07-08 02:24:10 +02:00
Hector Sanjuan
3e0a622fe1
Merge pull request #1398 from ipfs/fix/improve-defaults 
new defaults: disable_repinning:true and badger as default datastore
 2021-07-08 01:03:32 +02:00
Hector Sanjuan
27569bdf88 pintracker: avoid listing the state unless necessary 
This is a follow up to #1360 which further optimizes StatusAll calls by
avoiding listing and filtering the cluster state when requesting status for
operations that should be direclty in the operation tracker because they are
ongoing (queued, pinning, unpinning, error).
 2021-07-08 01:01:25 +02:00
Hector Sanjuan
aba8e8e892 config: set disable_repinning to true by default. 
Repinnings happen unexpectedly and just bite people during normal
rollouts. This feature, while useful in some cases, should be enabled by
people that are aware of the consequences.
 2021-07-08 00:40:24 +02:00
Hector Sanjuan
46a14f9602 Set badger as default datastore for ipfs-cluster-service 
This goes back to badger as the default. Testing performed on 0.14.0-rc2
showed very successful GC behaviour on large clusters.

Being that we run with badger at scale and that the main problem seems
resolved, it seems appropiate to default to badger.
 2021-07-08 00:37:03 +02:00
Hector Sanjuan
493089d6d0
Release candidate v0.14.0-rc2 2021-07-06 16:56:53 +02:00
Hector Sanjuan
adb15feb6e
Dependency upgrades (#1395) 
* build(deps): bump github.com/multiformats/go-multiaddr-dns

Bumps [github.com/multiformats/go-multiaddr-dns](https://github.com/multiformats/go-multiaddr-dns) from 0.2.0 to 0.3.1.
- [Release notes](https://github.com/multiformats/go-multiaddr-dns/releases)
- [Commits](https://github.com/multiformats/go-multiaddr-dns/compare/v0.2.0...v0.3.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/hashicorp/go-hclog from 0.15.0 to 0.16.0

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.15.0...v0.16.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/ipfs/go-unixfs from 0.2.4 to 0.2.5

Bumps [github.com/ipfs/go-unixfs](https://github.com/ipfs/go-unixfs) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/ipfs/go-unixfs/releases)
- [Commits](https://github.com/ipfs/go-unixfs/compare/v0.2.4...v0.2.5)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/libp2p/go-libp2p-peerstore

Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases)
- [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.2.6...v0.2.7)

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump go.uber.org/multierr from 1.6.0 to 1.7.0

Bumps [go.uber.org/multierr](https://github.com/uber-go/multierr) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/uber-go/multierr/releases)
- [Changelog](https://github.com/uber-go/multierr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/multierr/compare/v1.6.0...v1.7.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Chore: update deps

* Update changelog

* Update to go1.16. Downgrade unixfs.

* go mod tidy

* travis: use go install

* golint no more

* Update configuration for dependabot

* Fix wrong dependabot config

* dependabot

* Revert update of go-unixfs

* Dependency upgrades

* Bump github.com/libp2p/go-libp2p-gorpc from 0.1.2 to 0.1.3

Bumps [github.com/libp2p/go-libp2p-gorpc](https://github.com/libp2p/go-libp2p-gorpc) from 0.1.2 to 0.1.3.
- [Release notes](https://github.com/libp2p/go-libp2p-gorpc/releases)
- [Commits](https://github.com/libp2p/go-libp2p-gorpc/compare/v0.1.2...v0.1.3)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-gorpc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix deprecated objects with prometheus

* chore: update dependencies

* monitor: remove dependency to go-multicodec

go-multicodec has been deprecated and it was just a wrapper.

This switches directly to ugorji/go/codec's msgpack for cluster metrics
serialization.

* Upgrade mfs so it works with latest go-unixfs

* Bump github.com/ugorji/go/codec from 1.2.5 to 1.2.6 (#1391)

Bumps [github.com/ugorji/go/codec](https://github.com/ugorji/go) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/ugorji/go/releases)
- [Commits](https://github.com/ugorji/go/compare/v1.2.5...v1.2.6)

---
updated-dependencies:
- dependency-name: github.com/ugorji/go/codec
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/hashicorp/go-hclog from 0.16.0 to 0.16.1 (#1392)

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.16.0 to 0.16.1.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.16.0...v0.16.1)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-06 16:47:04 +02:00
Hector Sanjuan
a5ae9fffc4
Release candidate v0.14.0-rc1 2021-07-06 14:07:21 +02:00
Hector Sanjuan
3a8098260b
Dependency upgrades (#1378) 
* build(deps): bump github.com/multiformats/go-multiaddr-dns

Bumps [github.com/multiformats/go-multiaddr-dns](https://github.com/multiformats/go-multiaddr-dns) from 0.2.0 to 0.3.1.
- [Release notes](https://github.com/multiformats/go-multiaddr-dns/releases)
- [Commits](https://github.com/multiformats/go-multiaddr-dns/compare/v0.2.0...v0.3.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/hashicorp/go-hclog from 0.15.0 to 0.16.0

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.15.0...v0.16.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/ipfs/go-unixfs from 0.2.4 to 0.2.5

Bumps [github.com/ipfs/go-unixfs](https://github.com/ipfs/go-unixfs) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/ipfs/go-unixfs/releases)
- [Commits](https://github.com/ipfs/go-unixfs/compare/v0.2.4...v0.2.5)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/libp2p/go-libp2p-peerstore

Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases)
- [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.2.6...v0.2.7)

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump go.uber.org/multierr from 1.6.0 to 1.7.0

Bumps [go.uber.org/multierr](https://github.com/uber-go/multierr) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/uber-go/multierr/releases)
- [Changelog](https://github.com/uber-go/multierr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/multierr/compare/v1.6.0...v1.7.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Chore: update deps

* Update changelog

* Update to go1.16. Downgrade unixfs.

* go mod tidy

* travis: use go install

* golint no more

* Update configuration for dependabot

* Fix wrong dependabot config

* dependabot

* Revert update of go-unixfs

* Dependency upgrades

* Bump github.com/libp2p/go-libp2p-gorpc from 0.1.2 to 0.1.3

Bumps [github.com/libp2p/go-libp2p-gorpc](https://github.com/libp2p/go-libp2p-gorpc) from 0.1.2 to 0.1.3.
- [Release notes](https://github.com/libp2p/go-libp2p-gorpc/releases)
- [Commits](https://github.com/libp2p/go-libp2p-gorpc/compare/v0.1.2...v0.1.3)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-gorpc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix deprecated objects with prometheus

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-06 14:06:04 +02:00
Hector Sanjuan
ce2490c64f
Merge pull request #1389 from ipfs/fix/db-close-tests 
Fix: tests: close datastore on cluster node shutdown.
 2021-07-06 14:05:36 +02:00
Hector Sanjuan
8ce98ceae3 Fix: tests: close datastore on cluster node shutdown. 
This resulted in too-many-files-open when running with leveldb.
 2021-07-06 12:28:03 +02:00
Hector Sanjuan
54c3608899
Merge pull request #1377 from ipfs/fix/1360-efficient-pin-status 
Fix #1360: Efficient pinset status with filters
 2021-07-06 11:57:09 +02:00
Hector Sanjuan
edfcfa3fb0 Fix #1360: Efficient pinset status with filters 
This commit modifies the pintracker StatusAll call to take a status filter.

This allows to skip a PinLs call to ipfs when checking status for items that
are queued, pinning, unpinning or in error. Those status come directly from
the operation tracker. This should result in a significant performance
increase for those calls, particularly in nodes with several hundred thousand
pins and more, where the call to IPFS is very expensive.

A new TrackerStatusUnexpectedlyUnpinned status has been introduce to
differentiate between pin errors (tracked by the operation tracker) and "lost"
items (which before were pin errors too). This new status is handled by the
Recover() operation as before.
 2021-07-06 11:34:19 +02:00
Hector Sanjuan
2397d6d766
Merge pull request #1375 from ipfs/feat/1374-providers 
Support user-provided pin "origins"
 2021-07-06 11:32:49 +02:00
Hector Sanjuan
1952693ab5 Feat #1374: ipfshttp: do not swarm connect to more than 10 origins. 
Disallow silly behaviours by default.
 2021-07-02 16:38:04 +02:00
Hector Sanjuan
625b344cb7 Feat #1374: ipfshttp: support pin.Origins and swarm-connect to them 
This allows the ipfshttp IPFS-connector to support pin origins.

When present, it will issue "swarm connect" requests to IPFS, in the
background, with 0 weight (unsupported in go-ipfs yet) to a maximum of 10
peers. Errors are ignored.

branch feat/1374-providers # Changes to be committed: # modified: ipfshttp.go
../../cmd/ipfs-cluster-service/export.json #
../../cmd/ipfs-cluster-service/test/ #
 2021-07-02 02:09:27 +02:00
Hector Sanjuan
e07f30b2ab Feat #1374: Only accept Origins that have a p2p component 2021-07-02 01:49:30 +02:00
Hector Sanjuan
7e26ca8fc6 Feat #1374: Add Origins option to pins 
This adds a new pin option called Origins, consisting of a slice of multiaddresses.
 2021-07-02 01:42:51 +02:00
Hector Sanjuan
32e35a6a9b
Merge pull request #1372 from ipfs/feat/fix-1362-car-size 
Fix #1362: Add "size" to response when adding CARs with a unixfs file root
 2021-07-02 01:02:11 +02:00
Hector Sanjuan
c9783c064f
Merge pull request #1370 from ipfs/feat/badger-gc 
Fix #1320: Add automatic GC to Badger datastore
 2021-07-02 01:01:55 +02:00
Hector Sanjuan
f7a2e4a3fc
Merge pull request #1373 from ipfs/feat/docker-select-ds 
Docker: allow controlling datastore backend via IPFS_CLUSTER_DATASTORE
 2021-06-29 16:14:01 +02:00
Hector Sanjuan
74e4dfb2e4 Docker: allow controlling datastore backend via IPFS_CLUSTER_DATASTORE 2021-06-29 15:44:24 +02:00
Hector Sanjuan
55a6271c28
Merge pull request #1371 from ipfs/feat/fix-1366 
Fix #1366: IPFS Proxy API serializes wrong error types
 2021-06-29 15:38:01 +02:00
Hector Sanjuan
81d5759cc8 Fix #1366: IPFS Proxy API serializes wrong error types 
The errors returned by the IPFS Proxy API are not understood by IPFS.

This was caused by go-ipfs-cmds setting an API error format which requires the
errors to have a type: "error" field.

This commit brings this up to speed.
 2021-06-28 22:42:47 +02:00
Hector Sanjuan
b6238df481 Fix #1362: Add "size" to response when adding CARs with a unixfs file root. 
This sets the "size" value to the FileSize() value when the CAR file's root
is a unixfs file. (Folders are not files, size will still be 0).
 2021-06-28 22:37:32 +02:00
Hector Sanjuan
4ac2cf3eb0 Fix #1320: Add automatic GC to Badger datastore 
This takes advantage of go-ds-badger "auto-gc" feature.

It will run a GC cycle made of multiple GC rounds (until it cannot GC more)
automatically. The behaviour is enabled by default in the configuration and
can be disabled by setting "gc_interval" to "0m". Hopefully this prevents
badger datastores from growing crazy.
 2021-06-28 21:51:29 +02:00
Hector Sanjuan
d37df4fd95
Merge pull request #1364 from ipfs/feat/custom-backends 
Support a levelDB backend for cluster
 2021-06-28 19:21:52 +02:00
Hector Sanjuan
099e23cbd1 Run tests also using leveldb backend 2021-06-11 18:43:54 +02:00
Hector Sanjuan
0eef0ede89 Support a levelDB backend for cluster 
Badger can take 1000x the amount of needed space if not GC'ed or compacted
(#1320), even for non heavy usage. Cluster has no provisions to run datastore
GC operations and while they could be added, they are not ensured to
help. Improvements on Badger v3 might help but would still need to GC
explicitally.

Cluster was however designed to support any go-datastore as backend.

This commit adds LevelDB support. LevelDB go-datastore wrapper is mature, does
not need GC and should work well for most cluster usecases, which are not
overly demanding.

A new `--datastore` flag has been added on init. The store backend is selected
based on the value in the configuration, similar to how raft/crdt is. The
default is set to leveldb. From now on it should be easier to add additional
backends, i.e. badgerv3.
 2021-06-09 19:40:36 +02:00
Hector Sanjuan
8454cd6383
Merge pull request #1363 from vasco-santos/chore/use-latest-go-ipfs-in-docker-compose 
chore: use latest go ipfs in docker compose
 2021-06-09 19:08:19 +02:00
Vasco Santos
f98701507d chore: use latest in all instances 2021-06-09 15:39:34 +02:00
Vasco Santos
b58a032f0d chore: use latest go ipfs in docker compose 2021-06-09 15:35:14 +02:00
Hector Sanjuan
c5d0695c0a
Merge pull request #1361 from ipfs/fix/irc 
Update IRC link in repository badges
 2021-06-01 14:31:06 +02:00
Hector Sanjuan
75cc71420e Update IRC link in repository badges 2021-06-01 14:29:47 +02:00
Hector Sanjuan
55516fe7c6
Fix #1358: Set RawLeaves to true when unset and CidVersion=1 (#1359) 
Fixes #1358. Mimics go-ipfs defaults.

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
 2021-05-19 23:56:30 +02:00
Hector Sanjuan
bfe179e943
Set development version v0.13.3-next 2021-05-14 19:04:01 +02:00
Hector Sanjuan
dd87dba323
Release v0.13.3 2021-05-14 19:03:15 +02:00
Hector Sanjuan
f7c78df418
Dependency upgrades (#1357) 
* build(deps): bump github.com/multiformats/go-multiaddr-dns

Bumps [github.com/multiformats/go-multiaddr-dns](https://github.com/multiformats/go-multiaddr-dns) from 0.2.0 to 0.3.1.
- [Release notes](https://github.com/multiformats/go-multiaddr-dns/releases)
- [Commits](https://github.com/multiformats/go-multiaddr-dns/compare/v0.2.0...v0.3.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/hashicorp/go-hclog from 0.15.0 to 0.16.0

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.15.0...v0.16.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/ipfs/go-unixfs from 0.2.4 to 0.2.5

Bumps [github.com/ipfs/go-unixfs](https://github.com/ipfs/go-unixfs) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/ipfs/go-unixfs/releases)
- [Commits](https://github.com/ipfs/go-unixfs/compare/v0.2.4...v0.2.5)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/libp2p/go-libp2p-peerstore

Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases)
- [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.2.6...v0.2.7)

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump go.uber.org/multierr from 1.6.0 to 1.7.0

Bumps [go.uber.org/multierr](https://github.com/uber-go/multierr) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/uber-go/multierr/releases)
- [Changelog](https://github.com/uber-go/multierr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/multierr/compare/v1.6.0...v1.7.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Chore: update deps

* Update changelog

* Update to go1.16. Downgrade unixfs.

* go mod tidy

* travis: use go install

* golint no more

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-05-14 19:01:18 +02:00
Hector Sanjuan
15d2846306 Finish changelog for v0.13.3 release 2021-05-14 17:02:52 +02:00
Hector Sanjuan
49dd205168
Merge pull request #1356 from ipfs/fix/err-msg 
fix: error message for LogPin
 2021-05-12 15:55:58 +02:00