cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,433 Commits 3 Branches 70 Tags 15 MiB
b6a46cd8a4
Commit Graph

2433 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Hector Sanjuan
b6a46cd8a4 allocator: rework the whole allocator system 
The new "metrics" allocator is about to partition metrics and distribe
allocations among the partitions.

For example: given a region, an availability zone and free space on disk, the
allocator would be able to choose allocations by distributing among regions
and availability zones as much as possible, and for those peers in the same
region/az, selecting those with most free space first.

This requires a major overhaul of the allocator component.
 2021-09-13 12:24:00 +02:00
Hector Sanjuan
4060f4196b Revert "Informer/disk: deprecate RepoSize metric" 
This reverts commit 3e54c4c695.
 2021-09-10 18:53:59 +02:00
Hector Sanjuan
3e54c4c695 Informer/disk: deprecate RepoSize metric 
The allocator is hardcoded to descendalloc for freespace so it is not even useful
as it would allocate to peers with largest reposize first no matter what.

We are, in any case, reworking allocators and informers etc.
 2021-09-08 17:36:54 +02:00
Hector Sanjuan
396a348a65
Merge pull request #1462 from ipfs/fix/1461-flaky-test 
Adder: try to make test less flaky
 2021-08-30 12:04:43 +02:00
Hector Sanjuan
26007114b4 Adder: try to make test less flaky 2021-08-29 01:16:28 +02:00
Hector Sanjuan
f9b20caa2a changelog: fix v0.14.1 release date 2021-08-17 10:22:36 +02:00
Hector Sanjuan
267deeff28
Set development version v0.14.1-next 2021-08-16 18:03:18 +02:00
Hector Sanjuan
0162df60eb
Release v0.14.1 2021-08-16 18:02:52 +02:00
Hector Sanjuan
506b1657b9
Merge pull request #1452 from ipfs/v0.14.1/changelog 
Changelog for v0.14.1
 2021-08-16 17:54:52 +02:00
Hector Sanjuan
3f5490424b
Dependency upgrades (#1451) 
* Bump github.com/hashicorp/go-hclog from 0.16.1 to 0.16.2

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.16.1 to 0.16.2.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.16.1...v0.16.2)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-log/v2 from 2.2.0 to 2.3.0

Bumps [github.com/ipfs/go-log/v2](https://github.com/ipfs/go-log) from 2.2.0 to 2.3.0.
- [Release notes](https://github.com/ipfs/go-log/releases)
- [Commits](https://github.com/ipfs/go-log/compare/v2.2.0...v2.3.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-log/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p from 0.14.3 to 0.14.4

Bumps [github.com/libp2p/go-libp2p](https://github.com/libp2p/go-libp2p) from 0.14.3 to 0.14.4.
- [Release notes](https://github.com/libp2p/go-libp2p/releases)
- [Commits](https://github.com/libp2p/go-libp2p/compare/v0.14.3...v0.14.4)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-quic-transport from 0.11.1 to 0.11.2

Bumps [github.com/libp2p/go-libp2p-quic-transport](https://github.com/libp2p/go-libp2p-quic-transport) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/libp2p/go-libp2p-quic-transport/releases)
- [Commits](https://github.com/libp2p/go-libp2p-quic-transport/compare/v0.11.1...v0.11.2)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-quic-transport
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/google/uuid from 1.2.0 to 1.3.0

Bumps [github.com/google/uuid](https://github.com/google/uuid) from 1.2.0 to 1.3.0.
- [Release notes](https://github.com/google/uuid/releases)
- [Commits](https://github.com/google/uuid/compare/v1.2.0...v1.3.0)

---
updated-dependencies:
- dependency-name: github.com/google/uuid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-peerstore from 0.2.7 to 0.2.8

Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.2.7 to 0.2.8.
- [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases)
- [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.2.7...v0.2.8)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-peerstore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-datastore from 0.4.5 to 0.4.6

Bumps [github.com/ipfs/go-datastore](https://github.com/ipfs/go-datastore) from 0.4.5 to 0.4.6.
- [Release notes](https://github.com/ipfs/go-datastore/releases)
- [Commits](https://github.com/ipfs/go-datastore/compare/v0.4.5...v0.4.6)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-datastore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-fs-lock from 0.0.6 to 0.0.7

Bumps [github.com/ipfs/go-fs-lock](https://github.com/ipfs/go-fs-lock) from 0.0.6 to 0.0.7.
- [Release notes](https://github.com/ipfs/go-fs-lock/releases)
- [Commits](https://github.com/ipfs/go-fs-lock/compare/v0.0.6...v0.0.7)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-fs-lock
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-ipns from 0.1.0 to 0.1.2

Bumps [github.com/ipfs/go-ipns](https://github.com/ipfs/go-ipns) from 0.1.0 to 0.1.2.
- [Release notes](https://github.com/ipfs/go-ipns/releases)
- [Commits](https://github.com/ipfs/go-ipns/compare/v0.1.0...v0.1.2)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipns
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-noise from 0.2.0 to 0.2.2

Bumps [github.com/libp2p/go-libp2p-noise](https://github.com/libp2p/go-libp2p-noise) from 0.2.0 to 0.2.2.
- [Release notes](https://github.com/libp2p/go-libp2p-noise/releases)
- [Commits](https://github.com/libp2p/go-libp2p-noise/compare/v0.2.0...v0.2.2)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-noise
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-pubsub from 0.4.1 to 0.5.4

Bumps [github.com/libp2p/go-libp2p-pubsub](https://github.com/libp2p/go-libp2p-pubsub) from 0.4.1 to 0.5.4.
- [Release notes](https://github.com/libp2p/go-libp2p-pubsub/releases)
- [Commits](https://github.com/libp2p/go-libp2p-pubsub/compare/v0.4.1...v0.5.4)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-pubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update go-ipfs-pinner

* Bump github.com/libp2p/go-libp2p-kad-dht from 0.12.2 to 0.13.0

Bumps [github.com/libp2p/go-libp2p-kad-dht](https://github.com/libp2p/go-libp2p-kad-dht) from 0.12.2 to 0.13.0.
- [Release notes](https://github.com/libp2p/go-libp2p-kad-dht/releases)
- [Commits](https://github.com/libp2p/go-libp2p-kad-dht/compare/v0.12.2...v0.13.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-kad-dht
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-path from 0.0.9 to 0.1.0

Bumps [github.com/ipfs/go-path](https://github.com/ipfs/go-path) from 0.0.9 to 0.1.0.
- [Release notes](https://github.com/ipfs/go-path/releases)
- [Commits](https://github.com/ipfs/go-path/compare/v0.0.9...v0.1.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-path
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-raft from 0.1.7 to 0.1.8

Bumps [github.com/libp2p/go-libp2p-raft](https://github.com/libp2p/go-libp2p-raft) from 0.1.7 to 0.1.8.
- [Release notes](https://github.com/libp2p/go-libp2p-raft/releases)
- [Commits](https://github.com/libp2p/go-libp2p-raft/compare/v0.1.7...v0.1.8)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-raft
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-tls from 0.1.3 to 0.2.0

Bumps [github.com/libp2p/go-libp2p-tls](https://github.com/libp2p/go-libp2p-tls) from 0.1.3 to 0.2.0.
- [Release notes](https://github.com/libp2p/go-libp2p-tls/releases)
- [Commits](https://github.com/libp2p/go-libp2p-tls/compare/v0.1.3...v0.2.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-tls
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-merkledag from 0.3.2 to 0.4.0

Bumps [github.com/ipfs/go-merkledag](https://github.com/ipfs/go-merkledag) from 0.3.2 to 0.4.0.
- [Release notes](https://github.com/ipfs/go-merkledag/releases)
- [Commits](https://github.com/ipfs/go-merkledag/compare/v0.3.2...v0.4.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-merkledag
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Chore: dependency upgrades

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-16 17:52:31 +02:00
Hector Sanjuan
263c17c383 Changelog for v0.14.1 2021-08-16 17:51:57 +02:00
Hector Sanjuan
9fb80b3acb
Merge pull request #1444 from ipfs/fix/1427-wait-pin 
Change --wait behaviour to return on replication-factor-min
 2021-08-16 16:38:17 +02:00
Hector Sanjuan
3826620b81
Merge pull request #1446 from ipfs/fix/1409-msgpack-decode-error 
Fix #1409: Better support of metrics from older peers
 2021-08-16 16:36:20 +02:00
Hector Sanjuan
e2659af8f6 ctl: WaitFor replication factor min to be reached 
Fixes #1427. Currently, if --wait is used when pinning it will wait until all
statuses reported for a pin are either Pinned or Remote. If a peer was lagging
behind and not syncing the state properly (reporting "unpinned" for example),
that would be enough to block waiting.

This modifies the behaviour of wait to return when replication_factor_min is
reached, regardless of what other statuses are.
 2021-08-13 11:35:28 +02:00
Hector Sanjuan
c7847e9677 rest/client: WaitFor(): support waiting for N peers to reach target. 
This commit updates the WaitFor StatusFilterParams to add a Limit field, and
reworks the waiting logic to simplify it.

The Limit parameter causes WaitFor to return successfully when N number of
peers have reached the target status, regardless of what other statuses are
(as long as they are not errors)

Other than that, the function logic should remain the same, blocking until all
statuses are the target, with a special case about target-Pinned, which causes
Remote statuses to be ignored.
 2021-08-13 11:24:54 +02:00
Hector Sanjuan
4e61722394
Merge pull request #1445 from ipfs/fix/1415-307-redirect 
restapi: fix #1415: paths ending in / get 307-redirect instead of 301
 2021-08-12 17:02:31 +02:00
Hector Sanjuan
ecf287c8e6 Fix #1409: Better support of metrics from older peers 
Before: receiving a metric from a peer <= 0.13.3 causes decoding error on logs.
Now: metric is correctly parsed and a warning message is printed once.
 2021-08-12 00:03:05 +02:00
Hector Sanjuan
70386ea36a restapi: fix #1415: paths ending in / get 307-redirect instead of 301 
The Gorilla muxer StrictSlash option uses a 301 permanent redirect, which
results in POST requests becoming GET requests in most clients.  Thus we use
our own middleware that performs a 307 redirect.  See issue #1415 for more
details.
 2021-08-11 23:20:45 +02:00
Hector Sanjuan
7e5eac3ce0
Merge pull request #1438 from ipfs/fix/1436-recover-boot-bis 
Fix #1436: Do not block peer startup waiting for RecoverAll
 2021-08-06 14:46:30 +02:00
Hector Sanjuan
cd8938582c
Merge pull request #1439 from ipfs/fix/logging-facs-extra 
Fix: improve log level handling
 2021-08-06 14:46:19 +02:00
Hector Sanjuan
2f21724d1d logging: correctly use defaults for LoggingFacilitiesExtra 2021-08-06 11:31:05 +02:00
Hector Sanjuan
e4ea2fab78 Logging: Set badger's logging level to INFO by default 2021-08-06 11:30:58 +02:00
Hector Sanjuan
67497c4eb4 Fix #1436: Do not block peer startup waiting for RecoverAll 
On large pinsets this may take a very long time and prevents metrics and
re-boostrapping from starting, among other things. See bug description.

This lets watchPinset trigger an immediate RecoverAllLocal instead, but this
happens in its own goroutine and should allow everything else to start.
 2021-08-06 11:30:29 +02:00
Hector Sanjuan
0c01079eca
Merge pull request #1428 from iand/fix/dataraces 
Fix data races
 2021-08-04 17:10:32 +02:00
Ian Davis
cda59b656e
Guard request counter in ipfs mock with mutex 
Fixes several data races across multiple tests.
 2021-07-28 13:51:06 +01:00
Ian Davis
cb4023855c
Ensure read of alerts slice is performed while holding lock 
Fixes data race:

==================
WARNING: DATA RACE
Read at 0x00c029ae7f10 by goroutine 4785:
  github.com/ipfs/ipfs-cluster.(*Cluster).Alerts()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:395 +0x64
  github.com/ipfs/ipfs-cluster.TestClusterAlerts()
      /home/iand/wip/iand/ipfs-cluster/ipfscluster_test.go:2159 +0x238
  testing.tRunner()
      /opt/go/src/testing/testing.go:1194 +0x202

Previous write at 0x00c029ae7f10 by goroutine 5062:
  github.com/ipfs/ipfs-cluster.(*Cluster).alertsHandler()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:429 +0x48c
  github.com/ipfs/ipfs-cluster.(*Cluster).run.func5()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:596 +0x76

Goroutine 4785 (running) created at:
  testing.(*T).Run()
      /opt/go/src/testing/testing.go:1239 +0x5d7
  testing.runTests.func1()
      /opt/go/src/testing/testing.go:1512 +0xa6
  testing.tRunner()
      /opt/go/src/testing/testing.go:1194 +0x202
  testing.runTests()
      /opt/go/src/testing/testing.go:1510 +0x612
  testing.(*M).Run()
      /opt/go/src/testing/testing.go:1418 +0x3b3
  github.com/ipfs/ipfs-cluster.TestMain()
      /home/iand/wip/iand/ipfs-cluster/ipfscluster_test.go:134 +0x7dc
  main.main()
      _testmain.go:179 +0x271

Goroutine 5062 (running) created at:
  github.com/ipfs/ipfs-cluster.(*Cluster).run()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:594 +0x1f6
  github.com/ipfs/ipfs-cluster.NewCluster.func1()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:208 +0xa4
==================
--- FAIL: TestClusterAlerts (8.69s)
    testing.go:1093: race detected during execution of test
 2021-07-28 13:08:56 +01:00
Ian Davis
33f1334c4a
Guard access to informer rpc client with mutex 
Fixes data race

==================
WARNING: DATA RACE
Read at 0x00c02be544a8 by goroutine 4718:
  github.com/ipfs/ipfs-cluster/informer/disk.(*Informer).GetMetric()
      /home/iand/wip/iand/ipfs-cluster/informer/disk/disk.go:75 +0x135
  github.com/ipfs/ipfs-cluster.(*Cluster).sendInformerMetric()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:279 +0x149
  github.com/ipfs/ipfs-cluster.(*Cluster).pushInformerMetrics()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:324 +0x264
  github.com/ipfs/ipfs-cluster.(*Cluster).run.func3()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:583 +0xba

Previous write at 0x00c02be544a8 by goroutine 3049:
  github.com/ipfs/ipfs-cluster/informer/disk.(*Informer).Shutdown()
      /home/iand/wip/iand/ipfs-cluster/informer/disk/disk.go:65 +0x10e
  github.com/ipfs/ipfs-cluster.(*Cluster).Shutdown()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:768 +0x9d7
  github.com/ipfs/ipfs-cluster.shutdownCluster()
      /home/iand/wip/iand/ipfs-cluster/ipfscluster_test.go:420 +0x64
  github.com/ipfs/ipfs-cluster.shutdownClusters()
      /home/iand/wip/iand/ipfs-cluster/ipfscluster_test.go:414 +0x74
  github.com/ipfs/ipfs-cluster.TestClustersReplicationRealloc()
      /home/iand/wip/iand/ipfs-cluster/ipfscluster_test.go:1680 +0x10c6
  testing.tRunner()
      /opt/go/src/testing/testing.go:1194 +0x202

Goroutine 4718 (running) created at:
  github.com/ipfs/ipfs-cluster.(*Cluster).run()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:581 +0x16d
  github.com/ipfs/ipfs-cluster.NewCluster.func1()
      /home/iand/wip/iand/ipfs-cluster/cluster.go:208 +0xa4

Goroutine 3049 (running) created at:
  testing.(*T).Run()
      /opt/go/src/testing/testing.go:1239 +0x5d7
  testing.runTests.func1()
      /opt/go/src/testing/testing.go:1512 +0xa6
  testing.tRunner()
      /opt/go/src/testing/testing.go:1194 +0x202
  testing.runTests()
      /opt/go/src/testing/testing.go:1510 +0x612
  testing.(*M).Run()
      /opt/go/src/testing/testing.go:1418 +0x3b3
  github.com/ipfs/ipfs-cluster.TestMain()
      /home/iand/wip/iand/ipfs-cluster/ipfscluster_test.go:134 +0x7dc
  main.main()
      _testmain.go:179 +0x271
==================
--- FAIL: TestClustersReplicationRealloc (13.60s)
    ipfscluster_test.go:1641: Shutting down QmYc1fpiBd7owHgDsnr42E7XtKZCTVjjm7FLnQMozcKid3
    testing.go:1093: race detected during execution of test
 2021-07-28 13:05:23 +01:00
Hector Sanjuan
ea4dd7ed98
Set development version v0.14.0-next 2021-07-09 00:25:18 +02:00
Hector Sanjuan
f7e6f4a26c
Release v0.14.0 2021-07-09 00:24:00 +02:00
Hector Sanjuan
78aa96449a Final update to changelog for v0.14.0 2021-07-09 00:23:30 +02:00
Hector Sanjuan
d3e5d87ab5
Merge pull request #1400 from ipfs/v0.14.0/changelog 
Changelog for v0.14.0
 2021-07-08 23:57:13 +02:00
Hector Sanjuan
7a954e4fd2
Merge pull request #1405 from ipfs/feat/speed-up-pin-ls 
restapi: allocationsHandler: skip filtering pins when not needed.
 2021-07-08 23:56:07 +02:00
Hector Sanjuan
b2b42c29ae
Merge pull request #1404 from ipfs/fix/1403-ctl-https 
Fix #1403: ctl: ssl errors when talking to https remote peers
 2021-07-08 23:55:55 +02:00
Hector Sanjuan
dc6c05ae51 Changelog for v0.14.0 2021-07-08 23:43:59 +02:00
Hector Sanjuan
3ba7d79d0a restapi: allocationsHandler: skip filtering pins when not needed. 
The restapi component supports filters for the pinset. This was done to keep
expected output when sharding was fully supported by filtering out "internal"
pins.

However this filter requires looping on the full pinset and re-allocating  and
usually does nothing. The useless copy is significant for really big pinsets.

Additionally, ipfs-cluster-ctl set the filter by default to "pins". By setting
it to "all" instead we can skip the whole filtering step and, in practice, get the
same results.
 2021-07-08 23:27:55 +02:00
Hector Sanjuan
33c7b5d81a Fix #1403: ctl: ssl errors when talking to https remote peers 
When using SSL and not talking to libp2p-http endpoints, we should not
resolve the dns names in the multiaddresses as otherwise we cannot
verify the https certificates used by the remote endpoint.
 2021-07-08 23:21:36 +02:00
Hector Sanjuan
3ba27bebb4
Release candidate v0.14.0-rc3 2021-07-08 12:48:30 +02:00
Hector Sanjuan
164e674fcd
Merge pull request #1399 from ipfs/fix/1360-efficient-pin-status-followup 
pintracker: avoid listing the state unless necessary
 2021-07-08 02:24:10 +02:00
Hector Sanjuan
3e0a622fe1
Merge pull request #1398 from ipfs/fix/improve-defaults 
new defaults: disable_repinning:true and badger as default datastore
 2021-07-08 01:03:32 +02:00
Hector Sanjuan
27569bdf88 pintracker: avoid listing the state unless necessary 
This is a follow up to #1360 which further optimizes StatusAll calls by
avoiding listing and filtering the cluster state when requesting status for
operations that should be direclty in the operation tracker because they are
ongoing (queued, pinning, unpinning, error).
 2021-07-08 01:01:25 +02:00
Hector Sanjuan
aba8e8e892 config: set disable_repinning to true by default. 
Repinnings happen unexpectedly and just bite people during normal
rollouts. This feature, while useful in some cases, should be enabled by
people that are aware of the consequences.
 2021-07-08 00:40:24 +02:00
Hector Sanjuan
46a14f9602 Set badger as default datastore for ipfs-cluster-service 
This goes back to badger as the default. Testing performed on 0.14.0-rc2
showed very successful GC behaviour on large clusters.

Being that we run with badger at scale and that the main problem seems
resolved, it seems appropiate to default to badger.
 2021-07-08 00:37:03 +02:00
Hector Sanjuan
493089d6d0
Release candidate v0.14.0-rc2 2021-07-06 16:56:53 +02:00
Hector Sanjuan
adb15feb6e
Dependency upgrades (#1395) 
* build(deps): bump github.com/multiformats/go-multiaddr-dns

Bumps [github.com/multiformats/go-multiaddr-dns](https://github.com/multiformats/go-multiaddr-dns) from 0.2.0 to 0.3.1.
- [Release notes](https://github.com/multiformats/go-multiaddr-dns/releases)
- [Commits](https://github.com/multiformats/go-multiaddr-dns/compare/v0.2.0...v0.3.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/hashicorp/go-hclog from 0.15.0 to 0.16.0

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.15.0...v0.16.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/ipfs/go-unixfs from 0.2.4 to 0.2.5

Bumps [github.com/ipfs/go-unixfs](https://github.com/ipfs/go-unixfs) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/ipfs/go-unixfs/releases)
- [Commits](https://github.com/ipfs/go-unixfs/compare/v0.2.4...v0.2.5)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/libp2p/go-libp2p-peerstore

Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases)
- [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.2.6...v0.2.7)

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump go.uber.org/multierr from 1.6.0 to 1.7.0

Bumps [go.uber.org/multierr](https://github.com/uber-go/multierr) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/uber-go/multierr/releases)
- [Changelog](https://github.com/uber-go/multierr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/multierr/compare/v1.6.0...v1.7.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Chore: update deps

* Update changelog

* Update to go1.16. Downgrade unixfs.

* go mod tidy

* travis: use go install

* golint no more

* Update configuration for dependabot

* Fix wrong dependabot config

* dependabot

* Revert update of go-unixfs

* Dependency upgrades

* Bump github.com/libp2p/go-libp2p-gorpc from 0.1.2 to 0.1.3

Bumps [github.com/libp2p/go-libp2p-gorpc](https://github.com/libp2p/go-libp2p-gorpc) from 0.1.2 to 0.1.3.
- [Release notes](https://github.com/libp2p/go-libp2p-gorpc/releases)
- [Commits](https://github.com/libp2p/go-libp2p-gorpc/compare/v0.1.2...v0.1.3)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-gorpc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix deprecated objects with prometheus

* chore: update dependencies

* monitor: remove dependency to go-multicodec

go-multicodec has been deprecated and it was just a wrapper.

This switches directly to ugorji/go/codec's msgpack for cluster metrics
serialization.

* Upgrade mfs so it works with latest go-unixfs

* Bump github.com/ugorji/go/codec from 1.2.5 to 1.2.6 (#1391)

Bumps [github.com/ugorji/go/codec](https://github.com/ugorji/go) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/ugorji/go/releases)
- [Commits](https://github.com/ugorji/go/compare/v1.2.5...v1.2.6)

---
updated-dependencies:
- dependency-name: github.com/ugorji/go/codec
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/hashicorp/go-hclog from 0.16.0 to 0.16.1 (#1392)

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.16.0 to 0.16.1.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.16.0...v0.16.1)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-06 16:47:04 +02:00
Hector Sanjuan
a5ae9fffc4
Release candidate v0.14.0-rc1 2021-07-06 14:07:21 +02:00
Hector Sanjuan
3a8098260b
Dependency upgrades (#1378) 
* build(deps): bump github.com/multiformats/go-multiaddr-dns

Bumps [github.com/multiformats/go-multiaddr-dns](https://github.com/multiformats/go-multiaddr-dns) from 0.2.0 to 0.3.1.
- [Release notes](https://github.com/multiformats/go-multiaddr-dns/releases)
- [Commits](https://github.com/multiformats/go-multiaddr-dns/compare/v0.2.0...v0.3.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/hashicorp/go-hclog from 0.15.0 to 0.16.0

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.15.0...v0.16.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/ipfs/go-unixfs from 0.2.4 to 0.2.5

Bumps [github.com/ipfs/go-unixfs](https://github.com/ipfs/go-unixfs) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/ipfs/go-unixfs/releases)
- [Commits](https://github.com/ipfs/go-unixfs/compare/v0.2.4...v0.2.5)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* build(deps): bump github.com/libp2p/go-libp2p-peerstore

Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases)
- [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.2.6...v0.2.7)

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump go.uber.org/multierr from 1.6.0 to 1.7.0

Bumps [go.uber.org/multierr](https://github.com/uber-go/multierr) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/uber-go/multierr/releases)
- [Changelog](https://github.com/uber-go/multierr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/multierr/compare/v1.6.0...v1.7.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Chore: update deps

* Update changelog

* Update to go1.16. Downgrade unixfs.

* go mod tidy

* travis: use go install

* golint no more

* Update configuration for dependabot

* Fix wrong dependabot config

* dependabot

* Revert update of go-unixfs

* Dependency upgrades

* Bump github.com/libp2p/go-libp2p-gorpc from 0.1.2 to 0.1.3

Bumps [github.com/libp2p/go-libp2p-gorpc](https://github.com/libp2p/go-libp2p-gorpc) from 0.1.2 to 0.1.3.
- [Release notes](https://github.com/libp2p/go-libp2p-gorpc/releases)
- [Commits](https://github.com/libp2p/go-libp2p-gorpc/compare/v0.1.2...v0.1.3)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-gorpc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix deprecated objects with prometheus

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-06 14:06:04 +02:00
Hector Sanjuan
ce2490c64f
Merge pull request #1389 from ipfs/fix/db-close-tests 
Fix: tests: close datastore on cluster node shutdown.
 2021-07-06 14:05:36 +02:00
Hector Sanjuan
8ce98ceae3 Fix: tests: close datastore on cluster node shutdown. 
This resulted in too-many-files-open when running with leveldb.
 2021-07-06 12:28:03 +02:00
Hector Sanjuan
54c3608899
Merge pull request #1377 from ipfs/fix/1360-efficient-pin-status 
Fix #1360: Efficient pinset status with filters
 2021-07-06 11:57:09 +02:00
Hector Sanjuan
edfcfa3fb0 Fix #1360: Efficient pinset status with filters 
This commit modifies the pintracker StatusAll call to take a status filter.

This allows to skip a PinLs call to ipfs when checking status for items that
are queued, pinning, unpinning or in error. Those status come directly from
the operation tracker. This should result in a significant performance
increase for those calls, particularly in nodes with several hundred thousand
pins and more, where the call to IPFS is very expensive.

A new TrackerStatusUnexpectedlyUnpinned status has been introduce to
differentiate between pin errors (tracked by the operation tracker) and "lost"
items (which before were pin errors too). This new status is handled by the
Recover() operation as before.
 2021-07-06 11:34:19 +02:00