cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,464 Commits 3 Branches 70 Tags 15 MiB
0a146dae76
Commit Graph

2464 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Hector Sanjuan
0a146dae76 pintracker: support a priority channel for pinning 2021-11-30 04:20:35 +01:00
Hector Sanjuan
29c277b67f Pintracker: add and track retry counts in the operation manager. 
Report retry count in the PinStatus
 2021-11-30 04:20:35 +01:00
Hector Sanjuan
be18c645fa
Merge pull request #1491 from ipfs/dependency-upgrades 
Dependency upgrades
 2021-10-27 18:51:04 +02:00
Hector Sanjuan
555b624beb Dependency upgrades 2021-10-27 16:03:59 +02:00
Hector Sanjuan
3ddda1fb59 Merge branch 'master' into dependency-upgrades 2021-10-27 15:55:34 +02:00
Hector Sanjuan
a74b6faf96
Merge pull request #1484 from ipfs/feat/pin-time 
Add: include a timestamp that tracks when a Pin was added to the state
 2021-10-27 10:42:59 +02:00
Hector Sanjuan
f04afda51f
Merge pull request #1485 from ipfs/fix/default-allocator-compat 
Fix: backwards compatibility of the new allocator
 2021-10-27 10:38:05 +02:00
Hector Sanjuan
24d8b59609 Fix config test 2021-10-20 17:12:21 +02:00
Hector Sanjuan
53342d3c55 Fix go-multiaddr-net 2021-10-20 16:56:24 +02:00
Hector Sanjuan
32386d853a Dependency upgrades 2021-10-20 16:56:24 +02:00
dependabot[bot]
067db00d7a Bump github.com/multiformats/go-multihash from 0.0.15 to 0.0.16 
Bumps [github.com/multiformats/go-multihash](https://github.com/multiformats/go-multihash) from 0.0.15 to 0.0.16.
- [Release notes](https://github.com/multiformats/go-multihash/releases)
- [Commits](https://github.com/multiformats/go-multihash/compare/v0.0.15...v0.0.16)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multihash
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-20 16:56:24 +02:00
dependabot[bot]
8ce4b55607 Bump github.com/ipfs/go-path from 0.1.0 to 0.1.1 
Bumps [github.com/ipfs/go-path](https://github.com/ipfs/go-path) from 0.1.0 to 0.1.1.
- [Release notes](https://github.com/ipfs/go-path/releases)
- [Commits](https://github.com/ipfs/go-path/compare/v0.1.0...v0.1.1)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-path
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-20 16:56:24 +02:00
dependabot[bot]
7ece0473fa Bump github.com/libp2p/go-libp2p-quic-transport from 0.11.2 to 0.12.0 
Bumps [github.com/libp2p/go-libp2p-quic-transport](https://github.com/libp2p/go-libp2p-quic-transport) from 0.11.2 to 0.12.0.
- [Release notes](https://github.com/libp2p/go-libp2p-quic-transport/releases)
- [Commits](https://github.com/libp2p/go-libp2p-quic-transport/compare/v0.11.2...v0.12.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-quic-transport
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-20 16:56:24 +02:00
dependabot[bot]
e4a197796c Bump github.com/ipfs/go-ipfs-config from 0.15.0 to 0.16.0 
Bumps [github.com/ipfs/go-ipfs-config](https://github.com/ipfs/go-ipfs-config) from 0.15.0 to 0.16.0.
- [Release notes](https://github.com/ipfs/go-ipfs-config/releases)
- [Commits](https://github.com/ipfs/go-ipfs-config/compare/v0.15.0...v0.16.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipfs-config
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-20 16:56:24 +02:00
dependabot[bot]
42c363bbc2 Bump github.com/ipfs/go-cid from 0.0.7 to 0.1.0 
Bumps [github.com/ipfs/go-cid](https://github.com/ipfs/go-cid) from 0.0.7 to 0.1.0.
- [Release notes](https://github.com/ipfs/go-cid/releases)
- [Commits](https://github.com/ipfs/go-cid/compare/v0.0.7...v0.1.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-cid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-20 16:56:24 +02:00
Hector Sanjuan
c776051397 ctl: Include "added" column in pin ls command 
Also, change format from humanize to "yyyy-mm-dd HH:MM:SS" for times.
 2021-10-20 16:55:57 +02:00
Hector Sanjuan
e9857652f2 Add a timestamp to Pins 
This adds a Timestamp field to the pin objects. This allows to track when they were pinned.

This:

* Allows the pin-tracker to actually show accurate information on when the pin
  entered the system for pins that are not part of ongoing operations
  (currently it shows time.Now())
* Adds support for reporting timestamp on a pinning services api.
 2021-10-20 16:55:57 +02:00
Hector Sanjuan
3f895df8cf Fix: backwards compatibility for allocator default settings 
When the allocator is not defined in the configuration, it will take defaults
and assume there is a "tags" informer. That is not the case. When not defined,
we assume it should allocate only by "freespace".
 2021-10-20 16:55:35 +02:00
Hector Sanjuan
1dca6e0eca allocator: balanced allocator name should be "balanced" 
It was an oversight to not rename this.
 2021-10-20 16:55:35 +02:00
Hector Sanjuan
a741e3067d
Merge pull request #1486 from ipfs/feat/github-actions 
Move testing pipeline to github actions.
 2021-10-20 16:54:08 +02:00
Hector Sanjuan
066b1f4514 Move testing pipeline to github actions. 
Since travis cannot stop sucking.
 2021-10-20 16:52:46 +02:00
Hector Sanjuan
af9313d4fe Remove test files that should have not been committed 2021-10-19 13:51:17 +02:00
Hector Sanjuan
96db605c50
Merge pull request #1468 from ipfs/fix/159-improved-allocators 
Add tags informer and enable partition-based peer allocations for intelligent distribution
 2021-10-06 14:35:16 +02:00
Hector Sanjuan
db00d651bf Balanced allocator: weight-based ordering of partitions 
This fixes the issue about partitions not being picked based
on the amount of freespace available in them.

It additionally removes the metrics registry and carries information directly
in the metric.

Metrics have two additional fields: Weight and Partitionable.

Informers have been updated to make use of these fields. Partitions have
weights that equals to the weight of the metrics under them.

Older cluster versions will not set these fields. Partitionable is false by
default and weight has a GetWeight() function to convert value->weight when
unset. This provides backwards compatibility for the freespace metric.
 2021-10-06 14:10:06 +02:00
Hector Sanjuan
26e229df94 Rename allocator/metrics to allocator/balanced 2021-10-06 11:26:38 +02:00
Hector Sanjuan
6b31f44351 Address most comments from PR review 2021-10-05 14:04:28 +02:00
Hector Sanjuan
0dfe1ae063
Merge pull request #1471 from ipfs/feat/refactor-api 
API: Refactor REST API to enable re-use of functionality.
 2021-09-16 16:10:20 +02:00
Hector Sanjuan
63972f2b2e API: Refactor REST API. Extract all functionality. 
This is a preparatory PR to add additional APIs (Pinning Service API) easily
to cluster.

Instead of copy-pasting most of what the REST API does, I have refactored so
that the whole configuration, routing and request-handling utilities can be
re-used.

The worst part has been to divide the test between tests that test core
(common.API) functionality and tests that test specific REST API endpoint
functionality. I could not get away without an additional common/test package
to provide functions that are used from both places. This is a side effect of
testing both http and libp2p endpoints for every request etc.
 2021-09-16 15:52:25 +02:00
Hector Sanjuan
cf4fb74993 service/follow: Enable new metrics allocator 2021-09-15 21:49:26 +02:00
Hector Sanjuan
1c0abde8a5 Informer: add tags informer 
The tags informer produces metrics in the form tags:name/value
with the name and values defined in its configuration.
 2021-09-15 20:07:37 +02:00
Hector Sanjuan
ea5e18078c Informers: GetMetric() -> GetMetrics() 
Support returning multiple metrics per informer.
 2021-09-15 20:07:37 +02:00
Hector Sanjuan
b6a46cd8a4 allocator: rework the whole allocator system 
The new "metrics" allocator is about to partition metrics and distribe
allocations among the partitions.

For example: given a region, an availability zone and free space on disk, the
allocator would be able to choose allocations by distributing among regions
and availability zones as much as possible, and for those peers in the same
region/az, selecting those with most free space first.

This requires a major overhaul of the allocator component.
 2021-09-13 12:24:00 +02:00
Hector Sanjuan
4060f4196b Revert "Informer/disk: deprecate RepoSize metric" 
This reverts commit 3e54c4c695.
 2021-09-10 18:53:59 +02:00
Hector Sanjuan
3e54c4c695 Informer/disk: deprecate RepoSize metric 
The allocator is hardcoded to descendalloc for freespace so it is not even useful
as it would allocate to peers with largest reposize first no matter what.

We are, in any case, reworking allocators and informers etc.
 2021-09-08 17:36:54 +02:00
Hector Sanjuan
396a348a65
Merge pull request #1462 from ipfs/fix/1461-flaky-test 
Adder: try to make test less flaky
 2021-08-30 12:04:43 +02:00
Hector Sanjuan
26007114b4 Adder: try to make test less flaky 2021-08-29 01:16:28 +02:00
Hector Sanjuan
f9b20caa2a changelog: fix v0.14.1 release date 2021-08-17 10:22:36 +02:00
Hector Sanjuan
267deeff28
Set development version v0.14.1-next 2021-08-16 18:03:18 +02:00
Hector Sanjuan
0162df60eb
Release v0.14.1 2021-08-16 18:02:52 +02:00
Hector Sanjuan
506b1657b9
Merge pull request #1452 from ipfs/v0.14.1/changelog 
Changelog for v0.14.1
 2021-08-16 17:54:52 +02:00
Hector Sanjuan
3f5490424b
Dependency upgrades (#1451) 
* Bump github.com/hashicorp/go-hclog from 0.16.1 to 0.16.2

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.16.1 to 0.16.2.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.16.1...v0.16.2)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-log/v2 from 2.2.0 to 2.3.0

Bumps [github.com/ipfs/go-log/v2](https://github.com/ipfs/go-log) from 2.2.0 to 2.3.0.
- [Release notes](https://github.com/ipfs/go-log/releases)
- [Commits](https://github.com/ipfs/go-log/compare/v2.2.0...v2.3.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-log/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p from 0.14.3 to 0.14.4

Bumps [github.com/libp2p/go-libp2p](https://github.com/libp2p/go-libp2p) from 0.14.3 to 0.14.4.
- [Release notes](https://github.com/libp2p/go-libp2p/releases)
- [Commits](https://github.com/libp2p/go-libp2p/compare/v0.14.3...v0.14.4)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-quic-transport from 0.11.1 to 0.11.2

Bumps [github.com/libp2p/go-libp2p-quic-transport](https://github.com/libp2p/go-libp2p-quic-transport) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/libp2p/go-libp2p-quic-transport/releases)
- [Commits](https://github.com/libp2p/go-libp2p-quic-transport/compare/v0.11.1...v0.11.2)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-quic-transport
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/google/uuid from 1.2.0 to 1.3.0

Bumps [github.com/google/uuid](https://github.com/google/uuid) from 1.2.0 to 1.3.0.
- [Release notes](https://github.com/google/uuid/releases)
- [Commits](https://github.com/google/uuid/compare/v1.2.0...v1.3.0)

---
updated-dependencies:
- dependency-name: github.com/google/uuid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-peerstore from 0.2.7 to 0.2.8

Bumps [github.com/libp2p/go-libp2p-peerstore](https://github.com/libp2p/go-libp2p-peerstore) from 0.2.7 to 0.2.8.
- [Release notes](https://github.com/libp2p/go-libp2p-peerstore/releases)
- [Commits](https://github.com/libp2p/go-libp2p-peerstore/compare/v0.2.7...v0.2.8)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-peerstore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-datastore from 0.4.5 to 0.4.6

Bumps [github.com/ipfs/go-datastore](https://github.com/ipfs/go-datastore) from 0.4.5 to 0.4.6.
- [Release notes](https://github.com/ipfs/go-datastore/releases)
- [Commits](https://github.com/ipfs/go-datastore/compare/v0.4.5...v0.4.6)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-datastore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-fs-lock from 0.0.6 to 0.0.7

Bumps [github.com/ipfs/go-fs-lock](https://github.com/ipfs/go-fs-lock) from 0.0.6 to 0.0.7.
- [Release notes](https://github.com/ipfs/go-fs-lock/releases)
- [Commits](https://github.com/ipfs/go-fs-lock/compare/v0.0.6...v0.0.7)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-fs-lock
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-ipns from 0.1.0 to 0.1.2

Bumps [github.com/ipfs/go-ipns](https://github.com/ipfs/go-ipns) from 0.1.0 to 0.1.2.
- [Release notes](https://github.com/ipfs/go-ipns/releases)
- [Commits](https://github.com/ipfs/go-ipns/compare/v0.1.0...v0.1.2)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipns
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-noise from 0.2.0 to 0.2.2

Bumps [github.com/libp2p/go-libp2p-noise](https://github.com/libp2p/go-libp2p-noise) from 0.2.0 to 0.2.2.
- [Release notes](https://github.com/libp2p/go-libp2p-noise/releases)
- [Commits](https://github.com/libp2p/go-libp2p-noise/compare/v0.2.0...v0.2.2)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-noise
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-pubsub from 0.4.1 to 0.5.4

Bumps [github.com/libp2p/go-libp2p-pubsub](https://github.com/libp2p/go-libp2p-pubsub) from 0.4.1 to 0.5.4.
- [Release notes](https://github.com/libp2p/go-libp2p-pubsub/releases)
- [Commits](https://github.com/libp2p/go-libp2p-pubsub/compare/v0.4.1...v0.5.4)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-pubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update go-ipfs-pinner

* Bump github.com/libp2p/go-libp2p-kad-dht from 0.12.2 to 0.13.0

Bumps [github.com/libp2p/go-libp2p-kad-dht](https://github.com/libp2p/go-libp2p-kad-dht) from 0.12.2 to 0.13.0.
- [Release notes](https://github.com/libp2p/go-libp2p-kad-dht/releases)
- [Commits](https://github.com/libp2p/go-libp2p-kad-dht/compare/v0.12.2...v0.13.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-kad-dht
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-path from 0.0.9 to 0.1.0

Bumps [github.com/ipfs/go-path](https://github.com/ipfs/go-path) from 0.0.9 to 0.1.0.
- [Release notes](https://github.com/ipfs/go-path/releases)
- [Commits](https://github.com/ipfs/go-path/compare/v0.0.9...v0.1.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-path
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-raft from 0.1.7 to 0.1.8

Bumps [github.com/libp2p/go-libp2p-raft](https://github.com/libp2p/go-libp2p-raft) from 0.1.7 to 0.1.8.
- [Release notes](https://github.com/libp2p/go-libp2p-raft/releases)
- [Commits](https://github.com/libp2p/go-libp2p-raft/compare/v0.1.7...v0.1.8)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-raft
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/libp2p/go-libp2p-tls from 0.1.3 to 0.2.0

Bumps [github.com/libp2p/go-libp2p-tls](https://github.com/libp2p/go-libp2p-tls) from 0.1.3 to 0.2.0.
- [Release notes](https://github.com/libp2p/go-libp2p-tls/releases)
- [Commits](https://github.com/libp2p/go-libp2p-tls/compare/v0.1.3...v0.2.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-tls
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump github.com/ipfs/go-merkledag from 0.3.2 to 0.4.0

Bumps [github.com/ipfs/go-merkledag](https://github.com/ipfs/go-merkledag) from 0.3.2 to 0.4.0.
- [Release notes](https://github.com/ipfs/go-merkledag/releases)
- [Commits](https://github.com/ipfs/go-merkledag/compare/v0.3.2...v0.4.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-merkledag
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Chore: dependency upgrades

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-16 17:52:31 +02:00
Hector Sanjuan
263c17c383 Changelog for v0.14.1 2021-08-16 17:51:57 +02:00
Hector Sanjuan
9fb80b3acb
Merge pull request #1444 from ipfs/fix/1427-wait-pin 
Change --wait behaviour to return on replication-factor-min
 2021-08-16 16:38:17 +02:00
Hector Sanjuan
3826620b81
Merge pull request #1446 from ipfs/fix/1409-msgpack-decode-error 
Fix #1409: Better support of metrics from older peers
 2021-08-16 16:36:20 +02:00
Hector Sanjuan
e2659af8f6 ctl: WaitFor replication factor min to be reached 
Fixes #1427. Currently, if --wait is used when pinning it will wait until all
statuses reported for a pin are either Pinned or Remote. If a peer was lagging
behind and not syncing the state properly (reporting "unpinned" for example),
that would be enough to block waiting.

This modifies the behaviour of wait to return when replication_factor_min is
reached, regardless of what other statuses are.
 2021-08-13 11:35:28 +02:00
Hector Sanjuan
c7847e9677 rest/client: WaitFor(): support waiting for N peers to reach target. 
This commit updates the WaitFor StatusFilterParams to add a Limit field, and
reworks the waiting logic to simplify it.

The Limit parameter causes WaitFor to return successfully when N number of
peers have reached the target status, regardless of what other statuses are
(as long as they are not errors)

Other than that, the function logic should remain the same, blocking until all
statuses are the target, with a special case about target-Pinned, which causes
Remote statuses to be ignored.
 2021-08-13 11:24:54 +02:00
Hector Sanjuan
4e61722394
Merge pull request #1445 from ipfs/fix/1415-307-redirect 
restapi: fix #1415: paths ending in / get 307-redirect instead of 301
 2021-08-12 17:02:31 +02:00
Hector Sanjuan
ecf287c8e6 Fix #1409: Better support of metrics from older peers 
Before: receiving a metric from a peer <= 0.13.3 causes decoding error on logs.
Now: metric is correctly parsed and a warning message is printed once.
 2021-08-12 00:03:05 +02:00
Hector Sanjuan
70386ea36a restapi: fix #1415: paths ending in / get 307-redirect instead of 301 
The Gorilla muxer StrictSlash option uses a 301 permanent redirect, which
results in POST requests becoming GET requests in most clients.  Thus we use
our own middleware that performs a 307 redirect.  See issue #1415 for more
details.
 2021-08-11 23:20:45 +02:00
Hector Sanjuan
7e5eac3ce0
Merge pull request #1438 from ipfs/fix/1436-recover-boot-bis 
Fix #1436: Do not block peer startup waiting for RecoverAll
 2021-08-06 14:46:30 +02:00