cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,827 Commits 3 Branches 70 Tags 15 MiB
36f516696f
Commit Graph

318 Commits

Author SHA1 Message Date
Hector Sanjuan
36f516696f
Release v1.0.4 2022-09-27 12:44:43 +02:00
Hector Sanjuan
ffef2cea0a service: add crdt info, mark-dirty, mark-clean commands 
Provide access to some crdt-internal operations
 2022-09-26 19:39:04 +02:00
Hector Sanjuan
003da51c7b
Release v1.0.3 2022-09-16 12:01:03 +02:00
Hector Sanjuan
29d6f69819 Add "service state crdt dot" command 
This subcommand allows to export the peer's CRDT DAG as dot file.

It also opens the door to have more crdt-specific subcommands, i.e. CAR export
etc.
 2022-09-13 16:36:10 +02:00
Hector Sanjuan
5452b59a2e
Dependency upgrades (#1755) 
* Update go-libp2p to v0.22.0

* Testing with go1.19

* build(deps): bump github.com/multiformats/go-multicodec

Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/multiformats/go-multicodec/releases)
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipld/go-car from 0.4.0 to 0.5.0

Bumps [github.com/ipld/go-car](https://github.com/ipld/go-car) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/ipld/go-car/releases)
- [Commits](https://github.com/ipld/go-car/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/ipld/go-car
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/prometheus/client_golang

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.2 to 1.13.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.2...v1.13.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/hashicorp/go-hclog from 1.2.1 to 1.3.0

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 1.2.1 to 1.3.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v1.2.1...v1.3.0)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-ds-crdt from 0.3.6 to 0.3.7

Bumps [github.com/ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/ipfs/go-ds-crdt/releases)
- [Commits](https://github.com/ipfs/go-ds-crdt/compare/v0.3.6...v0.3.7)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-crdt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli/v2 from 2.10.2 to 2.14.1

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.10.2 to 2.14.1.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.10.2...v2.14.1)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-http from 0.3.0 to 0.4.0

Bumps [github.com/libp2p/go-libp2p-http](https://github.com/libp2p/go-libp2p-http) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/libp2p/go-libp2p-http/releases)
- [Commits](https://github.com/libp2p/go-libp2p-http/compare/v0.3.0...v0.4.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-gorpc from 0.4.0 to 0.5.0

Bumps [github.com/libp2p/go-libp2p-gorpc](https://github.com/libp2p/go-libp2p-gorpc) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/libp2p/go-libp2p-gorpc/releases)
- [Commits](https://github.com/libp2p/go-libp2p-gorpc/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-gorpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump contrib.go.opencensus.io/exporter/prometheus

Bumps [contrib.go.opencensus.io/exporter/prometheus](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/releases)
- [Commits](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/compare/v0.4.1...v0.4.2)

---
updated-dependencies:
- dependency-name: contrib.go.opencensus.io/exporter/prometheus
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-raft from 0.1.8 to 0.2.0

Bumps [github.com/libp2p/go-libp2p-raft](https://github.com/libp2p/go-libp2p-raft) from 0.1.8 to 0.2.0.
- [Release notes](https://github.com/libp2p/go-libp2p-raft/releases)
- [Commits](https://github.com/libp2p/go-libp2p-raft/compare/v0.1.8...v0.2.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-raft
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli from 1.22.9 to 1.22.10

Bumps [github.com/urfave/cli](https://github.com/urfave/cli) from 1.22.9 to 1.22.10.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v1.22.9...v1.22.10)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix checker/linter/staticcheck warnings

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-09-06 16:57:17 +02:00
Hector Sanjuan
b2ce7d916d
Release v1.0.2 2022-07-06 18:26:32 +02:00
Hector Sanjuan
04177fa545
Release candidate v1.0.2-rc1 2022-06-30 14:26:54 +02:00
Hector Sanjuan
90a3b63d82 ctl: Expiration date in "pin ls" was showing wrong field (timestamp) 
Fixes #1666.
 2022-06-20 21:36:42 +02:00
Hector Sanjuan
d7107791ed Rename links to use the ipfscluster.io domain 2022-06-16 17:43:30 +02:00
Hector Sanjuan
508791b547 Migrate from ipfs/ipfs-cluster to ipfs-cluster/ipfs-cluster 
This performs the necessary renamings.
 2022-06-16 17:43:30 +02:00
Hector Sanjuan
3911f8483a Release v1.0.1 2022-06-16 17:43:30 +02:00
Hector Sanjuan
03740e5bf0 ctl: fix "pin ls" hangs for pinsets > 1024 items. 
Seems we forgot to adapt "pin ls" and run it in a background goroutine, so it
just hangs.
 2022-06-16 17:43:29 +02:00
Hector Sanjuan
4daece2b98 Feat: add a new "pinqueue" informer component 
This new component broadcasts metrics about the current size of the pinqueue,
which can in turn be used to inform allocations.

It has a weight_bucket_size option that serves to divide the actual size by a
given factor. This allows considering peers with similar queue sizes to have
the same weight.

Additionally, some changes have been made to the balanced allocator so that a
combination of tags, pinqueue sizes and free-spaces can be used. When
allocating by [<tag>, pinqueue, freespace], the allocator will prioritize
choosing peers with the smallest pin queue weight first, and of those with the
same weight, it will allocate based on freespace.
 2022-06-16 17:43:29 +02:00
Hector Sanjuan
8dc18cd3d7 Metrics: add additional metrics 
* Gauge for total number of ipfs pins
* Counter for pin/add
* Counter for pin/add errors
* Counter for Block/Puts
* Counter for blocks added
* Counter for block/added size
* Counted for block/added errors
 2022-06-16 17:43:29 +02:00
Hector Sanjuan
6f01277bd1
Release v1.0.0 2022-04-22 16:42:17 +02:00
Hector Sanjuan
288e7f9e3d ctl: Set exit status to 1 when app.Run() fails 
1 corresponds to arguments errors and anything that prevents the app from
making a request.
 2022-04-12 11:49:26 +02:00
Hector Sanjuan
e4b91af901
Release candidate v1.0.0-rc5 2022-04-11 21:29:23 +02:00
Hector Sanjuan
81fa2c5665
Release candidate v1.0.0-rc4 2022-04-07 15:06:04 +02:00
Hector Sanjuan
a97ed10d0b Adopt api.Cid type - replaces cid.Cid everwhere. 
This commit introduces an api.Cid type and replaces the usage of cid.Cid
everywhere.

The main motivation here is to override MarshalJSON so that Cids are
JSON-ified as '"Qm...."' instead of '{ "/": "Qm....." }', as this "ipld"
representation of IDs is horrible to work with, and our APIs are not issuing
IPLD objects to start with.

Unfortunately, there is no way to do this cleanly, and the best way is to just
switch everything to our own type.
 2022-04-07 14:27:39 +02:00
Hector Sanjuan
a2b89f9b3b
Release candidate v1.0.0-rc3 2022-03-28 20:44:41 +02:00
Hector Sanjuan
a10f608270
Release candidate v1.0.0-rc2 2022-03-25 13:41:04 +01:00
Hector Sanjuan
116ee23b32
Release candidate v1.0.0-rc1 2022-03-24 20:06:15 +01:00
Hector Sanjuan
eee53bfa4f Streaming Peers(): make Peers() a streaming call 
This commit makes all the changes to make Peers() a streaming call.

While Peers is usually a non problematic call, for consistency, all calls
returning collections assembled through broadcast to cluster peers are now
streaming calls.
 2022-03-23 01:27:57 +01:00
Hector Sanjuan
0d73d33ef5 Pintracker: streaming methods 
This commit continues the work of taking advantage of the streaming
capabilities in go-libp2p-gorpc by improving the ipfsconnector and pintracker
components.

StatusAll and RecoverAll methods are now streaming methods, with the REST API
output changing accordingly to produce a stream of GlobalPinInfos rather than
a json array.

pin/ls request to the ipfs daemon now use ?stream=true and avoid having to
load the full pinset map on memory. StatusAllLocal and RecoverAllLocal
requests to the pin tracker stream all the way and no longer store the full
pinset, and the full PinInfo status slice before sending it out.

We have additionally switched to a pattern where streaming methods receive the
channel as an argument, allowing the caller to decide on whether to launch a
goroutine, do buffering etc.
 2022-03-22 15:38:01 +01:00
Hector Sanjuan
9b9d76f92d Pinset streaming and method type revamp 
This commit introduces the new go-libp2p-gorpc streaming capabilities for
Cluster. The main aim is to work towards heavily reducing memory usage when
working with very large pinsets.

As a side-effect, it takes the chance to revampt all types for all public
methods so that pointers to static what should be static objects are not used
anymore. This should heavily reduce heap allocations and GC activity.

The main change is that state.List now returns a channel from which to read
the pins, rather than pins being all loaded into a huge slice.

Things reading pins have been all updated to iterate on the channel rather
than on the slice. The full pinset is no longer fully loaded onto memory for
things that run regularly like StateSync().

Additionally, the /allocations endpoint of the rest API no longer returns an
array of pins, but rather streams json-encoded pin objects directly. This
change has extended to the restapi client (which puts pins into a channel as
they arrive) and to ipfs-cluster-ctl.

There are still pending improvements like StatusAll() calls which should also
stream responses, and specially BlockPut calls which should stream blocks
directly into IPFS on a single call.

These are coming up in future commits.
 2022-03-19 03:02:55 +01:00
Hector Sanjuan
333b933cd3 service: load pinsvc api first (just order consistency) 2022-03-11 00:45:44 +01:00
Hector Sanjuan
fd57a032e3 service: randomize pinsvcapi ports with --randomports too 2022-03-11 00:07:26 +01:00
Hector Sanjuan
5b0d9d68e3 Merge branch 'master' into feat/pinning-api 2022-03-10 13:41:54 +01:00
Hector Sanjuan
00b7925e78 ipfs-cluster-ctl: support "status cid1 cid2 ..." 2022-03-01 13:25:07 +01:00
Hector Sanjuan
3c56576439
Set development version v0.14.5-next 2022-02-16 17:12:14 +01:00
Hector Sanjuan
e95c4430ee
Release v0.14.5 2022-02-16 17:12:00 +01:00
Hector Sanjuan
2794bd89e8 Fix #1570: fix comment description about graph nodes' color 2022-02-14 14:54:24 +01:00
Hector Sanjuan
5f746f870b
Release candidate v0.14.5-rc1 2022-02-02 01:08:22 +01:00
Hector Sanjuan
524e2ecbc0 Wire in pinsvcapi to ipfs-cluster-service 2022-02-02 00:52:39 +01:00
Hector Sanjuan
dca8923aaa
Set development version v0.14.4-next 2022-01-11 19:13:39 +01:00
Hector Sanjuan
d80a8b42c8
Release v0.14.4 2022-01-11 19:13:28 +01:00
Hector Sanjuan
e7ae13e6cf service: add buffered io for import and export 2022-01-04 18:15:06 +01:00
Hector Sanjuan
e22655a3dc
Set development version v0.14.3-next 2022-01-03 18:01:52 +01:00
Hector Sanjuan
9704bb62e6
Release v0.14.3 2022-01-03 18:00:43 +01:00
Hector Sanjuan
f4c3fc4ce5 State import: allow replication factor and allocations overwrite on import 2021-12-14 11:21:19 +01:00
Hector Sanjuan
5a1d835d90
Set development version v0.14.2-next 2021-12-09 10:02:37 +01:00
Hector Sanjuan
c7cbdee01f
Release v0.14.2 2021-12-09 10:01:33 +01:00
Hector Sanjuan
54d696da60 ipfs-cluster-ctl: improve printing of health metrics 
Make sure we print the value in all cases. Use the weight for the free-space
metric, no need to parse the value anymore.
 2021-12-08 12:01:59 +01:00
Hector Sanjuan
ea8b392b72
Release candidate v0.14.2-rc1 2021-12-01 02:04:07 +01:00
Hector Sanjuan
7cf40de354 pintracker: Fix attempt count not increasing. Expose priority queueing. 
"RetryCount" has been renamed to "AttemptCount", because it counts attempts.
 2021-11-30 04:20:35 +01:00
Hector Sanjuan
29c277b67f Pintracker: add and track retry counts in the operation manager. 
Report retry count in the PinStatus
 2021-11-30 04:20:35 +01:00
Hector Sanjuan
a74b6faf96
Merge pull request #1484 from ipfs/feat/pin-time 
Add: include a timestamp that tracks when a Pin was added to the state
 2021-10-27 10:42:59 +02:00
Hector Sanjuan
c776051397 ctl: Include "added" column in pin ls command 
Also, change format from humanize to "yyyy-mm-dd HH:MM:SS" for times.
 2021-10-20 16:55:57 +02:00
Hector Sanjuan
3f895df8cf Fix: backwards compatibility for allocator default settings 
When the allocator is not defined in the configuration, it will take defaults
and assume there is a "tags" informer. That is not the case. When not defined,
we assume it should allocate only by "freespace".
 2021-10-20 16:55:35 +02:00
Hector Sanjuan
af9313d4fe Remove test files that should have not been committed 2021-10-19 13:51:17 +02:00