cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,661 Commits 3 Branches 70 Tags 15 MiB
58aea6c392
Commit Graph

336 Commits

Author SHA1 Message Date
Hector Sanjuan
58aea6c392
Merge pull request #1606 from ipfs/dependabot/go_modules/dependency-upgrades/contrib.go.opencensus.io/exporter/prometheus-0.4.1 
build(deps): bump contrib.go.opencensus.io/exporter/prometheus from 0.4.0 to 0.4.1
 2022-03-23 14:13:10 +01:00
Hector Sanjuan
10e93a7c46
Merge pull request #1608 from ipfs/dependabot/go_modules/dependency-upgrades/google.golang.org/protobuf-1.28.0 
build(deps): bump google.golang.org/protobuf from 1.27.1 to 1.28.0
 2022-03-23 14:13:02 +01:00
Hector Sanjuan
eee53bfa4f Streaming Peers(): make Peers() a streaming call 
This commit makes all the changes to make Peers() a streaming call.

While Peers is usually a non problematic call, for consistency, all calls
returning collections assembled through broadcast to cluster peers are now
streaming calls.
 2022-03-23 01:27:57 +01:00
dependabot[bot]
7aa1f3344d
build(deps): bump google.golang.org/protobuf from 1.27.1 to 1.28.0 
Bumps [google.golang.org/protobuf](https://github.com/protocolbuffers/protobuf-go) from 1.27.1 to 1.28.0.
- [Release notes](https://github.com/protocolbuffers/protobuf-go/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf-go/blob/master/release.bash)
- [Commits](https://github.com/protocolbuffers/protobuf-go/compare/v1.27.1...v1.28.0)

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-22 14:53:37 +00:00
Hector Sanjuan
0d73d33ef5 Pintracker: streaming methods 
This commit continues the work of taking advantage of the streaming
capabilities in go-libp2p-gorpc by improving the ipfsconnector and pintracker
components.

StatusAll and RecoverAll methods are now streaming methods, with the REST API
output changing accordingly to produce a stream of GlobalPinInfos rather than
a json array.

pin/ls request to the ipfs daemon now use ?stream=true and avoid having to
load the full pinset map on memory. StatusAllLocal and RecoverAllLocal
requests to the pin tracker stream all the way and no longer store the full
pinset, and the full PinInfo status slice before sending it out.

We have additionally switched to a pattern where streaming methods receive the
channel as an argument, allowing the caller to decide on whether to launch a
goroutine, do buffering etc.
 2022-03-22 15:38:01 +01:00
dependabot[bot]
d2fc658e59
build(deps): bump contrib.go.opencensus.io/exporter/prometheus 
Bumps [contrib.go.opencensus.io/exporter/prometheus](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus) from 0.4.0 to 0.4.1.
- [Release notes](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/releases)
- [Commits](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/compare/v0.4.0...v0.4.1)

---
updated-dependencies:
- dependency-name: contrib.go.opencensus.io/exporter/prometheus
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-19 11:40:12 +00:00
Hector Sanjuan
f5d78811c1 update go-libp2p-gorpc to 0.3.0 2022-03-19 12:09:23 +01:00
Hector Sanjuan
9b9d76f92d Pinset streaming and method type revamp 
This commit introduces the new go-libp2p-gorpc streaming capabilities for
Cluster. The main aim is to work towards heavily reducing memory usage when
working with very large pinsets.

As a side-effect, it takes the chance to revampt all types for all public
methods so that pointers to static what should be static objects are not used
anymore. This should heavily reduce heap allocations and GC activity.

The main change is that state.List now returns a channel from which to read
the pins, rather than pins being all loaded into a huge slice.

Things reading pins have been all updated to iterate on the channel rather
than on the slice. The full pinset is no longer fully loaded onto memory for
things that run regularly like StateSync().

Additionally, the /allocations endpoint of the rest API no longer returns an
array of pins, but rather streams json-encoded pin objects directly. This
change has extended to the restapi client (which puts pins into a channel as
they arrive) and to ipfs-cluster-ctl.

There are still pending improvements like StatusAll() calls which should also
stream responses, and specially BlockPut calls which should stream blocks
directly into IPFS on a single call.

These are coming up in future commits.
 2022-03-19 03:02:55 +01:00
Hector Sanjuan
6b0b0afdb0
Dependency upgrades (#1600) 
Dependency upgrades
 2022-03-15 11:50:44 +01:00
Hector Sanjuan
8583953357 Update go-ds-crdt with panic fix 2022-02-16 14:20:46 +01:00
Hector Sanjuan
3ffaf0fcc7
Merge pull request #1579 from ipfs/gorpc-upgrade 
go-libp2p-gorpc upgrade
 2022-02-15 19:33:43 +01:00
Hector Sanjuan
fa36958221 Merge branch 'master' into dependency-upgrades 2022-02-15 19:14:50 +01:00
Hector Sanjuan
9a0bca222b go-libp2p-gorpc upgrade 2022-02-15 19:12:12 +01:00
Hector Sanjuan
dc2b82e772
Merge pull request #1565 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-ipfs-files-0.1.1 
Bump github.com/ipfs/go-ipfs-files from 0.1.0 to 0.1.1
 2022-02-15 19:11:58 +01:00
Hector Sanjuan
5fb851babf
Merge pull request #1577 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-ds-crdt-0.3.1 
Bump github.com/ipfs/go-ds-crdt from 0.3.0 to 0.3.1
 2022-02-15 19:11:47 +01:00
dependabot[bot]
a1841a3162
Bump github.com/ipfs/go-ds-crdt from 0.3.0 to 0.3.1 
Bumps [github.com/ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/ipfs/go-ds-crdt/releases)
- [Commits](https://github.com/ipfs/go-ds-crdt/compare/v0.3.0...v0.3.1)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-crdt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-15 16:41:12 +00:00
Hector Sanjuan
975814229c crdt: increase number of workers from 5 to 50. Upgrade. 
The go-ds-crdt upgrade disables multi-head-processing by default again. We see
this causes a lot of branching.

We however increase the number of workers. With large deltas, it may be
possible that all the 5 workers are busy downloading a delta or processing
them, while we potentially have hundreds of children in the DAG. Thus it is
not bad to attempt to do more things in parallel.
 2022-02-15 16:54:42 +01:00
dependabot[bot]
a9250966a2
Bump github.com/ipfs/go-path from 0.2.1 to 0.2.2 
Bumps [github.com/ipfs/go-path](https://github.com/ipfs/go-path) from 0.2.1 to 0.2.2.
- [Release notes](https://github.com/ipfs/go-path/releases)
- [Commits](https://github.com/ipfs/go-path/compare/v0.2.1...v0.2.2)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-path
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-14 11:09:39 +00:00
dependabot[bot]
55e9ab622c
Bump github.com/ipfs/go-ipfs-files from 0.1.0 to 0.1.1 
Bumps [github.com/ipfs/go-ipfs-files](https://github.com/ipfs/go-ipfs-files) from 0.1.0 to 0.1.1.
- [Release notes](https://github.com/ipfs/go-ipfs-files/releases)
- [Commits](https://github.com/ipfs/go-ipfs-files/compare/v0.1.0...v0.1.1)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipfs-files
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-07 11:13:37 +00:00
Hector Sanjuan
670f9ac7d9
Merge pull request #1550 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-ipfs-files-0.1.0 
Bump github.com/ipfs/go-ipfs-files from 0.0.9 to 0.1.0
 2022-02-02 01:29:29 +01:00
Hector Sanjuan
92e5eaa58e
Merge pull request #1557 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/prometheus/client_golang-1.12.1 
Bump github.com/prometheus/client_golang from 1.11.0 to 1.12.1
 2022-02-02 01:07:43 +01:00
Hector Sanjuan
fbc45b17e4
Merge pull request #1558 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-ipfs-config-0.19.0 
Bump github.com/ipfs/go-ipfs-config from 0.18.0 to 0.19.0
 2022-02-02 01:07:25 +01:00
dependabot[bot]
8786588ac1
Bump github.com/ipfs/go-ds-crdt from 0.2.4 to 0.3.0 
Bumps [github.com/ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) from 0.2.4 to 0.3.0.
- [Release notes](https://github.com/ipfs/go-ds-crdt/releases)
- [Commits](https://github.com/ipfs/go-ds-crdt/compare/v0.2.4...v0.3.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-crdt
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-01 23:41:52 +00:00
Hector Sanjuan
881266a1a0 Update go-ds-crdt 
The new version supports auto-repair of the datastore.

Fixes #1007.
 2022-02-01 23:16:50 +01:00
dependabot[bot]
c331026ca5
Bump github.com/ipfs/go-ipfs-config from 0.18.0 to 0.19.0 
Bumps [github.com/ipfs/go-ipfs-config](https://github.com/ipfs/go-ipfs-config) from 0.18.0 to 0.19.0.
- [Release notes](https://github.com/ipfs/go-ipfs-config/releases)
- [Commits](https://github.com/ipfs/go-ipfs-config/compare/v0.18.0...v0.19.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipfs-config
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 11:07:15 +00:00
dependabot[bot]
36f85a96f8
Bump github.com/prometheus/client_golang from 1.11.0 to 1.12.1 
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.11.0 to 1.12.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.11.0...v1.12.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 11:06:44 +00:00
dependabot[bot]
fa7d7ded12
Bump github.com/ipfs/go-ipfs-files from 0.0.9 to 0.1.0 
Bumps [github.com/ipfs/go-ipfs-files](https://github.com/ipfs/go-ipfs-files) from 0.0.9 to 0.1.0.
- [Release notes](https://github.com/ipfs/go-ipfs-files/releases)
- [Commits](https://github.com/ipfs/go-ipfs-files/compare/v0.0.9...v0.1.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipfs-files
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-24 11:06:22 +00:00
Hector Sanjuan
f2ccf7a772 Update go-ds-crdt 2022-01-11 18:08:20 +01:00
Hector Sanjuan
a93886aeee
Merge pull request #1540 from ipfs/dependency-upgrades 
Dependency upgrades
 2022-01-11 17:11:13 +01:00
Hector Sanjuan
acbe12b034
Merge pull request #1539 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/hashicorp/go-hclog-1.1.0 
Bump github.com/hashicorp/go-hclog from 1.0.0 to 1.1.0
 2022-01-11 17:09:49 +01:00
Hector Sanjuan
862f4b515f Update protobuf compiler and generated files 2022-01-11 16:47:45 +01:00
dependabot[bot]
aa50177cc4
Bump github.com/hashicorp/go-hclog from 1.0.0 to 1.1.0 
Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 1.0.0 to 1.1.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v1.0...v1.1.0)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-11 15:33:10 +00:00
Hector Sanjuan
5abb804362 Update go-libp2p 2022-01-11 16:31:55 +01:00
Hector Sanjuan
dbb68c2a80
Merge pull request #1535 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/rs/cors-1.8.2 
Bump github.com/rs/cors from 1.8.0 to 1.8.2
 2022-01-10 23:04:10 +01:00
Hector Sanjuan
3537c8ebd5
Merge pull request #1527 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-path-0.2.1 
Bump github.com/ipfs/go-path from 0.1.2 to 0.2.1
 2022-01-10 23:03:52 +01:00
dependabot[bot]
261a87e205
Bump github.com/ipfs/go-path from 0.1.2 to 0.2.1 
Bumps [github.com/ipfs/go-path](https://github.com/ipfs/go-path) from 0.1.2 to 0.2.1.
- [Release notes](https://github.com/ipfs/go-path/releases)
- [Commits](https://github.com/ipfs/go-path/compare/v0.1.2...v0.2.1)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-path
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-10 14:52:07 +00:00
Hector Sanjuan
97a58adc2b
Merge pull request #1526 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-ipfs-api-0.3.0 
Bump github.com/ipfs/go-ipfs-api from 0.2.0 to 0.3.0
 2022-01-10 15:51:42 +01:00
dependabot[bot]
0b531d4f5e
Bump github.com/rs/cors from 1.8.0 to 1.8.2 
Bumps [github.com/rs/cors](https://github.com/rs/cors) from 1.8.0 to 1.8.2.
- [Release notes](https://github.com/rs/cors/releases)
- [Commits](https://github.com/rs/cors/compare/v1.8.0...v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/rs/cors
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-10 14:51:31 +00:00
Hector Sanjuan
8e3c6f5a75
Merge pull request #1530 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-log/v2-2.5.0 
Bump github.com/ipfs/go-log/v2 from 2.3.0 to 2.5.0
 2022-01-10 15:51:02 +01:00
Hector Sanjuan
31413c7e5d
Merge pull request #1531 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-ipld-cbor-0.0.6 
Bump github.com/ipfs/go-ipld-cbor from 0.0.5 to 0.0.6
 2022-01-10 15:50:45 +01:00
Hector Sanjuan
45553bcf87
Merge pull request #1533 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipld/go-car-0.3.3 
Bump github.com/ipld/go-car from 0.3.2 to 0.3.3
 2022-01-10 15:50:33 +01:00
Hector Sanjuan
7fa8d967e8
Merge pull request #1532 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-ipfs-pinner-0.2.1 
Bump github.com/ipfs/go-ipfs-pinner from 0.1.2 to 0.2.1
 2022-01-10 15:50:22 +01:00
Hector Sanjuan
615a6802c3
Merge pull request #1536 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/hashicorp/go-hclog-1.0.0 
Bump github.com/hashicorp/go-hclog from 0.16.2 to 1.0.0
 2022-01-10 15:49:21 +01:00
dependabot[bot]
6020095ba2
Bump github.com/ipld/go-car from 0.3.2 to 0.3.3 
Bumps [github.com/ipld/go-car](https://github.com/ipld/go-car) from 0.3.2 to 0.3.3.
- [Release notes](https://github.com/ipld/go-car/releases)
- [Commits](https://github.com/ipld/go-car/compare/v0.3.2...v0.3.3)

---
updated-dependencies:
- dependency-name: github.com/ipld/go-car
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-10 11:35:28 +00:00
dependabot[bot]
a135cdc7b0
Bump github.com/ipfs/go-ipfs-pinner from 0.1.2 to 0.2.1 
Bumps [github.com/ipfs/go-ipfs-pinner](https://github.com/ipfs/go-ipfs-pinner) from 0.1.2 to 0.2.1.
- [Release notes](https://github.com/ipfs/go-ipfs-pinner/releases)
- [Commits](https://github.com/ipfs/go-ipfs-pinner/compare/v0.1.2...v0.2.1)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipfs-pinner
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-10 11:35:22 +00:00
dependabot[bot]
22b84a75a4
Bump github.com/ipfs/go-ipfs-api from 0.2.0 to 0.3.0 
Bumps [github.com/ipfs/go-ipfs-api](https://github.com/ipfs/go-ipfs-api) from 0.2.0 to 0.3.0.
- [Release notes](https://github.com/ipfs/go-ipfs-api/releases)
- [Commits](https://github.com/ipfs/go-ipfs-api/compare/v0.2.0...v0.3.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipfs-api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-10 11:35:17 +00:00
Hector Sanjuan
fa16537768
Merge pull request #1524 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-ipfs-config-0.18.0 
Bump github.com/ipfs/go-ipfs-config from 0.16.0 to 0.18.0
 2022-01-10 12:34:14 +01:00
Hector Sanjuan
dd89fedd44
Merge pull request #1521 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/libp2p/go-libp2p-quic-transport-0.15.2 
Bump github.com/libp2p/go-libp2p-quic-transport from 0.15.0 to 0.15.2
 2022-01-10 12:33:33 +01:00
Hector Sanjuan
4bac2faa0a
Merge pull request #1522 from ipfs/dependabot/go_modules/dependency-upgrades/github.com/ipfs/go-unixfs-0.3.1 
Bump github.com/ipfs/go-unixfs from 0.2.6 to 0.3.1
 2022-01-10 12:32:53 +01:00
dependabot[bot]
5e88d489d3
Bump github.com/hashicorp/go-hclog from 0.16.2 to 1.0.0 
Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.16.2 to 1.0.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v0.16.2...v1.0)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-10 11:09:18 +00:00