Hector Sanjuan
d553227755
Dependency upgrades ( #1880 )
...
* build(deps): bump github.com/hsanjuan/ipfs-lite from 1.5.0 to 1.6.0
Bumps [github.com/hsanjuan/ipfs-lite](https://github.com/hsanjuan/ipfs-lite ) from 1.5.0 to 1.6.0.
- [Release notes](https://github.com/hsanjuan/ipfs-lite/releases )
- [Commits](https://github.com/hsanjuan/ipfs-lite/compare/v1.5.0...v1.6.0 )
---
updated-dependencies:
- dependency-name: github.com/hsanjuan/ipfs-lite
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/golang-jwt/jwt/v4 from 4.4.3 to 4.5.0
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt ) from 4.4.3 to 4.5.0.
- [Release notes](https://github.com/golang-jwt/jwt/releases )
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md )
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.4.3...v4.5.0 )
---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update pubsub too
* Fix compatiblity with latest libp2p, libipfs
* build(deps): bump github.com/libp2p/go-libp2p-raft from 0.3.0 to 0.4.0
Bumps [github.com/libp2p/go-libp2p-raft](https://github.com/libp2p/go-libp2p-raft ) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/libp2p/go-libp2p-raft/releases )
- [Commits](https://github.com/libp2p/go-libp2p-raft/compare/v0.3.0...v0.4.0 )
---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-raft
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/multiformats/go-multicodec
Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec ) from 0.7.0 to 0.8.1.
- [Release notes](https://github.com/multiformats/go-multicodec/releases )
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.7.0...v0.8.1 )
---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipld/go-car from 0.5.0 to 0.6.0
Bumps [github.com/ipld/go-car](https://github.com/ipld/go-car ) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/ipld/go-car/releases )
- [Changelog](https://github.com/ipld/go-car/blob/master/.goreleaser.yaml )
- [Commits](https://github.com/ipld/go-car/compare/v0.5.0...v0.6.0 )
---
updated-dependencies:
- dependency-name: github.com/ipld/go-car
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump golang.org/x/crypto from 0.5.0 to 0.6.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto ) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/golang/crypto/releases )
- [Commits](https://github.com/golang/crypto/compare/v0.5.0...v0.6.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/crypto
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipfs/go-ds-pebble from 0.1.0 to 0.2.2
Bumps [github.com/ipfs/go-ds-pebble](https://github.com/ipfs/go-ds-pebble ) from 0.1.0 to 0.2.2.
- [Release notes](https://github.com/ipfs/go-ds-pebble/releases )
- [Commits](https://github.com/ipfs/go-ds-pebble/compare/v0.1.0...v0.2.2 )
---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-pebble
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ugorji/go/codec from 1.2.8 to 1.2.10
Bumps [github.com/ugorji/go/codec](https://github.com/ugorji/go ) from 1.2.8 to 1.2.10.
- [Release notes](https://github.com/ugorji/go/releases )
- [Commits](https://github.com/ugorji/go/compare/v1.2.8...v1.2.10 )
---
updated-dependencies:
- dependency-name: github.com/ugorji/go/codec
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/libp2p/go-libp2p-http from 0.4.0 to 0.5.0
Bumps [github.com/libp2p/go-libp2p-http](https://github.com/libp2p/go-libp2p-http ) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/libp2p/go-libp2p-http/releases )
- [Commits](https://github.com/libp2p/go-libp2p-http/compare/v0.4.0...v0.5.0 )
---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-http
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipfs/go-ipfs-pinner from 0.2.1 to 0.3.0
Bumps [github.com/ipfs/go-ipfs-pinner](https://github.com/ipfs/go-ipfs-pinner ) from 0.2.1 to 0.3.0.
- [Release notes](https://github.com/ipfs/go-ipfs-pinner/releases )
- [Commits](https://github.com/ipfs/go-ipfs-pinner/compare/v0.2.1...v0.3.0 )
---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipfs-pinner
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipfs/go-unixfs from 0.4.3 to 0.4.4
Bumps [github.com/ipfs/go-unixfs](https://github.com/ipfs/go-unixfs ) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/ipfs/go-unixfs/releases )
- [Commits](https://github.com/ipfs/go-unixfs/compare/v0.4.3...v0.4.4 )
---
updated-dependencies:
- dependency-name: github.com/ipfs/go-unixfs
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipfs/go-path from 0.3.0 to 0.3.1
Bumps [github.com/ipfs/go-path](https://github.com/ipfs/go-path ) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/ipfs/go-path/releases )
- [Commits](https://github.com/ipfs/go-path/compare/v0.3.0...v0.3.1 )
---
updated-dependencies:
- dependency-name: github.com/ipfs/go-path
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipfs/go-ds-crdt from 0.3.9 to 0.3.10
Bumps [github.com/ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt ) from 0.3.9 to 0.3.10.
- [Release notes](https://github.com/ipfs/go-ds-crdt/releases )
- [Commits](https://github.com/ipfs/go-ds-crdt/compare/v0.3.9...v0.3.10 )
---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-crdt
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump golang.org/x/crypto from 0.6.0 to 0.7.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto ) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/golang/crypto/releases )
- [Commits](https://github.com/golang/crypto/compare/v0.6.0...v0.7.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/crypto
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipfs/go-ds-pebble from 0.2.2 to 0.2.3
Bumps [github.com/ipfs/go-ds-pebble](https://github.com/ipfs/go-ds-pebble ) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/ipfs/go-ds-pebble/releases )
- [Commits](https://github.com/ipfs/go-ds-pebble/compare/v0.2.2...v0.2.3 )
---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-pebble
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 14:41:29 +01:00
Hector Sanjuan
5452b59a2e
Dependency upgrades ( #1755 )
...
* Update go-libp2p to v0.22.0
* Testing with go1.19
* build(deps): bump github.com/multiformats/go-multicodec
Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec ) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/multiformats/go-multicodec/releases )
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.5.0...v0.6.0 )
---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipld/go-car from 0.4.0 to 0.5.0
Bumps [github.com/ipld/go-car](https://github.com/ipld/go-car ) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/ipld/go-car/releases )
- [Commits](https://github.com/ipld/go-car/compare/v0.4.0...v0.5.0 )
---
updated-dependencies:
- dependency-name: github.com/ipld/go-car
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/prometheus/client_golang
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang ) from 1.12.2 to 1.13.0.
- [Release notes](https://github.com/prometheus/client_golang/releases )
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md )
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.2...v1.13.0 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/hashicorp/go-hclog from 1.2.1 to 1.3.0
Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog ) from 1.2.1 to 1.3.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases )
- [Commits](https://github.com/hashicorp/go-hclog/compare/v1.2.1...v1.3.0 )
---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipfs/go-ds-crdt from 0.3.6 to 0.3.7
Bumps [github.com/ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt ) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/ipfs/go-ds-crdt/releases )
- [Commits](https://github.com/ipfs/go-ds-crdt/compare/v0.3.6...v0.3.7 )
---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-crdt
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/urfave/cli/v2 from 2.10.2 to 2.14.1
Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli ) from 2.10.2 to 2.14.1.
- [Release notes](https://github.com/urfave/cli/releases )
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md )
- [Commits](https://github.com/urfave/cli/compare/v2.10.2...v2.14.1 )
---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/libp2p/go-libp2p-http from 0.3.0 to 0.4.0
Bumps [github.com/libp2p/go-libp2p-http](https://github.com/libp2p/go-libp2p-http ) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/libp2p/go-libp2p-http/releases )
- [Commits](https://github.com/libp2p/go-libp2p-http/compare/v0.3.0...v0.4.0 )
---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-http
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/libp2p/go-libp2p-gorpc from 0.4.0 to 0.5.0
Bumps [github.com/libp2p/go-libp2p-gorpc](https://github.com/libp2p/go-libp2p-gorpc ) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/libp2p/go-libp2p-gorpc/releases )
- [Commits](https://github.com/libp2p/go-libp2p-gorpc/compare/v0.4.0...v0.5.0 )
---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-gorpc
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump contrib.go.opencensus.io/exporter/prometheus
Bumps [contrib.go.opencensus.io/exporter/prometheus](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus ) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/releases )
- [Commits](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/compare/v0.4.1...v0.4.2 )
---
updated-dependencies:
- dependency-name: contrib.go.opencensus.io/exporter/prometheus
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/libp2p/go-libp2p-raft from 0.1.8 to 0.2.0
Bumps [github.com/libp2p/go-libp2p-raft](https://github.com/libp2p/go-libp2p-raft ) from 0.1.8 to 0.2.0.
- [Release notes](https://github.com/libp2p/go-libp2p-raft/releases )
- [Commits](https://github.com/libp2p/go-libp2p-raft/compare/v0.1.8...v0.2.0 )
---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-raft
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/urfave/cli from 1.22.9 to 1.22.10
Bumps [github.com/urfave/cli](https://github.com/urfave/cli ) from 1.22.9 to 1.22.10.
- [Release notes](https://github.com/urfave/cli/releases )
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md )
- [Commits](https://github.com/urfave/cli/compare/v1.22.9...v1.22.10 )
---
updated-dependencies:
- dependency-name: github.com/urfave/cli
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* Fix checker/linter/staticcheck warnings
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-06 16:57:17 +02:00
Hector Sanjuan
508791b547
Migrate from ipfs/ipfs-cluster to ipfs-cluster/ipfs-cluster
...
This performs the necessary renamings.
2022-06-16 17:43:30 +02:00
Hector Sanjuan
1d98538411
Adders: stream blocks to destinations
...
This commit fixes #810 and adds block streaming to the final destinations when
adding. This should add major performance gains when adding data to clusters.
Before, everytime cluster issued a block, it was broadcasted individually to
all destinations (new libp2p stream), where it was block/put to IPFS (a single
block/put http roundtrip per block).
Now, blocks are streamed all the way from the adder module to the ipfs daemon,
by making every block as it arrives a single part in a multipart block/put
request.
Before, block-broadcast needed to wait for all destinations to finish in order
to process the next block. Now, buffers allow some destinations to be faster
than others while sending and receiving blocks.
Before, if a block put request failed to be broadcasted everywhere, an error
would happen at that moment.
Now, we keep streaming until the end and only then report any errors. The
operation succeeds as long as at least one stream finished successfully.
Errors block/putting to IPFS will not abort streams. Instead, subsequent
blocks are retried with a new request, although the method will return an
error when the stream finishes if there were errors at any point.
2022-03-24 17:24:58 +01:00
Hector Sanjuan
9b9d76f92d
Pinset streaming and method type revamp
...
This commit introduces the new go-libp2p-gorpc streaming capabilities for
Cluster. The main aim is to work towards heavily reducing memory usage when
working with very large pinsets.
As a side-effect, it takes the chance to revampt all types for all public
methods so that pointers to static what should be static objects are not used
anymore. This should heavily reduce heap allocations and GC activity.
The main change is that state.List now returns a channel from which to read
the pins, rather than pins being all loaded into a huge slice.
Things reading pins have been all updated to iterate on the channel rather
than on the slice. The full pinset is no longer fully loaded onto memory for
things that run regularly like StateSync().
Additionally, the /allocations endpoint of the rest API no longer returns an
array of pins, but rather streams json-encoded pin objects directly. This
change has extended to the restapi client (which puts pins into a channel as
they arrive) and to ipfs-cluster-ctl.
There are still pending improvements like StatusAll() calls which should also
stream responses, and specially BlockPut calls which should stream blocks
directly into IPFS on a single call.
These are coming up in future commits.
2022-03-19 03:02:55 +01:00
Hector Sanjuan
7e389d551c
Add test about local allocation
2022-02-01 22:29:54 +01:00
Hector Sanjuan
f83ff9b655
staticcheck: fix all staticcheck warnings in the project
2020-04-14 20:16:10 +02:00
Kishan Mohanbhai Sagathiya
6bd897ac47
Added UserAllocations support for add
2019-08-16 18:13:06 +05:30
Hector Sanjuan
b1769edf0c
Tests: add: do not delete test folders when shutting down peer
2019-08-13 18:19:10 +02:00
Hector Sanjuan
111cc29fc6
Add tests: fix tests for adder BlockPutHelper
2019-08-13 16:06:20 +02:00
Kishan Mohanbhai Sagathiya
70e429f925
Fix #852 : Improve error handling on add
...
Keep sending blocks while adding as long as one destination works.
2019-08-13 16:06:20 +02:00
Hector Sanjuan
c5864ada66
Tests: AddLocal, increase delay
...
The only way I can reproduce this failure is to reduce the delay.
2019-07-30 15:29:18 +02:00
Hector Sanjuan
8d3f99a5fc
Fix: fix flaky add test
...
This test has been failing a lot lately. Waiting for metrics might help.
2019-07-09 02:37:47 +02:00
Hector Sanjuan
8e6eefb714
Tests: multiple fixes
...
This fixes multiple issues in and around tests while
increasing ttls and delays in 100ms. Multiple issues, including
races, tests not running with consensus-crdt missing log messages
and better initialization have been fixed.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2019-05-20 23:45:04 +02:00
Hector Sanjuan
6447ea51d2
Remove *Serial types. Use pointers for all types.
...
This takes advantange of the latest features in go-cid, peer.ID and
go-multiaddr and makes the Go types serializable by default.
This means we no longer need to copy between Pin <-> PinSerial, or ID <->
IDSerial etc. We can now efficiently binary-encode these types using short
field keys and without parsing/stringifying (in many cases it just a cast).
We still get the same json output as before (with minor modifications for
Cids).
This should greatly improve Cluster performance and memory usage when dealing
with large collections of items.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-27 17:04:35 +00:00
Adrian Lanzafame
3b3f786d68
add opencensus tracing and metrics
...
This commit adds support for OpenCensus tracing
and metrics collection. This required support for
context.Context propogation throughout the cluster
codebase, and in particular, the ipfscluster component
interfaces.
The tracing propogates across RPC and HTTP boundaries.
The current default tracing backend is Jaeger.
The metrics currently exports the metrics exposed by
the opencensus http plugin as well as the pprof metrics
to a prometheus endpoint for scraping.
The current default metrics backend is Prometheus.
Metrics are currently exposed by default due to low
overhead, can be turned off if desired, whereas tracing
is off by default as it has a much higher performance
overhead, though the extent of the performance hit can be
adjusted with smaller sampling rates.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2019-02-04 18:53:21 +10:00
Hector Sanjuan
e7b1eacf83
Fix #543 : Use only healthy peers when adding everywhere (+tests)
...
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-09-27 08:12:29 +02:00