cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,866 Commits 3 Branches 70 Tags 15 MiB
b6e8a6efd2
Commit Graph

28 Commits

Author SHA1 Message Date
Hector Sanjuan
d553227755
Dependency upgrades (#1880) 
* build(deps): bump github.com/hsanjuan/ipfs-lite from 1.5.0 to 1.6.0

Bumps [github.com/hsanjuan/ipfs-lite](https://github.com/hsanjuan/ipfs-lite) from 1.5.0 to 1.6.0.
- [Release notes](https://github.com/hsanjuan/ipfs-lite/releases)
- [Commits](https://github.com/hsanjuan/ipfs-lite/compare/v1.5.0...v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/hsanjuan/ipfs-lite
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/golang-jwt/jwt/v4 from 4.4.3 to 4.5.0

Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.4.3 to 4.5.0.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.4.3...v4.5.0)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update pubsub too

* Fix compatiblity with latest libp2p, libipfs

* build(deps): bump github.com/libp2p/go-libp2p-raft from 0.3.0 to 0.4.0

Bumps [github.com/libp2p/go-libp2p-raft](https://github.com/libp2p/go-libp2p-raft) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/libp2p/go-libp2p-raft/releases)
- [Commits](https://github.com/libp2p/go-libp2p-raft/compare/v0.3.0...v0.4.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-raft
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/multiformats/go-multicodec

Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec) from 0.7.0 to 0.8.1.
- [Release notes](https://github.com/multiformats/go-multicodec/releases)
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.7.0...v0.8.1)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipld/go-car from 0.5.0 to 0.6.0

Bumps [github.com/ipld/go-car](https://github.com/ipld/go-car) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/ipld/go-car/releases)
- [Changelog](https://github.com/ipld/go-car/blob/master/.goreleaser.yaml)
- [Commits](https://github.com/ipld/go-car/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: github.com/ipld/go-car
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump golang.org/x/crypto from 0.5.0 to 0.6.0

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-ds-pebble from 0.1.0 to 0.2.2

Bumps [github.com/ipfs/go-ds-pebble](https://github.com/ipfs/go-ds-pebble) from 0.1.0 to 0.2.2.
- [Release notes](https://github.com/ipfs/go-ds-pebble/releases)
- [Commits](https://github.com/ipfs/go-ds-pebble/compare/v0.1.0...v0.2.2)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-pebble
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ugorji/go/codec from 1.2.8 to 1.2.10

Bumps [github.com/ugorji/go/codec](https://github.com/ugorji/go) from 1.2.8 to 1.2.10.
- [Release notes](https://github.com/ugorji/go/releases)
- [Commits](https://github.com/ugorji/go/compare/v1.2.8...v1.2.10)

---
updated-dependencies:
- dependency-name: github.com/ugorji/go/codec
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-http from 0.4.0 to 0.5.0

Bumps [github.com/libp2p/go-libp2p-http](https://github.com/libp2p/go-libp2p-http) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/libp2p/go-libp2p-http/releases)
- [Commits](https://github.com/libp2p/go-libp2p-http/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-ipfs-pinner from 0.2.1 to 0.3.0

Bumps [github.com/ipfs/go-ipfs-pinner](https://github.com/ipfs/go-ipfs-pinner) from 0.2.1 to 0.3.0.
- [Release notes](https://github.com/ipfs/go-ipfs-pinner/releases)
- [Commits](https://github.com/ipfs/go-ipfs-pinner/compare/v0.2.1...v0.3.0)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ipfs-pinner
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-unixfs from 0.4.3 to 0.4.4

Bumps [github.com/ipfs/go-unixfs](https://github.com/ipfs/go-unixfs) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/ipfs/go-unixfs/releases)
- [Commits](https://github.com/ipfs/go-unixfs/compare/v0.4.3...v0.4.4)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-unixfs
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-path from 0.3.0 to 0.3.1

Bumps [github.com/ipfs/go-path](https://github.com/ipfs/go-path) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/ipfs/go-path/releases)
- [Commits](https://github.com/ipfs/go-path/compare/v0.3.0...v0.3.1)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-path
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-ds-crdt from 0.3.9 to 0.3.10

Bumps [github.com/ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) from 0.3.9 to 0.3.10.
- [Release notes](https://github.com/ipfs/go-ds-crdt/releases)
- [Commits](https://github.com/ipfs/go-ds-crdt/compare/v0.3.9...v0.3.10)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-crdt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump golang.org/x/crypto from 0.6.0 to 0.7.0

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/compare/v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-ds-pebble from 0.2.2 to 0.2.3

Bumps [github.com/ipfs/go-ds-pebble](https://github.com/ipfs/go-ds-pebble) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/ipfs/go-ds-pebble/releases)
- [Commits](https://github.com/ipfs/go-ds-pebble/compare/v0.2.2...v0.2.3)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-pebble
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-03-06 14:41:29 +01:00
Hector Sanjuan
d19c7facff Fix: leaking goroutines on aborted /add requests 
It has been observed that some peers have a growing number of goroutines,
usually stuck in go-libp2p-gorpc.MultiStream() function, which is waiting to
read items from the arguments channel.

We suspect this is due to aborted /add requests. In situations when the add
request is aborted or fails, Finalize() is never called and the blocks channel
stays open, so MultiStream() can never exit, and the BlockStreamer can never
stop streaming etc.

As a fix, we added the requirement to call Close() when we stop using a
ClusterDAGService (error or not). This should ensure that the blocks channel
is always closed and not just on Finalize().
 2022-07-08 17:39:59 +02:00
Hector Sanjuan
144c66359e Test module: allow the MockDAGService to be write-only. 
The adder needs a write-only DAGService.
 2022-06-16 17:43:30 +02:00
Hector Sanjuan
508791b547 Migrate from ipfs/ipfs-cluster to ipfs-cluster/ipfs-cluster 
This performs the necessary renamings.
 2022-06-16 17:43:30 +02:00
Hector Sanjuan
a97ed10d0b Adopt api.Cid type - replaces cid.Cid everwhere. 
This commit introduces an api.Cid type and replaces the usage of cid.Cid
everywhere.

The main motivation here is to override MarshalJSON so that Cids are
JSON-ified as '"Qm...."' instead of '{ "/": "Qm....." }', as this "ipld"
representation of IDs is horrible to work with, and our APIs are not issuing
IPLD objects to start with.

Unfortunately, there is no way to do this cleanly, and the best way is to just
switch everything to our own type.
 2022-04-07 14:27:39 +02:00
Hector Sanjuan
0d73d33ef5 Pintracker: streaming methods 
This commit continues the work of taking advantage of the streaming
capabilities in go-libp2p-gorpc by improving the ipfsconnector and pintracker
components.

StatusAll and RecoverAll methods are now streaming methods, with the REST API
output changing accordingly to produce a stream of GlobalPinInfos rather than
a json array.

pin/ls request to the ipfs daemon now use ?stream=true and avoid having to
load the full pinset map on memory. StatusAllLocal and RecoverAllLocal
requests to the pin tracker stream all the way and no longer store the full
pinset, and the full PinInfo status slice before sending it out.

We have additionally switched to a pattern where streaming methods receive the
channel as an argument, allowing the caller to decide on whether to launch a
goroutine, do buffering etc.
 2022-03-22 15:38:01 +01:00
Hector Sanjuan
072f0bc722 Feat: support adding CAR files 
This commit adds a new add option: "format".

This option specifies how IPFS Cluster is expected to build the DAG when
adding content. By default, it takes a "unixfs", which chunks and DAG-ifies as
it did before, resulting in a UnixFSv1 DAG.

Alternatively, it can be set to "car". In this case, Cluster will directly
read blocks from the CAR file and add them.

Adding CAR files or doing normal processing is independent from letting
cluster do sharding or not. If sharding is ever enabled, Cluster could
potentially shard a large CAR file among peers.

Currently, importing CAR files is limited to a single CAR file with a single
root (the one that is pinned). Future iterations may support multiple CARs
and/or multiple roots by transparently wrapping them.
 2021-04-21 13:55:06 +02:00
Hector Sanjuan
717ed85823 gofmt -s fixes 2020-04-14 23:44:18 +02:00
Hector Sanjuan
2c47585870 Handle wrapping outside the ipfs adder 
The ipfs adder does not wrap anymore.
 2019-10-01 12:19:22 +02:00
Kishan Mohanbhai Sagathiya
70e429f925 Fix #852: Improve error handling on add 
Keep sending blocks while adding as long as one destination works.
 2019-08-13 16:06:20 +02:00
Hector Sanjuan
ea85cf7805 Rename "test.Test*" to "test.*" (test.TestCid1 -> test.Cid1) 
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
 2019-02-27 20:19:10 +00:00
Hector Sanjuan
9df6344a07 Avoid using string testing CIDs and use cid.Cids directly 
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
 2019-02-27 20:09:31 +00:00
Hector Sanjuan
ced9f2f7f1 Update to go-ipfs-files 2.0.3 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-12-17 14:29:44 +01:00
Hector Sanjuan
15413edbc6 Unrewrite deps 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-12-17 14:16:38 +01:00
Łukasz Magiera
406d2b53f8 files2.0: Close subfiles in adder 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
 2018-12-17 14:16:38 +01:00
Łukasz Magiera
9b0ff603b8 files2.0: use type helpers 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
 2018-12-17 14:16:38 +01:00
Łukasz Magiera
c6944e4376 files2.0: update for no errors from NewMultiFileReader 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
 2018-12-17 14:16:38 +01:00
Łukasz Magiera
d0678aad6e files2.0: fix ContextCancelled test 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
 2018-12-17 14:16:38 +01:00
Łukasz Magiera
00df384e1c go-ipfs-files 2.0 updates 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
 2018-12-17 14:16:38 +01:00
Hector Sanjuan
b86f3cedb4 Files package has been extracted from cmdkit 
Related: https://github.com/ipfs/go-ipfs-cmdkit/pull/31

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-10-24 01:28:05 +02:00
Hector Sanjuan
b2914c2516 Fix #495: Hijack proxy /add correctly. 
No more hacks around /add. This uses the local adder when hijacking /add.
It supports the parameters and works pretty well with the ipfs CLI, showing
progress and everything.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-18 02:48:35 +02:00
Hector Sanjuan
c2312cbb05 Fix: close files. Close test files. Clean properly. Fix windows tests. 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-09 12:05:27 +02:00
Hector Sanjuan
26cbecca4c Fix tests 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-09 03:42:39 +02:00
Hector Sanjuan
7460be32c4 Try more 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:47:47 +02:00
Hector Sanjuan
fc2d7beacc Try syncing test folders 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:31:40 +02:00
Hector Sanjuan
7ef7a8d158 sync test files after creation 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
327a81b85a golint govets 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
623120fd50 Start cluster tests 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00