Commit Graph

38 Commits

Author SHA1 Message Date
Hector Sanjuan
1d98538411 Adders: stream blocks to destinations
This commit fixes #810 and adds block streaming to the final destinations when
adding. This should add major performance gains when adding data to clusters.

Before, everytime cluster issued a block, it was broadcasted individually to
all destinations (new libp2p stream), where it was block/put to IPFS (a single
block/put http roundtrip per block).

Now, blocks are streamed all the way from the adder module to the ipfs daemon,
by making every block as it arrives a single part in a multipart block/put
request.

Before, block-broadcast needed to wait for all destinations to finish in order
to process the next block. Now, buffers allow some destinations to be faster
than others while sending and receiving blocks.

Before, if a block put request failed to be broadcasted everywhere, an error
would happen at that moment.

Now, we keep streaming until the end and only then report any errors. The
operation succeeds as long as at least one stream finished successfully.

Errors block/putting to IPFS will not abort streams. Instead, subsequent
blocks are retried with a new request, although the method will return an
error when the stream finishes if there were errors at any point.
2022-03-24 17:24:58 +01:00
Hector Sanjuan
759dd4802d Fix 1598: Include allocations in responses from /add
This adds a new allocations field to add response objects which
provides the cluster peers to which the content has been allocated.

In the case of sharded dags, it provides peers for the current shard.
2022-03-15 11:03:11 +01:00
Hector Sanjuan
00dffe23b8 Adder: Add "no-pin" option.
This does 3 things:

- Add a NoPin option to the adder. When set to true, the adding process does not
send a pin in the end.

- When user-allocations are set and local=true happens, we do not overwrite
  the allocations returned by the allocator to include the local peer
  anymore, as this could alter user-allocations.

- Some code improvement (remove pointers).
2022-02-28 20:10:12 +01:00
Hector Sanjuan
b6238df481 Fix #1362: Add "size" to response when adding CARs with a unixfs file root.
This sets the "size" value to the FileSize() value when the CAR file's root
is a unixfs file. (Folders are not files, size will still be 0).
2021-06-28 22:37:32 +02:00
Hector Sanjuan
072f0bc722 Feat: support adding CAR files
This commit adds a new add option: "format".

This option specifies how IPFS Cluster is expected to build the DAG when
adding content. By default, it takes a "unixfs", which chunks and DAG-ifies as
it did before, resulting in a UnixFSv1 DAG.

Alternatively, it can be set to "car". In this case, Cluster will directly
read blocks from the CAR file and add them.

Adding CAR files or doing normal processing is independent from letting
cluster do sharding or not. If sharding is ever enabled, Cluster could
potentially shard a large CAR file among peers.

Currently, importing CAR files is limited to a single CAR file with a single
root (the one that is pinned). Future iterations may support multiple CARs
and/or multiple roots by transparently wrapping them.
2021-04-21 13:55:06 +02:00
Hector Sanjuan
b3853caf36 Dependency ugprade: changes needed
* Libp2p protectors no longer needed, use PSK directly
* Generate cluster 32-byte secret here (helper gone from pnet)
* Switch to go-log/v2 in all places
* DHT bootstrapping not needed. Adjust DHT options for tests.
* Do not rely on dissappeared CidToDsKey and DsKeyToCid functions fro dshelp.
* Disable QUIC (does not support private networks)
* Fix tests: autodiscovery started working properly
2020-03-22 14:50:25 +01:00
Yang Hau
7986d94242
fix: Fix typos (#1001)
Fix typos in files
2020-02-03 10:30:04 +01:00
Hector Sanjuan
6f3a551b33 Adder: improve comment 2019-12-18 11:32:28 +01:00
Hector Sanjuan
442b441985 Fix #983: Make sure single-file labels correspond to their name 2019-12-18 00:24:16 +01:00
Hector Sanjuan
9c61bf6f26 Fix #983: Correctly print paths when adding 2019-12-17 23:01:25 +01:00
Hector Sanjuan
2c47585870 Handle wrapping outside the ipfs adder
The ipfs adder does not wrap anymore.
2019-10-01 12:19:22 +02:00
Hector Sanjuan
91f1d160a7 Attempt to fixes to adder and alignment with newer versions 2019-10-01 12:19:22 +02:00
jmank88
76c798e313 support --nocopy when adding URLs 2019-03-26 10:22:37 -05:00
Hector Sanjuan
c4b18cd5f6 Address issues from self-review
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-27 18:43:29 +00:00
Hector Sanjuan
6447ea51d2 Remove *Serial types. Use pointers for all types.
This takes advantange of the latest features in go-cid, peer.ID and
go-multiaddr and makes the Go types serializable by default.

This means we no longer need to copy between Pin <-> PinSerial, or ID <->
IDSerial etc. We can now efficiently binary-encode these types using short
field keys and without parsing/stringifying (in many cases it just a cast).

We still get the same json output as before (with minor modifications for
Cids).

This should greatly improve Cluster performance and memory usage when dealing
with large collections of items.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-27 17:04:35 +00:00
Hector Sanjuan
15413edbc6 Unrewrite deps
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-12-17 14:16:38 +01:00
Łukasz Magiera
406d2b53f8 files2.0: Close subfiles in adder
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
2018-12-17 14:16:38 +01:00
Łukasz Magiera
00df384e1c go-ipfs-files 2.0 updates
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
2018-12-17 14:16:38 +01:00
Hector Sanjuan
b86f3cedb4 Files package has been extracted from cmdkit
Related: https://github.com/ipfs/go-ipfs-cmdkit/pull/31

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-24 01:28:05 +02:00
Adrian Lanzafame
31474f6490
update go-cid and go-libp2p
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-09-24 11:35:38 +10:00
Hector Sanjuan
0c5de3a849 Add/improve godoc descriptions for some modules.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-23 14:14:06 +02:00
Hector Sanjuan
05102c13a0 AddParams: support "cid-version" and "hash" function.
Additionally, remove dependency to go-ipfs after extraction from
"go-mfs".

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-21 19:38:41 +02:00
Hector Sanjuan
b2914c2516 Fix #495: Hijack proxy /add correctly.
No more hacks around /add. This uses the local adder when hijacking /add.
It supports the parameters and works pretty well with the ipfs CLI, showing
progress and everything.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-18 02:48:35 +02:00
Hector Sanjuan
7b826a5335 Dependency upgrades
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 14:39:34 +02:00
Hector Sanjuan
f707621a39 Allow to use adder only once
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 11:51:27 +02:00
Hector Sanjuan
3c80169174 dag -> dgs
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 13:24:10 +02:00
Hector Sanjuan
c2312cbb05 Fix: close files. Close test files. Clean properly. Fix windows tests.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 12:05:27 +02:00
Hector Sanjuan
78b8f47c14 Fix: Wrap-in-directory
Usually we had wrap-in-directory enabled by default because otherwise
we had an error when adding single, non-directory files.

wrap-in-directory happens automatically when adding more than one file
so that was no problem. Thigns also worked when adding a folder and Wrap was
disabled. The only case was adding a single with wrap disabled (a default option).

This patches the ipfsadd/add.go file to remember the last added file so that
we can use it's Cid as the resulting root of the adding process without
having to fetch it from our dummy dagservice.

We have to pass this CID to our Finalize() functions, because it turns out that
in this case (single file without wrap-in-directory), the last block added to the DAG
is not the IPFS root (the ipfsadd/Adder adds the mfs root folder last always).

This was the case when wrap-in-directory was enabled by default.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 01:16:30 +02:00
Hector Sanjuan
e3834cf8ab Force-enable wrap-in-directory
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-08 21:43:20 +02:00
Hector Sanjuan
1228d1cfc6 Golint and improved error messages
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-08 21:29:21 +02:00
Hector Sanjuan
0ed14a4b39 Adder: add context to add functions. Fix allocation everywhere. Helpers.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-08 21:10:42 +02:00
Hector Sanjuan
5bb4d1a4bc misc
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:24:33 +02:00
Hector Sanjuan
87f4fcf958 Make the adder modules ipld.DAGService modules.
This removes a bunch of the channel dance and block forwarding
by having the adder submodules be DAGServices themselves and take
Add() directly from the ipfsAdder.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
3575f05753 Support adding Output (api.AddedOutput)
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.

This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
8f1a15b279 Move adder.Params to api.AddParams. Re-use in other modules
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
327a81b85a golint govets
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
f8ff9883b0 wip
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
a96241941e WIP
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00