This adds a new allocations field to add response objects which
provides the cluster peers to which the content has been allocated.
In the case of sharded dags, it provides peers for the current shard.
This does 3 things:
- Add a NoPin option to the adder. When set to true, the adding process does not
send a pin in the end.
- When user-allocations are set and local=true happens, we do not overwrite
the allocations returned by the allocator to include the local peer
anymore, as this could alter user-allocations.
- Some code improvement (remove pointers).
This commit adds a new add option: "format".
This option specifies how IPFS Cluster is expected to build the DAG when
adding content. By default, it takes a "unixfs", which chunks and DAG-ifies as
it did before, resulting in a UnixFSv1 DAG.
Alternatively, it can be set to "car". In this case, Cluster will directly
read blocks from the CAR file and add them.
Adding CAR files or doing normal processing is independent from letting
cluster do sharding or not. If sharding is ever enabled, Cluster could
potentially shard a large CAR file among peers.
Currently, importing CAR files is limited to a single CAR file with a single
root (the one that is pinned). Future iterations may support multiple CARs
and/or multiple roots by transparently wrapping them.
This adds a new pin option: Mode that can be set to "direct" or "recursive".
The Mode is used to set the MaxDepth Pin field accordingly. When set to 0, we
will call pin/add using the type=direct.
This adds a new PinOption: ExpireAt.
The StateSync ticker will check and unpin expired pins from the Cluster.
ipfs-cluster-ctl supports an "expire-in" which gives a duration.
This commit introduces `--local` option for `ctl add` which would add
content only the local ipfs peer and then pin it according to pin
options (fetching from the local peer)
For achieving this, a new local dag service is introduced
This was a leftover. For consisency, all CIDs coming out of the API
should have the canonical cid form ( { "/": "cid" } ), otherwise
we are inconsistent.
This straigthens some mistakes with the outputs of the /add endpoints.
Currently, we had exactly the same output format which:
* was not exactly the ipfs API output format but was sort of similar
* made some weird concessions to be compatible (like having a string-type "size")
* was not aligned with Cluster API conventions (lowercase keys)
This corrects all this:
* The Cluster API /add output format now uses the right types and lowercase keys.
* `Hash` is now `Cid`, because the field carries a Cid.
* We copy error handling with request trailers from IPFS, and avoid carrying the
errors in the output objects.
* The proxy now returns exactly the types as ipfs would
* We add the X-Chunked-Output: 1 header, which is custom and redundant, but
otherwise breaks js-ipfs-api integrations with the /add endpoint.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
.Add(paths) will interpret http* paths as WebFiles. These are read performing
a GET request to the location. Otherwise, the path is interpreted as a local
disk file/folder, and read from disk. ipfs-cluster-ctl has been updated
accordingly.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
No more hacks around /add. This uses the local adder when hijacking /add.
It supports the parameters and works pretty well with the ipfs CLI, showing
progress and everything.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.
This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
