This straigthens some mistakes with the outputs of the /add endpoints.
Currently, we had exactly the same output format which:
* was not exactly the ipfs API output format but was sort of similar
* made some weird concessions to be compatible (like having a string-type "size")
* was not aligned with Cluster API conventions (lowercase keys)
This corrects all this:
* The Cluster API /add output format now uses the right types and lowercase keys.
* `Hash` is now `Cid`, because the field carries a Cid.
* We copy error handling with request trailers from IPFS, and avoid carrying the
errors in the output objects.
* The proxy now returns exactly the types as ipfs would
* We add the X-Chunked-Output: 1 header, which is custom and redundant, but
otherwise breaks js-ipfs-api integrations with the /add endpoint.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This is a workaround to have clients behave properly with the /add
endpoint by asking them to close connections when done, effectively
disabling keep-alive for this.
This means we don't need to disable keep-alives fully on all servers,
since the rest of endpoints are not affected (they are not streaming
endpoints).
Reference https://github.com/ipfs/go-ipfs/issues/5168
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
See discussion in https://github.com/ipfs/go-ipfs/issues/5168
We cannot stream responses with keep-alives enabled.
I prefer this to not be a client feature, as otherwise users might end up
shooting themselves in the foot.
Note, the price is a corrupted request body which gets added
normally and gives wrong hashes!
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
No more hacks around /add. This uses the local adder when hijacking /add.
It supports the parameters and works pretty well with the ipfs CLI, showing
progress and everything.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
tests cover local and sharded adds of files
ipfs mock and ipfs block put/get calls cleaned up
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
RPC call to put a block in ipfs
IPFSConnector method to implement the RPC call
cluster restapi reads from channel and puts
blocks into IPFS via RPC in ctl add handler
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
to the ipfscluster.IPFSConnector interface and then
to the implementation of that interface in ipfsconn/ipfshttp.
This allows calls from MapPinTracker to cancel requests made
to the local IPFS node.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
ipfshttp: cancel POST request when timeout reached
ipfshttp/config: fix config test
ipfshttp: use struct styling for multi-line func calls
ipfshttp/config: add general ClientTimeout
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
query args correctly, requiring both a trailing slash and
non-trailing slash handle pattern to be defined for the
pin and unpin handlers.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
The extractCid function was added to enable the extraction of
a cid argument from either the url path or query string.
This puts the proxy behaviour on par with the current IPFS API.
The function does rely on the fact that ipfs-cluster doesn't
intercept any command that has more than one subcommand.
If that changes, this function will have to be updated.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
This fixes#326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.
It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.
Documentation for the configuration file has been updated.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
