Support the new endpoint for later metrics in `rest/api/client`
Support the new method created in `rest/api/client` in
ipfs-cluster-ctl. i.e. `ipfs-cluster-ctl health metrics <name>` would
show the peers and the last list of metrics logged for each as returned
by the Peer Monitor, in a friendly way.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Opened new endpoint `GET /health/metrics/<name>` which would respond
with metrics of type <name>
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
By default, CORS headers allowing GET requests from everywhere are
set. This should facilitate the IPFS Web UI integration with the
Cluster API.
This commit refactors the sendResponse methods in the API, merging
them into one as it was difficult to follow the flows that actually
send something to the client. All tests now check the presence of
the configured headers too, to make sure no route was missed.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Adding more missing pieces in config
Use the right package(not the inbuilt one)
Setup rpc client for proxy in the cluster
Add back SetClient and Shutdown into Connector as they are required to
implement Component interface
Add `ipfsproxy` as into list of logging identifier and add its default
log level
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Go through `ipfsproxy` repo and change things that are inappropriate
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Extract the IPFS Proxy from ipfshttp and make it an api module
The `ipfshttp` IPFSConnector implementation includes the so called IPFS
Proxy. An endpoint which offers an IPFS API, hijacking some interesting
requests and forwarding the rest to the ipfs daemon.
`ipfshttp` should contain an implementation of IPFSConnector whose only
task should be to talk to IPFS
A new module should be created, `api/ipfsproxy`, an API Component
implementation for Cluster. The whole proxy code should be moved here.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
This straigthens some mistakes with the outputs of the /add endpoints.
Currently, we had exactly the same output format which:
* was not exactly the ipfs API output format but was sort of similar
* made some weird concessions to be compatible (like having a string-type "size")
* was not aligned with Cluster API conventions (lowercase keys)
This corrects all this:
* The Cluster API /add output format now uses the right types and lowercase keys.
* `Hash` is now `Cid`, because the field carries a Cid.
* We copy error handling with request trailers from IPFS, and avoid carrying the
errors in the output objects.
* The proxy now returns exactly the types as ipfs would
* We add the X-Chunked-Output: 1 header, which is custom and redundant, but
otherwise breaks js-ipfs-api integrations with the /add endpoint.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This is a workaround to have clients behave properly with the /add
endpoint by asking them to close connections when done, effectively
disabling keep-alive for this.
This means we don't need to disable keep-alives fully on all servers,
since the rest of endpoints are not affected (they are not streaming
endpoints).
Reference https://github.com/ipfs/go-ipfs/issues/5168
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This commit adds peername to PinInfo and GlobalPinInfo so that we have
a nicer and more meaningfull output for `ipfs-cluster-ctl` queries like
`status`, `sync` and `recover`
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
This commit adds peername to PinInfo and GlobalPinInfo so that we have
a nicer and more meaningfull output for queries like
`ipfs-cluster-ctl status`
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
.Add(paths) will interpret http* paths as WebFiles. These are read performing
a GET request to the location. Otherwise, the path is interpreted as a local
disk file/folder, and read from disk. ipfs-cluster-ctl has been updated
accordingly.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
See discussion in https://github.com/ipfs/go-ipfs/issues/5168
We cannot stream responses with keep-alives enabled.
I prefer this to not be a client feature, as otherwise users might end up
shooting themselves in the foot.
Note, the price is a corrupted request body which gets added
normally and gives wrong hashes!
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
No more hacks around /add. This uses the local adder when hijacking /add.
It supports the parameters and works pretty well with the ipfs CLI, showing
progress and everything.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This removes PeerAddr and uses APIAddr directly, figuring out if it is
a Peer multiaddress or not.
PeerAddr is actually kept for compatiblity.
It also fixes a bad panic when resolving returned 0 results
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This removes a bunch of the channel dance and block forwarding
by having the adder submodules be DAGServices themselves and take
Add() directly from the ipfsAdder.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.
This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
validation of url params in restapi fileadd handler
use --only-hashes in sharness to simplify cid parse
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
tests cover local and sharded adds of files
ipfs mock and ipfs block put/get calls cleaned up
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
deterministically sample bytes into files of different sizes
reorganized hash storage for easy access from all subpacks
import tests pass with new directory structure
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
api client test
api test
one sharness test
refactoring of testingData for access from other packages
always wrap files added by cluster
remove unused flag 'progress'
changes to support hidden flag
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
addFile function is now a Cluster method accessed by RPC
residue from attempting to stream responses removed
ipfs-cluster-ctl ls bug fixed
problem with importer/add not printing resolved
new test now checks for this
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
non-recursive cdag pins
priority pins for shards to allocated peer
cdag, meta and shard pin types used
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
1. Refactored importer endpoint, including writing cluster-specific
file adder, to get print info from importer
2. Refactored importer consumption to select equally from
channels of different output signals and manage context
timeouts correctly (only in local add here, sharding to follow)
3. Added output streaming and an error/termination handling protocol
4. Discovered that naive eager response streaming cuts off
reads from request data stream and breaks behavior, for
now all responses come after file ingestion.
5. Added ipfs add style flags (trickle, rawleaves etc.) and
refactored importer endpoint to take in these parameters
to provide identicle behavior to ipfs
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
sharding passes manual tests on single node cluster,
adding the shards of a directory and pinning the
clusterDAG to cluster/ipfs state
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
A sharder config provides a default shard size
getAllocations calls newly exposed rpcs
ipld cbor cluster dag node construction
logic to handle Flushing final shard
logic to initialize and finalize shard sessions
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>