This adds support for handling preflight requests in the REST API
and fixes currently mostly broken CORS.
Before we just let the user add custom response headers to the
configuration "headers" key but this is not the best way because
CORs headers and requests need special handling and doing it wrong
has security implications.
Therefore, I have added specific CORS-related configuration options
which control CORS behavour. We are forced to change the "headers"
defaults and will notify the users about this in the changelog.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This changes the current strategy to extract headers from the IPFS daemon to
use them for hijacked endpoints in the proxy. The ipfs daemon is a bit of a
mess and what we were doing is not really reliable, specially when it comes to
setting CORS headers right (which we were not doing).
The new approach is:
* For every hijacked request, make an OPTIONS request to the same path, with
the given Origin, to the IPFS daemon and extract some CORS headers from
that. Use those in the hijacked response
* Avoid hijacking OPTIONS request, they should always go through so the IPFS
daemon controls all the CORS-preflight things as it wants.
* Similar to before, have a only-once-triggered request to extract other
interesting or custom headers from a fixed IPFS endpoint. This allows us to
have the proxy forward other custom headers and to catch
`Access-Control-Expose-Methods`. The difference is that the endpoint use for
this and the additional headers are configurable by the user (but with hidden
configuration options because this is quite exotic from regular usage).
Now the implementation:
* Replaced the standard Muxer with gorilla/mux (I have also taken the change
to update the gxed version to the latest tag). This gives us much better
matching control over routes and allows us to not handle OPTIONS requests.
* This allows also to remove the extractArgument code and have proper handlers
for the endpoints passing command arguments as the last segment of the URL. A
very simple handler that wraps the default ones can be used to extract the
argument from the url and put it in the query. Overall much cleaner this way.
* No longer capture interesting headers from any random proxied request. This
made things complicated with a wrapping handler. We will just trigger the one
request to do it when we need it.
* When preparing the headers for the hijacked responses:
* Trigger the OPTIONS request and figure out which CORS things we should set
* Set the additional headers (perhaps triggering a POST request to fetch them)
* Set our own headers.
* Moved all the headers stuff to a new headers.go file.
* Added configuration options (hidden by default) to:
* Customize the extract headers endpoint
* Customize what additional headers are extracted
* Use HTTPs when talking to the IPFS API
* I haven't tested this, but I did not want to have hardcoded 'http://' urls
around, as before.
* Added extra testing for this, and tested manually a lot comparing the
daemon original output with our hijacked endpoint outputs while looking
at the API traffic with ngrep and making sure the requets happen as expected.
Also tested with IPFS companion in FF and Chrome.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Optimized filter to tracker status matching by using bitwise
comparisions
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Added a fail case where an invalid filter is passed in.
Update `api.rpcClient.Call` to `api.rpcClient.CallContext` and pass
in the Request context r.Context() so that context can be cancelled
when the request is cancelled by caller
Fixes#445
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Added filter option to `ipfs-cluster-ctl status`
When the --filter is passed, it will only fetch the peer information
where status of the pin matches with the filter value.
Valid filter values are tracker status types(i.e., "pinned",
"pin_error", "unpinning" etc), an alias of tracker status type (i.e.,
"queued" or "error"), comma separated list of tracker status type
and/or it aliases(i.e., "error,pinning")
On passing invalid filter value no status information will be shown
In particular, the filter would remove elements from []GlobalPinInfo
when none of the peers in GlobalPinInfo match the filter. If one peer
in the GlobalPinInfo matches the filter, the whole object is returned,
including the information for the other peers which may or not match it.
filter option works on statusAll("GET /pins"). For fetching pin status
for a CID("GET /pins/<cid>"), filter option would have no effect
Fixes#445
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
-Fixed logic issue in match condition of 'filterStatus' function
-Added and verified success of test provided by @lanzafame
-Attempted to condense code and apply other cleanup provided by @lanzafame
License: MIT
Signed-off-by: Paul Jewell <sona1111@zoho.com>
Move ctl-health sharness tests to apprpriate file
Since the API is using the RPC mock to request metrics and it always
returns a mocked test metric we might just do c.Metrics("somemetricstype")
and check that there is no error. Here we just want to check that the
client is hitting an API endpoint (and understands the response).
Fixes#587
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Added API and client tests for GET /monitor/metrics/{metrics_type}
Fixes#587
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
We should deprecate passing in Host/Port in the config,
but in the meantime, it hardcoded /dns4/, meaning that if
someone placed an ipv6 address in there things would break badly
and weirdly.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Before we resolved all /dns*/ multiaddresses before we used them.
When using HTTPs, the Go HTTP Client only sees the resolved IP address
and it is unable to negotiate TLS with a cerficate because the request
is not going to the hostname the certificate is signed for, but to
the IP. This leverages a recent feature in go-multiaddr-net
and uses directly the user-provided hostname.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This commit makes the proxy extract useful fixed headers (like CORS) from
the IPFS daemon API responses and then apply them to the responses
from hijacked endpoints like /add or /repo/stat.
It does this by caching a list of headers from the first IPFS API
response which has them. If we have not performed any proxied request or
managed to obtain the headers we're interested in, this will try triggering a
request to "/api/v0/version" to obtain them first.
This should fix the issues with using Cluster proxy with IPFS Companion and
Chrome.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Put a note for deprecating json fields that they are only here to
maintain compatibility and they will be removed in future
Start using env vars starting with `CLUSTER_IPFSPROXY`
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Remove proxy_ and Proxy from proxy config objects without breaking
compatibility with previous revisions
Fixes#616
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Warning about old config style(proxy inside ipfs connector) getting
removed in future
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
The FSM tries to decode an operation on top of the
*LogOp. We might still be using the *LogOp.Cid.Allocations
slice. We need to make a deep of *LogOp.Cid before
returning from ApplyTo.
This one was tricky...
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Issue #572 exposes metrics but they carry the peer ID in binary.
This was ok with our internal codecs but it doesn't seem to work
very well with json, and makes the output format unusable.
This makes the Metric.Peer field a string.
Additinoally, fixes calling the command without arguments and displaying
the date in the right format.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Support the new endpoint for later metrics in `rest/api/client`
Support the new method created in `rest/api/client` in
ipfs-cluster-ctl. i.e. `ipfs-cluster-ctl health metrics <name>` would
show the peers and the last list of metrics logged for each as returned
by the Peer Monitor, in a friendly way.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Opened new endpoint `GET /health/metrics/<name>` which would respond
with metrics of type <name>
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
By default, CORS headers allowing GET requests from everywhere are
set. This should facilitate the IPFS Web UI integration with the
Cluster API.
This commit refactors the sendResponse methods in the API, merging
them into one as it was difficult to follow the flows that actually
send something to the client. All tests now check the presence of
the configured headers too, to make sure no route was missed.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Adding more missing pieces in config
Use the right package(not the inbuilt one)
Setup rpc client for proxy in the cluster
Add back SetClient and Shutdown into Connector as they are required to
implement Component interface
Add `ipfsproxy` as into list of logging identifier and add its default
log level
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Go through `ipfsproxy` repo and change things that are inappropriate
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Extract the IPFS Proxy from ipfshttp and make it an api module
The `ipfshttp` IPFSConnector implementation includes the so called IPFS
Proxy. An endpoint which offers an IPFS API, hijacking some interesting
requests and forwarding the rest to the ipfs daemon.
`ipfshttp` should contain an implementation of IPFSConnector whose only
task should be to talk to IPFS
A new module should be created, `api/ipfsproxy`, an API Component
implementation for Cluster. The whole proxy code should be moved here.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
This straigthens some mistakes with the outputs of the /add endpoints.
Currently, we had exactly the same output format which:
* was not exactly the ipfs API output format but was sort of similar
* made some weird concessions to be compatible (like having a string-type "size")
* was not aligned with Cluster API conventions (lowercase keys)
This corrects all this:
* The Cluster API /add output format now uses the right types and lowercase keys.
* `Hash` is now `Cid`, because the field carries a Cid.
* We copy error handling with request trailers from IPFS, and avoid carrying the
errors in the output objects.
* The proxy now returns exactly the types as ipfs would
* We add the X-Chunked-Output: 1 header, which is custom and redundant, but
otherwise breaks js-ipfs-api integrations with the /add endpoint.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This is a workaround to have clients behave properly with the /add
endpoint by asking them to close connections when done, effectively
disabling keep-alive for this.
This means we don't need to disable keep-alives fully on all servers,
since the rest of endpoints are not affected (they are not streaming
endpoints).
Reference https://github.com/ipfs/go-ipfs/issues/5168
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This commit adds peername to PinInfo and GlobalPinInfo so that we have
a nicer and more meaningfull output for `ipfs-cluster-ctl` queries like
`status`, `sync` and `recover`
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
This commit adds peername to PinInfo and GlobalPinInfo so that we have
a nicer and more meaningfull output for queries like
`ipfs-cluster-ctl status`
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
.Add(paths) will interpret http* paths as WebFiles. These are read performing
a GET request to the location. Otherwise, the path is interpreted as a local
disk file/folder, and read from disk. ipfs-cluster-ctl has been updated
accordingly.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
See discussion in https://github.com/ipfs/go-ipfs/issues/5168
We cannot stream responses with keep-alives enabled.
I prefer this to not be a client feature, as otherwise users might end up
shooting themselves in the foot.
Note, the price is a corrupted request body which gets added
normally and gives wrong hashes!
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
