This commit introduces an api.Cid type and replaces the usage of cid.Cid
everywhere.
The main motivation here is to override MarshalJSON so that Cids are
JSON-ified as '"Qm...."' instead of '{ "/": "Qm....." }', as this "ipld"
representation of IDs is horrible to work with, and our APIs are not issuing
IPLD objects to start with.
Unfortunately, there is no way to do this cleanly, and the best way is to just
switch everything to our own type.
This commit fixes#810 and adds block streaming to the final destinations when
adding. This should add major performance gains when adding data to clusters.
Before, everytime cluster issued a block, it was broadcasted individually to
all destinations (new libp2p stream), where it was block/put to IPFS (a single
block/put http roundtrip per block).
Now, blocks are streamed all the way from the adder module to the ipfs daemon,
by making every block as it arrives a single part in a multipart block/put
request.
Before, block-broadcast needed to wait for all destinations to finish in order
to process the next block. Now, buffers allow some destinations to be faster
than others while sending and receiving blocks.
Before, if a block put request failed to be broadcasted everywhere, an error
would happen at that moment.
Now, we keep streaming until the end and only then report any errors. The
operation succeeds as long as at least one stream finished successfully.
Errors block/putting to IPFS will not abort streams. Instead, subsequent
blocks are retried with a new request, although the method will return an
error when the stream finishes if there were errors at any point.
This commit continues the work of taking advantage of the streaming
capabilities in go-libp2p-gorpc by improving the ipfsconnector and pintracker
components.
StatusAll and RecoverAll methods are now streaming methods, with the REST API
output changing accordingly to produce a stream of GlobalPinInfos rather than
a json array.
pin/ls request to the ipfs daemon now use ?stream=true and avoid having to
load the full pinset map on memory. StatusAllLocal and RecoverAllLocal
requests to the pin tracker stream all the way and no longer store the full
pinset, and the full PinInfo status slice before sending it out.
We have additionally switched to a pattern where streaming methods receive the
channel as an argument, allowing the caller to decide on whether to launch a
goroutine, do buffering etc.
This commit introduces the new go-libp2p-gorpc streaming capabilities for
Cluster. The main aim is to work towards heavily reducing memory usage when
working with very large pinsets.
As a side-effect, it takes the chance to revampt all types for all public
methods so that pointers to static what should be static objects are not used
anymore. This should heavily reduce heap allocations and GC activity.
The main change is that state.List now returns a channel from which to read
the pins, rather than pins being all loaded into a huge slice.
Things reading pins have been all updated to iterate on the channel rather
than on the slice. The full pinset is no longer fully loaded onto memory for
things that run regularly like StateSync().
Additionally, the /allocations endpoint of the rest API no longer returns an
array of pins, but rather streams json-encoded pin objects directly. This
change has extended to the restapi client (which puts pins into a channel as
they arrive) and to ipfs-cluster-ctl.
There are still pending improvements like StatusAll() calls which should also
stream responses, and specially BlockPut calls which should stream blocks
directly into IPFS on a single call.
These are coming up in future commits.
Currently we were only specifying the block format. When adding with
a custom hash function, even though we produced the right cids, IPFS
did not know the hash function and ended up storing them using SHA256.
Additionally, since NodeWithMeta serializes the CID, we do not need
to carry a Format parameter (which specifies the Codec): it is already
embedded.
Tests have been added and BlockPut in ipfshttp now checks that the
response's CID matches the data sent. This will catch errors like
what was happening, but also any data corruption between cluster and
IPFS during the block upload.
* add ipv6 listening addresses to the default config
* ipfsproxy: support multiple listeners. Add default ipv6.
* mm
* restapi: support multiple listen addresses. enable ipv6
* cluster_config: format default listen addresses
* commands: update for multiple listeners. Fix randomports for udp and ipv6.
* ipfs-cluster-service: fix randomports test
* multiple listeners: fix remaining tests
* golint
* Disable ipv6 in defaults
It is not supported by docker by default. It is not supported in travis-CI
build environments. User can enable it now manually.
* proxy: disable ipv6 in test
* ipfshttp: fix test
Co-authored-by: @RubenKelevra <cyrond@gmail.com>
- cluster method, ipfs connector method, rpc and rest apis,
command, etc for repo gc
- Remove extra space from policy generator
- Added special timeout for `/repo/gc` call to IPFS
- Added `RepoGCLocal` cluster rpc method, which will be used to run gc
on local IPFS daemon
- Added peer name to the repo gc struct
- Sorted with peer ids, while formatting(only affects cli
results)
- Special timeout setting where timeout gets checked from last update
- Added `local` argument, which would run gc only on contacted peer
This introduces a pin/update operation which allows to Pin a new item to
cluster indicating that said pin is an update to an already-existing pin.
When this is the case, all the configuration for the existing pin is copied to
the new one (including allocations). The IPFS connector will then trigger
pin/update directly in IPFS, allowing an efficient pinning based on
DAG-differences. Since the allocations where the same for both pins,
the pin/update can proceed.
PinUpdate does not unpin the previous pin (it is not possible to do this
atomically in cluster like it happens in IPFS). The user can manually do it
after the pin/update is done.
Internally, after a lot of deliberations on what the optimal way for this is,
I opted for adding a `PinUpdate` option to the `PinOptions` type (carries the
CID to update from). In order to carry this option from the REST API to the
IPFS Connector, it is serialized in the Protobuf (and stored in the
datastore). There is no other way to do this in a simple fashion since the Pin
object is piece of information that is sent around.
Additionally, making it a PinOption plays well with the Pin/PinPath APIs which
need little changes. Effectively, you are pinning a new thing. You are just
indicating that it should be configured from an existing one.
Fixes#732
- Fixed coverage by adding progress to response in mock
- Used switch instead of ifs
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
The switch to base32 as default output format for cidv1 keys in IPFS pin/ls
responses causes an error in PinLsCid as the base32 string does not correspond
to the base58 string expected by cluster when using an old peer.
This affects old Cluster peers running with new IPFS versions and new cluster
peers running with old IPFS versions for v1 CIDs.
Since Current master uses an updated cid dependency which also uses base32 by
default, master would already work with the latest IPFS daemon, so this is
just allowing to use cluster peers with older ipfs daemons, and preventing a
similar breakage in the future.
This adds a new "crdt" consensus component using go-ds-crdt.
This implies several refactors to fully make cluster consensus-component
independent:
* Delete mapstate and fully adopt dsstate (after people have migrated).
* Return errors from state methods rather than ignoring them.
* Add a new "datastore" modules so that we can configure datastores in the
main configuration like other components.
* Let the consensus components fully define the "state.State". Thus, they do
not receive the state, they receive the storage where we put the state (a
go-datastore).
* Allow to customize how the monitor component obtains Peers() (the current
peerset), including avoiding using the current peerset. At the moment the
crdt consensus uses the monitoring component to define the current peerset.
Therefore the monitor component cannot rely on the consensus component to
produce a peerset.
* Re-factor/re-implementation of "ipfs-cluster-service state"
operations. Includes the dissapearance of the "migrate" one.
The CRDT consensus component defines creates a crdt-datastore (with ipfs-lite)
and uses it to intitialize a dssate. Thus the crdt-store is elegantly
wrapped. Any modifications to the state get automatically replicated to other
peers. We store all the CRDT DAG blocks in the local datastore.
The consensus components only expose a ReadOnly state, as any modifications to
the shared state should happen through them.
DHT and PubSub facilities must now be created outside of Cluster and passed in
so they can be re-used by different components.
Squashed commit of the following:
commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 20 11:02:00 2019 +0000
Merge pull request #634 from ipfs/issue_450
Support PinPath, UnpinPath (resolve before pinning)
commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 22:16:25 2019 +0530
formatResponse accepts api.Pin and not api.PinSerial
commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 19 21:36:46 2019 +0530
Merge branch 'master' into issue_450
commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:41:11 2019 +0530
Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450
commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Feb 18 08:31:39 2019 +0530
PinPath: more improvements
Added tracing for new methods
commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date: Wed Feb 13 15:22:49 2019 +0000
Merge branch 'master' into issue_450
commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:10:13 2019 +0530
Merge branch 'master' into issue_450
commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 12 13:07:25 2019 +0530
PinPath: more improvements
- Worth having `PinOptions` as a separate field in the struct and
constructing the query in the test with ToQuery()
- sharness: "intialization" line can be placed outside the tests at
the top
commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Sun Feb 10 21:43:50 2019 +0530
Using if-continue pattern instead of if-else
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Feb 7 10:25:52 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 21:11:20 2019 +0530
Removing resolved path
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:45:07 2019 +0530
Fix go vet
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 16:07:03 2019 +0530
Fixed linting error
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 09:20:41 2019 +0530
Fixed tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Feb 6 00:07:10 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date: Tue Feb 5 16:58:50 2019 +0000
PinPath: address some feedback + improvements
* Changed client's Pin() API and PinPath to be consistent
* Added helper methods to turn PinPath to query and back
* Make code and tests build
* Use TestCidResolved everywhere
* Fix cluster.PinPath arguments
* Fix formatting of responses with --no-status
* Make tests readable and call Fatal when needed
* Use a pathTestCases variable
commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Feb 5 18:34:26 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Addressed review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 22:41:27 2019 +0530
Fixing tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 29 14:52:53 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- PinPath and UnpinPath should return api.Pin
- PinPath should accept pin options
- Removing duplicate logic for Resolve from cluster
- And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 17:08:41 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
PinPath(in both rest and rpc) should return a serializable struct in the
form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
to marshal and unmarshal)
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 23 07:18:56 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 22 18:14:32 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- RPC must always use serializable structs
- In command, just use pin with path as cid is also a valid path
- Addressing many other small review comments as in
https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 09:36:42 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Extra logic for path checking should go into resolve so that it can be
properly reused
Added sharness tests
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 16 08:08:07 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
error strings should not be capitalized
Fixes#450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 23:40:25 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
Tests
Fixes#450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 15 10:40:54 2019 +0530
Merge branch 'master' into issue_450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Mon Jan 14 20:58:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- IPFS Connector should act as a pure IPFS client, any extra logic
should go to cluster.go
- Use cid.Undef, instead of cid.Cid{}
Fixes#450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Thu Jan 10 08:57:17 2019 +0530
Support PinPath, UnpinPath(resolve before pinning)
- Separate handlers, methods and rpc apis for PinPath and UnpinPath from
Pin and Unpin
- Support ipld paths as well
Fixes#450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:38:35 2019 +0530
Merge branch 'issue_450_old' into HEAD
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Wed Jan 9 19:36:41 2019 +0530
Revert "WIP: Figure out why test does not impleme"
This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date: Tue Jul 24 23:23:10 2018 -0400
WIP: Figure out why test does not implement IPFSConnector interface...
License: MIT
Signed-off-by: cd10012 <ced361@nyu.edu>
commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date: Tue Jan 8 10:37:59 2019 +0530
Support PinPath, UnpinPath (resolve before pinning)
This commit adds API support for pinning using path
`POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
will resolve the path into a cid and perform perform pinning or
unpinning
Fixes#450
License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Co-authored-by: Hector Sanjuan <hector@protocol.ai>
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
Since the beginning, we have used a Go map to store the shared state (pinset)
in memory. The mapstate knew how to serialize itself so that libp2p-raft would
know how to write to disk when it:
* Saved snapshots of the state on shutdown
* Sent the state to a newcomer peer
hashicorp.Raft assumes an in-memory state which is snapshotted from time to
time and read from disk on boot.
This commit adds a `dsstate` implementation of the state interface using
`go-datastore`. This allows to effortlessly switch to a disk-backed state in
the future (as we will need), and also have at our disposal the different
implementations and utilities of Datastore for fine-tuning (caching, batching
etc.).
`mapstate` has been reworked to use dsstate. Ideally, we would not even need
`mapstate`, as it would suffice to initialize `dsstate` with a
`MapDatastore`. BUT, we still need it separate to be able to auto-migrate to
the new format.
This will be the last migration with the current system. Once this has been
released and users have been able to upgrade we will just remove `mapstate` as
it is now.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This commit adds support for OpenCensus tracing
and metrics collection. This required support for
context.Context propogation throughout the cluster
codebase, and in particular, the ipfscluster component
interfaces.
The tracing propogates across RPC and HTTP boundaries.
The current default tracing backend is Jaeger.
The metrics currently exports the metrics exposed by
the opencensus http plugin as well as the pprof metrics
to a prometheus endpoint for scraping.
The current default metrics backend is Prometheus.
Metrics are currently exposed by default due to low
overhead, can be turned off if desired, whereas tracing
is off by default as it has a much higher performance
overhead, though the extent of the performance hit can be
adjusted with smaller sampling rates.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
This changes the current strategy to extract headers from the IPFS daemon to
use them for hijacked endpoints in the proxy. The ipfs daemon is a bit of a
mess and what we were doing is not really reliable, specially when it comes to
setting CORS headers right (which we were not doing).
The new approach is:
* For every hijacked request, make an OPTIONS request to the same path, with
the given Origin, to the IPFS daemon and extract some CORS headers from
that. Use those in the hijacked response
* Avoid hijacking OPTIONS request, they should always go through so the IPFS
daemon controls all the CORS-preflight things as it wants.
* Similar to before, have a only-once-triggered request to extract other
interesting or custom headers from a fixed IPFS endpoint. This allows us to
have the proxy forward other custom headers and to catch
`Access-Control-Expose-Methods`. The difference is that the endpoint use for
this and the additional headers are configurable by the user (but with hidden
configuration options because this is quite exotic from regular usage).
Now the implementation:
* Replaced the standard Muxer with gorilla/mux (I have also taken the change
to update the gxed version to the latest tag). This gives us much better
matching control over routes and allows us to not handle OPTIONS requests.
* This allows also to remove the extractArgument code and have proper handlers
for the endpoints passing command arguments as the last segment of the URL. A
very simple handler that wraps the default ones can be used to extract the
argument from the url and put it in the query. Overall much cleaner this way.
* No longer capture interesting headers from any random proxied request. This
made things complicated with a wrapping handler. We will just trigger the one
request to do it when we need it.
* When preparing the headers for the hijacked responses:
* Trigger the OPTIONS request and figure out which CORS things we should set
* Set the additional headers (perhaps triggering a POST request to fetch them)
* Set our own headers.
* Moved all the headers stuff to a new headers.go file.
* Added configuration options (hidden by default) to:
* Customize the extract headers endpoint
* Customize what additional headers are extracted
* Use HTTPs when talking to the IPFS API
* I haven't tested this, but I did not want to have hardcoded 'http://' urls
around, as before.
* Added extra testing for this, and tested manually a lot comparing the
daemon original output with our hijacked endpoint outputs while looking
at the API traffic with ngrep and making sure the requets happen as expected.
Also tested with IPFS companion in FF and Chrome.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This commit makes the proxy extract useful fixed headers (like CORS) from
the IPFS daemon API responses and then apply them to the responses
from hijacked endpoints like /add or /repo/stat.
It does this by caching a list of headers from the first IPFS API
response which has them. If we have not performed any proxied request or
managed to obtain the headers we're interested in, this will try triggering a
request to "/api/v0/version" to obtain them first.
This should fix the issues with using Cluster proxy with IPFS Companion and
Chrome.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
No more hacks around /add. This uses the local adder when hijacking /add.
It supports the parameters and works pretty well with the ipfs CLI, showing
progress and everything.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
tests cover local and sharded adds of files
ipfs mock and ipfs block put/get calls cleaned up
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
The extractCid function was added to enable the extraction of
a cid argument from either the url path or query string.
This puts the proxy behaviour on par with the current IPFS API.
The function does rely on the fact that ipfs-cluster doesn't
intercept any command that has more than one subcommand.
If that changes, this function will have to be updated.
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
This fixes#326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.
It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.
Documentation for the configuration file has been updated.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Added go tests
Refactored cluster connect graph to new file
Refactored dot file printing to new repo
Fixed code climate issues
Added sharness test
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
Up to now, we hardcoded progress to "false" in the proxy, regardless
of what the original request said. We now leave it as it is, and
just ignore any progress updates when processing the response.
Since the response is buffered and sent back all together, they are
still useless, but at least the clients (ipfs cli) won't show a 0%
progress bar when successfully adding a file.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
I have updated API endpoints to be /allocations rather than /pinlinst
It's more self-explanatory.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
The disk informer uses "ipfs repo stat" to fetch the RepoSize value and
uses it as a metric.
The numpinalloc allocator is now a generalized ascendalloc which
sorts metrics in ascending order and return the ones with lowest
values.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
