Commit Graph

93 Commits

Author SHA1 Message Date
Kishan Sagathiya
30fd5ee6a0 Feat: Allow and run multiple informer components (#962)
This introduces the possiblity of running Cluster with multiple informer components. The first one in the list is the used for "allocations". This is just groundwork for working with several informers in the future.
2019-12-05 15:08:43 +01:00
Hector Sanjuan
249d9007d2 Merge branch 'master' into feat/cluster-gc 2019-11-07 18:35:42 +01:00
Hector Sanjuan
af29bbc944 Pin-expiration: error when pinning expired pins 2019-11-05 12:22:52 +01:00
Kishan Mohanbhai Sagathiya
492b5612e7 Add ability to run Garbage Collector on all peers
- cluster method, ipfs connector method, rpc and rest apis,
command, etc for repo gc
    - Remove extra space from policy generator
    - Added special timeout for `/repo/gc` call to IPFS
    - Added `RepoGCLocal` cluster rpc method, which will be used to run gc
    on local IPFS daemon
    - Added peer name to the repo gc struct
    - Sorted with peer ids, while formatting(only affects cli
    results)
    - Special timeout setting where timeout gets checked from last update
    - Added `local` argument, which would run gc only on contacted peer
2019-10-22 11:13:19 +05:30
Hector Sanjuan
1eade4ae58 Fix #732: Introduce native pin/update
This introduces a pin/update operation which allows to Pin a new item to
cluster indicating that said pin is an update to an already-existing pin.

When this is the case, all the configuration for the existing pin is copied to
the new one (including allocations). The IPFS connector will then trigger
pin/update directly in IPFS, allowing an efficient pinning based on
DAG-differences. Since the allocations where the same for both pins,
the pin/update can proceed.

PinUpdate does not unpin the previous pin (it is not possible to do this
atomically in cluster like it happens in IPFS). The user can manually do it
after the pin/update is done.

Internally, after a lot of deliberations on what the optimal way for this is,
I opted for adding a `PinUpdate` option to the `PinOptions` type (carries the
CID to update from). In order to carry this option from the REST API to the
IPFS Connector, it is serialized in the Protobuf (and stored in the
datastore). There is no other way to do this in a simple fashion since the Pin
object is piece of information that is sent around.

Additionally, making it a PinOption plays well with the Pin/PinPath APIs which
need little changes. Effectively, you are pinning a new thing. You are just
indicating that it should be configured from an existing one.

Fixes #732
2019-08-09 16:11:52 +02:00
Hector Sanjuan
7c636061bd
Improve pin/unpin method signatures (#843)
* Improve pin/unpin method signatures:

These changes the following Cluster Go API methods:

* -> Cluster.Pin(ctx, cid, options) (pin, error)
* -> Cluster.Unpin(ctx, cid) (pin, error)
* -> Cluster.PinPath(ctx, path, opts) (pin,error)

Pin and Unpin now return the pinned object.

The signature of the methods now matches that of the API Client, is clearer as
to what options the user can set and is aligned with PinPath, UnpinPath, which
returned pin methods.

The REST API now returns the Pinned/Unpinned object rather than 204-Accepted.

This was necessary for a cleaner pin/update approach, which I'm working on in
another branch.

Most of the changes here are updating tests to the new signatures

* Adapt load-balancing client to new Pin/Unpin signatures

* cluster.go: Fix typo

Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>

* cluster.go: Fix typo

Co-Authored-By: Kishan Sagathiya <kishansagathiya@gmail.com>
2019-07-22 15:39:11 +02:00
Hector Sanjuan
b804e61ef0 Update deps along with go-libp2p-core refactor
Lots of rewrites in imports...
2019-06-14 13:10:45 +02:00
Hector Sanjuan
8e6eefb714 Tests: multiple fixes
This fixes multiple issues in and around tests while
increasing ttls and delays in 100ms. Multiple issues, including
races, tests not running with consensus-crdt missing log messages
and better initialization have been fixed.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2019-05-20 23:45:04 +02:00
Hector Sanjuan
d51c2a0377 Merge branch 'master' into feat/monitor-ring 2019-05-16 15:46:30 +02:00
Kishan Mohanbhai Sagathiya
f05af75abc Tests for identity separation
Added tests for identity.go and modifies others according ly

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2019-05-08 21:54:59 +05:30
Adrian Lanzafame
b1dbc1a584
pass ctx to NewCluster in tests
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2019-04-29 18:10:30 +10:00
Adrian Lanzafame
3c09ebcc71
add Alerts measure
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2019-04-26 17:56:44 +10:00
Hector Sanjuan
acbd7fda60 Consensus: add new "crdt" consensus component
This adds a new "crdt" consensus component using go-ds-crdt.

This implies several refactors to fully make cluster consensus-component
independent:

* Delete mapstate and fully adopt dsstate (after people have migrated).
* Return errors from state methods rather than ignoring them.
* Add a new "datastore" modules so that we can configure datastores in the
   main configuration like other components.
* Let the consensus components fully define the "state.State". Thus, they do
not receive the state, they receive the storage where we put the state (a
go-datastore).
* Allow to customize how the monitor component obtains Peers() (the current
  peerset), including avoiding using the current peerset. At the moment the
  crdt consensus uses the monitoring component to define the current peerset.
  Therefore the monitor component cannot rely on the consensus component to
  produce a peerset.
* Re-factor/re-implementation of "ipfs-cluster-service state"
  operations. Includes the dissapearance of the "migrate" one.

The CRDT consensus component defines creates a crdt-datastore (with ipfs-lite)
and uses it to intitialize a dssate. Thus the crdt-store is elegantly
wrapped. Any modifications to the state get automatically replicated to other
peers. We store all the CRDT DAG blocks in the local datastore.

The consensus components only expose a ReadOnly state, as any modifications to
the shared state should happen through them.

DHT and PubSub facilities must now be created outside of Cluster and passed in
so they can be re-used by different components.
2019-04-17 19:14:26 +02:00
Kishan Sagathiya
962d249e74
Remove basic monitor (#726)
Remove basic monitor

This commit removes `basic` monitor component, because it is not being
used by default since few releases ago pubsub monitor was introduced.

Issue #689
2019-03-21 22:48:40 +05:30
Hector Sanjuan
ea85cf7805 Rename "test.Test*" to "test.*" (test.TestCid1 -> test.Cid1)
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-27 20:19:10 +00:00
Hector Sanjuan
9df6344a07 Avoid using string testing CIDs and use cid.Cids directly
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-27 20:09:31 +00:00
Hector Sanjuan
6447ea51d2 Remove *Serial types. Use pointers for all types.
This takes advantange of the latest features in go-cid, peer.ID and
go-multiaddr and makes the Go types serializable by default.

This means we no longer need to copy between Pin <-> PinSerial, or ID <->
IDSerial etc. We can now efficiently binary-encode these types using short
field keys and without parsing/stringifying (in many cases it just a cast).

We still get the same json output as before (with minor modifications for
Cids).

This should greatly improve Cluster performance and memory usage when dealing
with large collections of items.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-27 17:04:35 +00:00
Kishan Mohanbhai Sagathiya
5fb2b6ae95 Add PinPath/UnpinPath support.
Squashed commit of the following:

commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date:   Wed Feb 20 11:02:00 2019 +0000

    Merge pull request #634 from ipfs/issue_450

    Support PinPath, UnpinPath (resolve before pinning)

commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 19 22:16:25 2019 +0530

    formatResponse accepts api.Pin and not api.PinSerial

commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 19 21:36:46 2019 +0530

    Merge branch 'master' into issue_450

commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Mon Feb 18 08:41:11 2019 +0530

    Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450

commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Mon Feb 18 08:31:39 2019 +0530

    PinPath: more improvements

    Added tracing for new methods

commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date:   Wed Feb 13 15:22:49 2019 +0000

    Merge branch 'master' into issue_450

commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 12 13:10:13 2019 +0530

    Merge branch 'master' into issue_450

commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 12 13:07:25 2019 +0530

    PinPath: more improvements

    - Worth having `PinOptions` as a separate field in the struct and
    constructing the query in the test with ToQuery()
    - sharness: "intialization" line can be placed outside the tests at
    the top

commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Sun Feb 10 21:43:50 2019 +0530

    Using if-continue pattern instead of if-else

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Thu Feb 7 10:25:52 2019 +0530

    Merge branch 'master' into issue_450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 21:11:20 2019 +0530

    Removing resolved path

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 16:45:07 2019 +0530

    Fix go vet

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 16:07:03 2019 +0530

    Fixed linting error

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 09:20:41 2019 +0530

    Fixed tests

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 00:07:10 2019 +0530

    Merge branch 'master' into issue_450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date:   Tue Feb 5 16:58:50 2019 +0000

    PinPath: address some feedback + improvements

    * Changed client's Pin() API and PinPath to be consistent
    * Added helper methods to turn PinPath to query and back
    * Make code and tests build
    * Use TestCidResolved everywhere
    * Fix cluster.PinPath arguments
    * Fix formatting of responses with --no-status
    * Make tests readable and call Fatal when needed
    * Use a pathTestCases variable

commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 5 18:34:26 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    Addressed review comments as in
    https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 29 22:41:27 2019 +0530

    Fixing tests

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 29 14:52:53 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    - PinPath and UnpinPath should return api.Pin
    - PinPath should accept pin options
    - Removing duplicate logic for Resolve from cluster
    - And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 23 17:08:41 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    PinPath(in both rest and rpc) should return a serializable struct in the
    form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
    to marshal and unmarshal)

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 23 07:18:56 2019 +0530

    Merge branch 'master' into issue_450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 22 18:14:32 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    - RPC must always use serializable structs
    - In command, just use pin with path as cid is also a valid path
    - Addressing many other small review comments as in
    https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 16 09:36:42 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    Extra logic for path checking should go into resolve so that it can be
    properly reused
    Added sharness tests

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 16 08:08:07 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    error strings should not be capitalized

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 15 23:40:25 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    Tests

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 15 10:40:54 2019 +0530

    Merge branch 'master' into issue_450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Mon Jan 14 20:58:17 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    - IPFS Connector should act as a pure IPFS client, any extra logic
    should go to cluster.go
    - Use cid.Undef, instead of cid.Cid{}

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Thu Jan 10 08:57:17 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    - Separate handlers, methods and rpc apis for PinPath and UnpinPath from
    Pin and Unpin
    - Support ipld paths as well

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 9 19:38:35 2019 +0530

    Merge branch 'issue_450_old' into HEAD

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 9 19:36:41 2019 +0530

    Revert "WIP: Figure out why test does not impleme"

    This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date:   Tue Jul 24 23:23:10 2018 -0400

    WIP: Figure out why test does not implement IPFSConnector interface...

    License: MIT
    Signed-off-by: cd10012 <ced361@nyu.edu>

commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 8 10:37:59 2019 +0530

    Support PinPath, UnpinPath (resolve before pinning)

    This commit adds API support for pinning using path

    `POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
    will resolve the path into a cid and perform perform pinning or
    unpinning

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

Co-authored-by: Hector Sanjuan <hector@protocol.ai>

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:13:03 +00:00
Adrian Lanzafame
3b3f786d68
add opencensus tracing and metrics
This commit adds support for OpenCensus tracing
and metrics collection. This required support for
context.Context propogation throughout the cluster
codebase, and in particular, the ipfscluster component
interfaces.

The tracing propogates across RPC and HTTP boundaries.
The current default tracing backend is Jaeger.

The metrics currently exports the metrics exposed by
the opencensus http plugin as well as the pprof metrics
to a prometheus endpoint for scraping.
The current default metrics backend is Prometheus.

Metrics are currently exposed by default due to low
overhead, can be turned off if desired, whereas tracing
is off by default as it has a much higher performance
overhead, though the extent of the performance hit can be
adjusted with smaller sampling rates.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2019-02-04 18:53:21 +10:00
Hector Sanjuan
862c1eb3ea Fix #382: Extract headers from IPFS API requests & apply them to hijacked ones.
This commit makes the proxy extract useful fixed headers (like CORS) from
the IPFS daemon API responses and then apply them to the responses
from hijacked endpoints like /add or /repo/stat.

It does this by caching a list of headers from the first IPFS API
response which has them. If we have not performed any proxied request or
managed to obtain the headers we're interested in, this will try triggering a
request to "/api/v0/version" to obtain them first.

This should fix the issues with using Cluster proxy with IPFS Companion and
Chrome.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-12-18 16:05:12 +01:00
Kishan Sagathiya
aef68f4101 Issue #453 Extract the IPFS Proxy from ipfshttp
Fixed breaking tests

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-11-11 16:23:36 +05:30
Hector Sanjuan
7d16108751 Start using libp2p/go-libp2p-gorpc
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-17 15:28:03 +02:00
Hector Sanjuan
cbdd075bb2
Merge pull request #551 from ipfs/fix/minor-version-compat
Allow running peers with different cluster versions in the same cluster
2018-09-27 08:13:53 +02:00
Hector Sanjuan
59714f69d4 Allow running peers with different cluster versions in the same cluster
This patch modifies the RPC protocol tag to use Major and Minor parts of the
version and not all of it.

This means all peers on the 0.5.x can run in the same cluster.

As cluster has become more mature and I see less risks in letting peers from
similar versions run together. This is useful when upgrading too.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-09-27 07:35:57 +02:00
Kishan Sagathiya
b76b65e446 Issue #446 Adding peername to PinInfo
Removed unnecessary peername assignment
Modified tests according to the changes made to add peername to PinInfo

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-09-24 23:04:16 +05:30
Adrian Lanzafame
31474f6490
update go-cid and go-libp2p
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-09-24 11:35:38 +10:00
Hector Sanjuan
2ffa3d80de Fix 466: Hijack repo/stat in the proxy and return aggregates from the cluster.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 20:43:27 +02:00
Adrian Lanzafame
c6eada9db5 uses new gorpc method to distinguish err type
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-08-14 13:34:02 +02:00
Adrian Lanzafame
df2753dfc6 implements a stateless pintracker
Also updates to the optracker to make retrieving information easier.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-08-14 13:33:36 +02:00
Hector Sanjuan
c2312cbb05 Fix: close files. Close test files. Clean properly. Fix windows tests.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 12:05:27 +02:00
Hector Sanjuan
8f1a15b279 Move adder.Params to api.AddParams. Re-use in other modules
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
327a81b85a golint govets
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
623120fd50 Start cluster tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
65dc17a78b testfixing
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Wyatt Daviau
463840bab3 more pin tests
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
ecec7c24b0 new pin-test: sharded pins sharing a shard in cdag
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
fabf191883 AddFile tests
tests cover local and sharded adds of files
ipfs mock and ipfs block put/get calls cleaned up

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
238f3726f3 Pin datastructure updated to support sharding
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
6b8bcdfbc3 Finishing feedback and adding new test
Also updating Makefile to fetch sharness more robustly

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
edb38b2830 fixing make check errors
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
11e8e9d62c Addressing second round of comments
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
205b61e768 add BlockPut to ipfs mock
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Hector Sanjuan
4d8f975d9b StateSync(): some improvements
This commit:

* Does not collect and return changed items when doing StateSync (they are
not used)
* Removes the StateSync RPC method (no longer used)
* Uses tracker.StatusAll() rather than requesting Status on each Cid (should
be faster with upcoming pintracker)
* Does not launch a go-routine to track every item. Track is an async
operation. This likely causes 1000s goroutines to be started with no good
reason.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-25 09:58:18 +02:00
Hector Sanjuan
5ca8ca39eb Monitor/tests: Allow to run tests using the basic monitor.
Do it in additional stage in Travis.

Also, test fixes.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-09 11:39:21 +02:00
Hector Sanjuan
bb8c20b2fb Enable pubsubmon in cluster e2e tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-07 18:47:05 +02:00
Hector Sanjuan
029cd77c27
Merge pull request #398 from ipfs/feat/promote-consensus
Emancipate the consensus component
2018-05-07 08:29:14 +02:00
Hector Sanjuan
33d9cdd3c4 Feat: emancipate Consensus from the Cluster component
This commit promotes the Consensus component (and Raft) to become a fully
independent thing like other components, passed to NewCluster during
initialization. Cluster (main component) no longer creates the consensus
layer internally. This has triggered a number of breaking changes
that I will explain below.

Motivation: Future work will require the possibility of running Cluster
with a consensus layer that is not Raft. The "consensus" layer is in charge
of maintaining two things:
  * The current cluster peerset, as required by the implementation
  * The current cluster pinset (shared state)

While the pinset maintenance has always been in the consensus layer, the
peerset maintenance was handled by the main component (starting by the "peers"
key in the configuration) AND the Raft component (internally)
and this generated lots of confusion: if the user edited the peers in the
configuration they would be greeted with an error.

The bootstrap process (adding a peer to an existing cluster) and configuration
key also complicated many things, since the main component did it, but only
when the consensus was initialized and in single peer mode.

In all this we also mixed the peerstore (list of peer addresses in the libp2p
host) with the peerset, when they need not to be linked.

By initializing the consensus layer before calling NewCluster, all the
difficulties in maintaining the current implementation in the same way
have come to light. Thus, the following changes have been introduced:

* Remove "peers" and "bootstrap" keys from the configuration: we no longer
edit or save the configuration files. This was a very bad practice, requiring
write permissions by the process to the file containing the private key and
additionally made things like Puppet deployments of cluster difficult as
configuration would mutate from its initial version. Needless to say all the
maintenance associated to making sure peers and bootstrap had correct values
when peers are bootstrapped or removed. A loud and detailed error message has
been added when staring cluster with an old config, along with instructions on
how to move forward.

* Introduce a PeerstoreFile ("peerstore") which stores peer addresses: in
ipfs, the peerstore is not persisted because it can be re-built from the
network bootstrappers and the DHT. Cluster should probably also allow
discoverability of peers addresses (when not bootstrapping, as in that case
we have it), but in the meantime, we will read and persist the peerstore
addresses for cluster peers in this file, different from the configuration.
Note that dns multiaddresses are now fully supported and no IPs are saved
when we have DNS multiaddresses for a peer.

* The former "peer_manager" code is now a pstoremgr module, providing utilities
to parse, add, list and generally maintain the libp2p host peerstore, including
operations on the PeerstoreFile. This "pstoremgr" can now also be extended to
perform address autodiscovery and other things indepedently from Cluster.

* Create and initialize Raft outside of the main Cluster component: since we
can now launch Raft independently from Cluster, we have more degrees of
freedom. A new "staging" option when creating the object allows a raft peer to
be launched in Staging mode, waiting to be added to a running consensus, and
thus, not electing itself as leader or doing anything like we were doing
before. This additionally allows us to track when the peer has become a
Voter, which only happens when it's caught up with the state, something that
was wonky previously.

* The raft configuration now includes an InitPeerset key, which allows to
provide a peerset for new peers and which is ignored when staging==true. The
whole Raft initialization code is way cleaner and stronger now.

* Cluster peer bootsrapping is now an ipfs-cluster-service feature. The
--bootstrap flag works as before (additionally allowing comma-separated-list
of entries). What bootstrap does, is to initialize Raft with staging == true,
and then call Join in the main cluster component. Only when the Raft peer
transitions to Voter, consensus becomes ready, and cluster becomes Ready.
This is cleaner, works better and is less complex than before (supporting
both flags and config values). We also backup and clean the state whenever
we are boostrapping, automatically

* ipfs-cluster-service no longer runs the daemon. Starting cluster needs
now "ipfs-cluster-service daemon". The daemon specific flags (bootstrap,
alloc) are now flags for the daemon subcommand. Here we mimic ipfs ("ipfs"
does not start the daemon but print help) and pave the path for merging both
service and ctl in the future.

While this brings some breaking changes, it significantly reduces the
complexity of the configuration, the code and most importantly, the
documentation. It should be easier now to explain the user what is the
right way to launch a cluster peer, and more difficult to make mistakes.

As a side effect, the PR also:

* Fixes #381 - peers with dynamic addresses
* Fixes #371 - peers should be Raft configuration option
* Fixes #378 - waitForUpdates may return before state fully synced
* Fixes #235 - config option shadowing (no cfg saves, no need to shadow)

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-07 07:39:41 +02:00
Adrian Lanzafame
9e20e4e3b2 ipfsconn/ipfshttp: Pass ctx through from rpc_api
to the ipfscluster.IPFSConnector interface and then
to the implementation of that interface in ipfsconn/ipfshttp.
This allows calls from MapPinTracker to cancel requests made
to the local IPFS node.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-05-02 15:24:26 +02:00
Hector Sanjuan
dd4128affc Fix #339: Reduce Sleeps in tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-05 16:49:26 +02:00
Hector Sanjuan
41b17bf477 Cluster: add libp2p host parameter to constructor.
NewCluster() now takes an optional Host parameter.
The rationale is to allow to re-use an existing libp2p Host
when creating the cluster.

The NewClusterHost method now allows to create a host
with the options used by cluster.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-15 00:04:54 +01:00