Commit Graph

158 Commits

Author SHA1 Message Date
Hector Sanjuan
9df6344a07 Avoid using string testing CIDs and use cid.Cids directly
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-27 20:09:31 +00:00
Hector Sanjuan
6447ea51d2 Remove *Serial types. Use pointers for all types.
This takes advantange of the latest features in go-cid, peer.ID and
go-multiaddr and makes the Go types serializable by default.

This means we no longer need to copy between Pin <-> PinSerial, or ID <->
IDSerial etc. We can now efficiently binary-encode these types using short
field keys and without parsing/stringifying (in many cases it just a cast).

We still get the same json output as before (with minor modifications for
Cids).

This should greatly improve Cluster performance and memory usage when dealing
with large collections of items.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-27 17:04:35 +00:00
Hector Sanjuan
0fed61192a Remove backwards compatibility hacks
The things removed here have been live for more than 2 releases.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 14:02:09 +00:00
Kishan Mohanbhai Sagathiya
5fb2b6ae95 Add PinPath/UnpinPath support.
Squashed commit of the following:

commit 38cf569c6aed77c46ee4e0f8baa4d1a9daf8f03e
Merge: d125f69 aaada42
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date:   Wed Feb 20 11:02:00 2019 +0000

    Merge pull request #634 from ipfs/issue_450

    Support PinPath, UnpinPath (resolve before pinning)

commit aaada42054e1f1c7b2abb1270859d0de41a0e5d8
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 19 22:16:25 2019 +0530

    formatResponse accepts api.Pin and not api.PinSerial

commit b5da4bea045865814cc422da71827b44ddd44b90
Merge: ba59036 cc8dd7e
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 19 21:36:46 2019 +0530

    Merge branch 'master' into issue_450

commit ba5903649c1df1dba20f4d6f7e3573d6fe24921f
Merge: f002914 d59880c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Mon Feb 18 08:41:11 2019 +0530

    Merge branch 'issue_450' of github.com:ipfs/ipfs-cluster into issue_450

commit f00291494c0c02621c2296cbb7ac71e4c23aa9ec
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Mon Feb 18 08:31:39 2019 +0530

    PinPath: more improvements

    Added tracing for new methods

commit d59880c338eaa8214fe06b4f930a540793d78407
Merge: 0ca4c7c b4f0eb3
Author: Hector Sanjuan <hsanjuan@users.noreply.github.com>
Date:   Wed Feb 13 15:22:49 2019 +0000

    Merge branch 'master' into issue_450

commit 0ca4c7c3b0670ed9c8279f8274d36e3485c10030
Merge: d35017a ecef9ea
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 12 13:10:13 2019 +0530

    Merge branch 'master' into issue_450

commit d35017a8de91ca9fc9a9a047c48c75134cee9f98
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 12 13:07:25 2019 +0530

    PinPath: more improvements

    - Worth having `PinOptions` as a separate field in the struct and
    constructing the query in the test with ToQuery()
    - sharness: "intialization" line can be placed outside the tests at
    the top

commit 68e3b90417ffbad89d41a70ac81d85f9037f8848
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Sun Feb 10 21:43:50 2019 +0530

    Using if-continue pattern instead of if-else

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 3c29799f3b85be328b27508332ab92049d8b82f3
Merge: 956790b 4324889
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Thu Feb 7 10:25:52 2019 +0530

    Merge branch 'master' into issue_450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 956790b381db9858e4194f983e898b07dc51ba66
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 21:11:20 2019 +0530

    Removing resolved path

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 7191cc46cedfbec116a9746937e28881b50ca044
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 16:45:07 2019 +0530

    Fix go vet

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit f8b3d5b63b1b7569e2a3e0d82894fd4491c246c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 16:07:03 2019 +0530

    Fixed linting error

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 23c57eb467755a1f21387a1615a7f34e97348053
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 09:20:41 2019 +0530

    Fixed tests

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 0caedd94aefeb3b6649dedc214cb4b849ace2ea4
Merge: 17e555e 5a7ee1d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Feb 6 00:07:10 2019 +0530

    Merge branch 'master' into issue_450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 17e555e4a7c574413df90aac70c5cc29cab98f54
Author: Hector Sanjuan <code@hector.link>
Date:   Tue Feb 5 16:58:50 2019 +0000

    PinPath: address some feedback + improvements

    * Changed client's Pin() API and PinPath to be consistent
    * Added helper methods to turn PinPath to query and back
    * Make code and tests build
    * Use TestCidResolved everywhere
    * Fix cluster.PinPath arguments
    * Fix formatting of responses with --no-status
    * Make tests readable and call Fatal when needed
    * Use a pathTestCases variable

commit f0e7369c47c5ddadc8ed45df5fd2d4d9b2d42b38
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Feb 5 18:34:26 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    Addressed review comments as in
    https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-198751932

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit a8b4f181d2d7afed32ee41331dfaab19fd66a173
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 29 22:41:27 2019 +0530

    Fixing tests

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit e39b95ca19e4d75506f4f492678245ef13936a44
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 29 14:52:53 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    - PinPath and UnpinPath should return api.Pin
    - PinPath should accept pin options
    - Removing duplicate logic for Resolve from cluster
    - And many other review comments https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-195509504

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit d146075126320896665ba58d337a13789f68ea86
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 23 17:08:41 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    PinPath(in both rest and rpc) should return a serializable struct in the
    form `{"\":"Q...cid..string..."}` (as used in "github.com/ipfs/go-cid"
    to marshal and unmarshal)

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 1f4869568a8adb450275257154ea3a26d03a30f3
Merge: 7acfd28 a244af9
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 23 07:18:56 2019 +0530

    Merge branch 'master' into issue_450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 7acfd282732ddf2282a67d4f9d0170a494eb3ed4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 22 18:14:32 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    - RPC must always use serializable structs
    - In command, just use pin with path as cid is also a valid path
    - Addressing many other small review comments as in
    https://github.com/ipfs/ipfs-cluster/pull/634#pullrequestreview-192122534

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 36905041e1e3f0b204942030aab3ab7b5b9e4d62
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 16 09:36:42 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    Extra logic for path checking should go into resolve so that it can be
    properly reused
    Added sharness tests

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 9116bda3534e77bb391d873051bb520a1b01a326
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 16 08:08:07 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    error strings should not be capitalized

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit ca7e61861374f456300a85ddc0374e594f74f963
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 15 23:40:25 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    Tests

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 522fbcd899f01c01680375561a32a87464157c0a
Merge: f1a56ab f7bc468
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 15 10:40:54 2019 +0530

    Merge branch 'master' into issue_450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit f1a56ab925fb74c0c44273a4524afa4843cf757f
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Mon Jan 14 20:58:17 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    - IPFS Connector should act as a pure IPFS client, any extra logic
    should go to cluster.go
    - Use cid.Undef, instead of cid.Cid{}

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit c83b91054f6774f1f9d4930cfc3f1fa28236f57c
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Thu Jan 10 08:57:17 2019 +0530

    Support PinPath, UnpinPath(resolve before pinning)

    - Separate handlers, methods and rpc apis for PinPath and UnpinPath from
    Pin and Unpin
    - Support ipld paths as well

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 719dff88129366ce3ccb5e04cb6f8082a0915c5c
Merge: 91ceb47 21170c4
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 9 19:38:35 2019 +0530

    Merge branch 'issue_450_old' into HEAD

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 91ceb4796259ca7ef2974ec43e6a278a12796b13
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Wed Jan 9 19:36:41 2019 +0530

    Revert "WIP: Figure out why test does not impleme"

    This reverts commit 28a3a3f25dce6f296c8cbef86221644c099a7e75.

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

commit 28a3a3f25dce6f296c8cbef86221644c099a7e75
Author: cd10012 <ced361@nyu.edu>
Date:   Tue Jul 24 23:23:10 2018 -0400

    WIP: Figure out why test does not implement IPFSConnector interface...

    License: MIT
    Signed-off-by: cd10012 <ced361@nyu.edu>

commit 21170c48e77e69583db64544b08120a9baf40d8d
Author: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
Date:   Tue Jan 8 10:37:59 2019 +0530

    Support PinPath, UnpinPath (resolve before pinning)

    This commit adds API support for pinning using path

    `POST /pins/<ipfs or ipns path>` and `DELETE /pins/<ipfs or ipns path>`
    will resolve the path into a cid and perform perform pinning or
    unpinning

    Fixes #450

    License: MIT
    Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

Co-authored-by: Hector Sanjuan <hector@protocol.ai>

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-20 11:13:03 +00:00
Hector Sanjuan
3059ab387a
Merge pull request #663 from roignpar/issue_656
Read config values from env on init command
2019-02-19 17:48:47 +00:00
Robert Ignat
be8c56f949 Add ApplyEnvVars test to ipfshttp config
License: MIT
Signed-off-by: Robert Ignat <robert.ignat91@gmail.com>
2019-02-18 17:47:02 +02:00
Hector Sanjuan
72d41790bd Ipfshttp: Fix: context tracing for sendInformerMetrics
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-16 00:11:07 +00:00
Hector Sanjuan
f22d01976d Fix context cancelled too early.
Every 10th pin the ipfs connector calls repo stat.

Because the call is async, the context was cancelled immediately and failed.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2019-02-15 18:45:04 +00:00
Robert Ignat
168cf76224 Change ApplyEnvVars strategy for all config components
Get jsonConfig from Config, apply env vars to it, load jsonConfig
back into Config.

License: MIT
Signed-off-by: Robert Ignat <robert.ignat91@gmail.com>
2019-02-15 19:07:20 +02:00
Robert Ignat
aa5d545164 Implement ApplyEnvVars for ipfshttp Config
License: MIT
Signed-off-by: Robert Ignat <robert.ignat91@gmail.com>
2019-02-13 23:13:28 +02:00
Robert Ignat
032f02802f Implement ApplyEnvVars for all ComponentConfigs
License: MIT
Signed-off-by: Robert Ignat <robert.ignat91@gmail.com>
2019-02-08 23:57:16 +02:00
Robert Ignat
ed30ac1ab4 Add ApplyEnvVars() to ComponentConfig interface
* cluster and restapi configs can also get values from environment variables
* other config components don't read any values from the environment

License: MIT
Signed-off-by: Robert Ignat <robert.ignat91@gmail.com>
2019-02-07 20:51:20 +02:00
Adrian Lanzafame
3b3f786d68
add opencensus tracing and metrics
This commit adds support for OpenCensus tracing
and metrics collection. This required support for
context.Context propogation throughout the cluster
codebase, and in particular, the ipfscluster component
interfaces.

The tracing propogates across RPC and HTTP boundaries.
The current default tracing backend is Jaeger.

The metrics currently exports the metrics exposed by
the opencensus http plugin as well as the pprof metrics
to a prometheus endpoint for scraping.
The current default metrics backend is Prometheus.

Metrics are currently exposed by default due to low
overhead, can be turned off if desired, whereas tracing
is off by default as it has a much higher performance
overhead, though the extent of the performance hit can be
adjusted with smaller sampling rates.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2019-02-04 18:53:21 +10:00
Hector Sanjuan
ced9f2f7f1 Update to go-ipfs-files 2.0.3
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-12-17 14:29:44 +01:00
Hector Sanjuan
15413edbc6 Unrewrite deps
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-12-17 14:16:38 +01:00
Łukasz Magiera
406d2b53f8 files2.0: Close subfiles in adder
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
2018-12-17 14:16:38 +01:00
Łukasz Magiera
9b0ff603b8 files2.0: use type helpers
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
2018-12-17 14:16:38 +01:00
Łukasz Magiera
c6944e4376 files2.0: update for no errors from NewMultiFileReader
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
2018-12-17 14:16:38 +01:00
Łukasz Magiera
00df384e1c go-ipfs-files 2.0 updates
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
2018-12-17 14:16:38 +01:00
Kishan Sagathiya
e21d84dbee Issue #453 Extract the IPFS Proxy from ipfshttp
Warning about old config style(proxy inside ipfs connector) getting
removed in future

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-11-27 10:36:00 +05:30
Kishan Sagathiya
f98ed71ff7 Issue #453 Extract the IPFS Proxy from ipfshttp
Fixed some messages and comments

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-11-11 21:09:13 +05:30
Kishan Sagathiya
41edb04486 Issue #453 Extract the IPFS Proxy from ipfshttp
Added test for ipfsproxy config

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-11-04 09:05:53 +05:30
Kishan Sagathiya
56cc3b88d7 Issue #453 Extract the IPFS Proxy from ipfshttp
Added test for ipfsproxy

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-11-04 08:57:09 +05:30
Kishan Sagathiya
411eea664c Merge branch 'master' of github.com:ipfs/ipfs-cluster into issue_453 2018-11-03 20:24:15 +05:30
Kishan Sagathiya
3a5ad6111a Issue #453 Extract the IPFS Proxy from ipfshttp
Changes as requieed
Rename IPFSProxy struct to Server

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-11-01 15:54:05 +05:30
Hector Sanjuan
91ddce4963 Fix the paths handled by the proxy
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-11-01 10:36:14 +01:00
Hector Sanjuan
85a7dc5114 Allocate memory for response rpc types
Might trigger a very weird race when pointing to nil

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-29 12:03:03 +01:00
Hector Sanjuan
b86f3cedb4 Files package has been extracted from cmdkit
Related: https://github.com/ipfs/go-ipfs-cmdkit/pull/31

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-24 01:28:05 +02:00
Hector Sanjuan
7d16108751 Start using libp2p/go-libp2p-gorpc
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-17 15:28:03 +02:00
Kishan Sagathiya
fdb573c96f Issue #453 Extract the IPFS Proxy from ipfshttp
Adding more missing pieces in config
Use the right package(not the inbuilt one)
Setup rpc client for proxy in the cluster
Add back SetClient and Shutdown into Connector as they are required to
implement Component interface
Add `ipfsproxy` as into list of logging identifier and add its default
log level

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-10-14 22:42:50 +05:30
Kishan Sagathiya
2e66720486 Issue #453 Extract the IPFS Proxy from ipfshttp
Go through `ipfshttp` repo and remove unnecessary proxy related code

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-10-14 18:28:15 +05:30
Kishan Sagathiya
d338f8de53 Issue #453 Extract the IPFS Proxy from ipfshttp
Added config
Change functions accordingly to add new `ipfsproxy` component

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-10-14 16:07:42 +05:30
Kishan Sagathiya
155a65cac3 Issue #453 Extract the IPFS Proxy from ipfshttp
Extract the IPFS Proxy from ipfshttp and make it an api module

The `ipfshttp` IPFSConnector implementation includes the so called IPFS
Proxy. An endpoint which offers an IPFS API, hijacking some interesting
requests and forwarding the rest to the ipfs daemon.

`ipfshttp` should contain an implementation of IPFSConnector whose only
task should be to talk to IPFS
A new module should be created, `api/ipfsproxy`, an API Component
implementation for Cluster. The whole proxy code should be moved here.

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-10-13 19:58:28 +05:30
Hector Sanjuan
f65349e9c8
Merge pull request #569 from ipfs/fix/add-output-format
Add endpoints output: improvements and compliance
2018-10-05 11:44:59 +02:00
Adrian Lanzafame
c0dbce61a8
ipfshttp: remove custom proxy and use stdlib reverse proxy
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-10-04 15:03:27 +10:00
Hector Sanjuan
b6306a6e61 Add endpoints output: improvements and compliance
This straigthens some mistakes with the outputs of the /add endpoints.

Currently, we had exactly the same output format which:

* was not exactly the ipfs API output format but was sort of similar
* made some weird concessions to be compatible (like having a string-type "size")
* was not aligned with Cluster API conventions (lowercase keys)

This corrects all this:

* The Cluster API /add output format now uses the right types and lowercase keys.
* `Hash` is now `Cid`, because the field carries a Cid.
* We copy error handling with request trailers from IPFS, and avoid carrying the
  errors in the output objects.
* The proxy now returns exactly the types as ipfs would
* We add the X-Chunked-Output: 1 header, which is custom and redundant, but
otherwise breaks js-ipfs-api integrations with the /add endpoint.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-04 00:57:36 +02:00
Hector Sanjuan
6e83ef5f24 Fix #548: Re-enable keep-alives. Set Connection: Close when adding.
This is a workaround to have clients behave properly with the /add
endpoint by asking them to close connections when done, effectively
disabling keep-alive for this.

This means we don't need to disable keep-alives fully on all servers,
since the rest of endpoints are not affected (they are not streaming
endpoints).

Reference https://github.com/ipfs/go-ipfs/issues/5168

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-09-29 02:07:57 +02:00
Adrian Lanzafame
31474f6490
update go-cid and go-libp2p
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-09-24 11:35:38 +10:00
Hector Sanjuan
2ffa3d80de Fix 466: Hijack repo/stat in the proxy and return aggregates from the cluster.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 20:43:27 +02:00
Hector Sanjuan
6c28d6bf3e Fix PinLsCid and pins with maxdepth 0 after migration
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 11:39:34 +02:00
Hector Sanjuan
63fa1ba213
Merge pull request #505 from ipfs/fix/495-proxy-add
Fix #495: Hijack proxy /add correctly. Disable keep-alives.
2018-08-20 10:29:20 +02:00
Hector Sanjuan
77df433c2d
Merge pull request #504 from ipfs/fix/pinls
IPFSPinLsCid should try to find pins in the direct pinset too.
2018-08-20 10:10:32 +02:00
Hector Sanjuan
be651da0d0 Fix: disable KeepAlives in server
See discussion in https://github.com/ipfs/go-ipfs/issues/5168

We cannot stream responses with keep-alives enabled.

I prefer this to not be a client feature, as otherwise users might end up
shooting themselves in the foot.

Note, the price is a corrupted request body which gets added
normally and gives wrong hashes!

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-18 02:48:36 +02:00
Hector Sanjuan
00a757fc0b Proxy/add: flush AddedOutput
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-18 02:48:35 +02:00
Hector Sanjuan
b2914c2516 Fix #495: Hijack proxy /add correctly.
No more hacks around /add. This uses the local adder when hijacking /add.
It supports the parameters and works pretty well with the ipfs CLI, showing
progress and everything.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-18 02:48:35 +02:00
Hector Sanjuan
ee8d7f7098 IPFSHttp: Add "size-only=true" flag to repo/stat calls
This will make them very fast.

Fixes #280

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-17 16:53:40 +02:00
Hector Sanjuan
2e7e71d13f IPFSPinLsCid should try to find pins in the direct pinset too.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-15 13:19:39 +02:00
Hector Sanjuan
50fc3c4e95 Address comments from review
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 13:22:47 +02:00
Hector Sanjuan
0151f5e312 Fixes for adding: set default timeouts to 0. Improve flags and param names.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-08 21:11:26 +02:00
Hector Sanjuan
6ffc8b8347 Update metrics regularly after a few blockputs
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
65dc17a78b testfixing
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
a96241941e WIP
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
68ec9316f7 Finish rebase of sharding branch. It builds!
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Wyatt Daviau
fabf191883 AddFile tests
tests cover local and sharded adds of files
ipfs mock and ipfs block put/get calls cleaned up

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
238f3726f3 Pin datastructure updated to support sharding
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
2d47e09d7c Resolving dep conflicts and small rebase inconsistancies
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
6b8bcdfbc3 Finishing feedback and adding new test
Also updating Makefile to fetch sharness more robustly

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
9f74f6f47d Addressing third round of comments
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
edb38b2830 fixing make check errors
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
fa74bc230d ipfs block put call now format aware
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
11e8e9d62c Addressing second round of comments
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
ceba32f871 Test for ipfscon:blockPut
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
40f8eeedb5 cluster-ctl add to ipfs
RPC call to put a block in ipfs
IPFSConnector method to implement the RPC call
cluster restapi reads from channel and puts
blocks into IPFS via RPC in ctl add handler

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Hector Sanjuan
78a25e38f8 Fix #462: Support DNS multiaddresses for node_multiaddress
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-06-12 10:09:03 +02:00
Hector Sanjuan
6d9ddfaf85
Merge pull request #427 from ipfs/fix/pin-context-timeout
ipfshttp: Use custom http clients for pin/unpin
2018-05-25 09:44:53 +02:00
Adrian Lanzafame
80e7baca91
ipfshttp: hande request timeouts to ifps daemon correctly
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-05-24 17:13:56 +10:00
Hector Sanjuan
5563297d40 Set refs as default pinning method, with 10 parallel pins
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-14 17:00:33 +02:00
Hector Sanjuan
aa1f74e69b Fix: some govet warnings
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-11 19:59:10 +02:00
Hector Sanjuan
9856bcdb94 Pintracker: remove timeouts
Pinning/unpinning timeouts are controlled by the ipfs connector component.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-02 15:24:26 +02:00
Adrian Lanzafame
f68c7f5354 ipfshttp: add pin/unpin specific timeouts
and get the tests passing and add Pin/UnpinQueued
tracker statuses back in.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-05-02 15:24:26 +02:00
Adrian Lanzafame
9e20e4e3b2 ipfsconn/ipfshttp: Pass ctx through from rpc_api
to the ipfscluster.IPFSConnector interface and then
to the implementation of that interface in ipfsconn/ipfshttp.
This allows calls from MapPinTracker to cancel requests made
to the local IPFS node.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-05-02 15:24:26 +02:00
Adrian Lanzafame
a2f59b26af ipfshttp/config: add ClientPostTimeout value
ipfshttp: cancel POST request when timeout reached

ipfshttp/config: fix config test

ipfshttp: use struct styling for multi-line func calls

ipfshttp/config: add general ClientTimeout

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-05-02 15:24:26 +02:00
Adrian Lanzafame
b50a05f898
ipfsconn/ipfshttp: go1.9 ServeMux doesn't redirect
query args correctly, requiring both a trailing slash and
non-trailing slash handle pattern to be defined for the
pin and unpin handlers.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-04-30 15:21:03 +10:00
Adrian Lanzafame
22ec210c25
ipfsconn/ipfshttp: handle cid args passed in url path correctly
The extractCid function was added to enable the extraction of
a cid argument from either the url path or query string.
This puts the proxy behaviour on par with the current IPFS API.
The function does rely on the fact that ipfs-cluster doesn't
intercept any command that has more than one subcommand.
If that changes, this function will have to be updated.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-04-30 15:21:03 +10:00
Hector Sanjuan
a9a58a50a0 Config: provide utility to parse duration arguments. Use it.
Should reduce complexity (codeclimate).

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-16 14:22:11 +01:00
Wyatt Daviau
1b7b9185e2 support for recursive pins
extension to api types
modifications to ipfsconnector

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-03-12 11:06:42 -04:00
Hector Sanjuan
e8f9fe5566 ipfshttp: Re-use pinning test for both methods
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-09 15:01:29 +01:00
Hector Sanjuan
6a858efbcc ipfshttp: refactor post(). Fix race condition. Support custom http.Client.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-09 15:01:29 +01:00
Hector Sanjuan
fb4812ec79 Feat #326: Adds "refs -r" pinning method support + multiple pin workers
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-09 15:01:29 +01:00
Hector Sanjuan
a180f1a5c5
Merge pull request #291 from ipfs/feat/connectivity-graph
Feat/connectivity graph
2018-01-26 12:42:18 +01:00
Hector Sanjuan
67e505964e Tests: make ipfshttp connector tests bind on random port
For jenkins

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-01-24 20:17:42 +01:00
Hector Sanjuan
ddb5da18c9 Tests: Bind testing clusters on random port
Jenkins likes this very much.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-01-24 20:16:55 +01:00
ZenGround0
4b26ccd144
Merge branch 'master' into feat/connectivity-graph 2018-01-23 08:34:43 -05:00
Wyatt Daviau
d2ef32f48f Testing and polishing connection graph
Added go tests
Refactored cluster connect graph to new file
Refactored dot file printing to new repo
Fixed code climate issues
Added sharness test

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-01-22 10:03:37 -05:00
Wyatt
e712c87570 First draft of ConnectGraph collection:
added ConnectGraph type and serialization
added cli command hitting cluster api
added cluster api client method + endpoint calling into rpc
added rpc calling into main cluster component
added clustercomponent's function to collect ConnectGraph
added functionality in ipfsconn to retrieve ipfs swarm peers
added dot file printing given ConnectGraphSerial

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-01-22 09:07:12 -05:00
Wyatt Daviau
a99b403273 Sharness hardening:
Better error messages in test-lib init functions and ipfshttp
Cleanup functions are now called after every test file

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-01-09 14:17:21 -05:00
Hector Sanjuan
0693ff429e fix spelling: Fix spelling errors
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2017-12-06 15:15:54 +01:00
Hector Sanjuan
7cd2d3d0e1 fix inef assignments: fix some unused assignments
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2017-12-06 15:15:52 +01:00
Hector Sanjuan
a656e45375 cluster: safeguard consensus not set when calling ID
SwarmConnect on the ipfs connector calls rpc Peers() which
requests IDs for every peer member. If that peer member
is booting, it might get the request after RPC is setup
but before consensus is initialized. In which case
a panic happens. Probability that this happens is small, but still.

Also increase the connect swarms delay to 30 seconds, which
should be a bit longer than the default wait_for_leader timeout,
otherwise we might connect swarms while there's not even a leader.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-11-15 16:38:21 +01:00
Hector Sanjuan
f14f2f4863 Fix #224: Better handling of progress updates when proxy-adding file
Up to now, we hardcoded progress to "false" in the proxy, regardless
of what the original request said. We now leave it as it is, and
just ignore any progress updates when processing the response.

Since the response is buffered and sent back all together, they are
still useless, but at least the clients (ipfs cli) won't show a 0%
progress bar when successfully adding a file.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-11-13 14:09:20 +01:00
Hector Sanjuan
00e871ddec Fix #202: Fix informers and allocators for 32-bit architectures 2017-10-26 16:01:41 +02:00
Hector Sanjuan
fb8fdb94c5 Issue #162: Improve Config.ToJSON() tests
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-10-20 10:42:41 +02:00
Hector Sanjuan
3cf4d90536 Issue #162: Add ipfshttp.Config tests
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-10-19 21:02:38 +02:00
Hector Sanjuan
8f06baa1bf Issue #162: Rework configuration format
The following commit reimplements ipfs-cluster configuration under
the following premises:

  * Each component is initialized with a configuration object
  defined by its module
  * Each component decides how the JSON representation of its
  configuration looks like
  * Each component parses and validates its own configuration
  * Each component exposes its own defaults
  * Component configurations are make the sections of a
  central JSON configuration file (which replaces the current
  JSON format)
  * Component configurations implement a common interface
  (config.ComponentConfig) with a set of common operations
  * The central configuration file is managed by a
  config.ConfigManager which:
    * Registers ComponentConfigs
    * Assigns the correspondent sections from the JSON file to each
    component and delegates the parsing
    * Delegates the JSON generation for each section
    * Can be notified when the configuration is updated and must be
    saved to disk

The new service.json would then look as follows:

```json
{
  "cluster": {
    "id": "QmTVW8NoRxC5wBhV7WtAYtRn7itipEESfozWN5KmXUQnk2",
    "private_key": "<...>",
    "secret": "00224102ae6aaf94f2606abf69a0e278251ecc1d64815b617ff19d6d2841f786",
    "peers": [],
    "bootstrap": [],
    "leave_on_shutdown": false,
    "listen_multiaddress": "/ip4/0.0.0.0/tcp/9096",
    "state_sync_interval": "1m0s",
    "ipfs_sync_interval": "2m10s",
    "replication_factor": -1,
    "monitor_ping_interval": "15s"
  },
  "consensus": {
    "raft": {
      "heartbeat_timeout": "1s",
      "election_timeout": "1s",
      "commit_timeout": "50ms",
      "max_append_entries": 64,
      "trailing_logs": 10240,
      "snapshot_interval": "2m0s",
      "snapshot_threshold": 8192,
      "leader_lease_timeout": "500ms"
    }
  },
  "api": {
    "restapi": {
      "listen_multiaddress": "/ip4/127.0.0.1/tcp/9094",
      "read_timeout": "30s",
      "read_header_timeout": "5s",
      "write_timeout": "1m0s",
      "idle_timeout": "2m0s"
    }
  },
  "ipfs_connector": {
    "ipfshttp": {
      "proxy_listen_multiaddress": "/ip4/127.0.0.1/tcp/9095",
      "node_multiaddress": "/ip4/127.0.0.1/tcp/5001",
      "connect_swarms_delay": "7s",
      "proxy_read_timeout": "10m0s",
      "proxy_read_header_timeout": "5s",
      "proxy_write_timeout": "10m0s",
      "proxy_idle_timeout": "1m0s"
    }
  },
  "monitor": {
    "monbasic": {
      "check_interval": "15s"
    }
  },
  "informer": {
    "disk": {
      "metric_ttl": "30s",
      "metric_type": "freespace"
    },
    "numpin": {
      "metric_ttl": "10s"
    }
  }
}
```

This new format aims to be easily extensible per component. As such,
it already surfaces quite a few new options which were hardcoded
before.

Additionally, since Go API have changed, some redundant methods have been
removed and small refactoring has happened to take advantage of the new
way.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-10-18 00:00:12 +02:00
Wyatt
598aaf2623 Undoing gx rewrites 2017-10-13 12:44:43 -04:00
Wyatt
c178afe371 Replace get requests with posts 2017-10-13 12:38:18 -04:00
dgrisham
a46ab3dda9 freespace metric partial impl 2017-08-03 11:24:19 -06:00
Hector Sanjuan
bb82c27b25 Fix #87: Implement ipfs-cluster-ctl pin ls <cid>
I have updated API endpoints to be /allocations rather than /pinlinst

It's more self-explanatory.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-04-06 21:12:16 +02:00
Hector Sanjuan
9b3449e3f8 Remove debugging leftovers in ipfshttp
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-30 15:38:41 +02:00
Hector Sanjuan
18034356fa Fix #57: Hijack /add requests and pin items in cluster after ipfs adds them.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-30 11:27:40 +02:00