Commit Graph

87 Commits

Author SHA1 Message Date
Hector Sanjuan
b93b2fedd4 Fix interpreting Host parameter correctly.
We should deprecate passing in Host/Port in the config,
but in the meantime, it hardcoded /dns4/, meaning that if
someone placed an ipv6 address in there things would break badly
and weirdly.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-12-19 21:19:47 +01:00
Hector Sanjuan
7c26f1ecb8 Fix HTTPs with DNS multiaddresses
Before we resolved all /dns*/ multiaddresses before we used them.

When using HTTPs, the Go HTTP Client only sees the resolved IP address
and it is unable to negotiate TLS with a cerficate because the request
is not going to the hostname the certificate is signed for, but to
the IP. This leverages a recent feature in go-multiaddr-net
and uses directly the user-provided hostname.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-12-19 21:15:37 +01:00
Hector Sanjuan
11cc0485c7 API/client: Do not clean folders mid-test.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-12-17 14:55:34 +01:00
Hector Sanjuan
ced9f2f7f1 Update to go-ipfs-files 2.0.3
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-12-17 14:29:44 +01:00
Hector Sanjuan
15413edbc6 Unrewrite deps
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-12-17 14:16:38 +01:00
Łukasz Magiera
406d2b53f8 files2.0: Close subfiles in adder
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
2018-12-17 14:16:38 +01:00
Łukasz Magiera
c6944e4376 files2.0: update for no errors from NewMultiFileReader
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
2018-12-17 14:16:38 +01:00
Łukasz Magiera
00df384e1c go-ipfs-files 2.0 updates
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
2018-12-17 14:16:38 +01:00
Hector Sanjuan
1d924bfc49 Remove shard testing folders
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-11-01 19:15:08 +01:00
Hector Sanjuan
19b1124999 Make metrics human
Issue #572 exposes metrics but they carry the peer ID in binary.

This was ok with our internal codecs but it doesn't seem to work
very well with json, and makes the output format unusable.

This makes the Metric.Peer field a string.

Additinoally, fixes calling the command without arguments and displaying
the date in the right format.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-26 14:11:30 +02:00
Hector Sanjuan
9330ac82e2 Fix tests with latest libp2p
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-25 14:55:01 +02:00
Hector Sanjuan
b86f3cedb4 Files package has been extracted from cmdkit
Related: https://github.com/ipfs/go-ipfs-cmdkit/pull/31

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-24 01:28:05 +02:00
Hector Sanjuan
79fe5919d2 Fix: escape filter parameter query
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-22 23:45:33 +02:00
Hector Sanjuan
48c89fbe4d
Merge pull request #572 from kishansagathiya/issue_449
Issue #449 API endpoint for Peer Monitor metrics
2018-10-22 23:32:50 +02:00
Kishan Sagathiya
ec4588a5ce Issue #449 API endpoint for Peer Monitor metrics
Rename method PeerMonitorLatestMetrics to Metrics
Addressing first round of comment as in
https://github.com/ipfs/ipfs-cluster/pull/572#pullrequestreview-165367171

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-10-21 13:36:20 +05:30
Kishan Sagathiya
3ac7d6c9cc Issue #449 API endpoint for Peer Monitor metrics
Support the new endpoint for later metrics in `rest/api/client`

Support the new method created in `rest/api/client` in
ipfs-cluster-ctl. i.e. `ipfs-cluster-ctl health metrics <name>` would
show the peers and the last list of metrics logged for each as returned
by the Peer Monitor, in a friendly way.

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-10-21 12:49:56 +05:30
Hector Sanjuan
4e3553fb1f Fix: escape the add query parameters correctly in the client
When adding something that has a name with spaces, things fail
very badly.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-19 20:19:52 +02:00
Hector Sanjuan
7d16108751 Start using libp2p/go-libp2p-gorpc
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-17 15:28:03 +02:00
Hector Sanjuan
b6306a6e61 Add endpoints output: improvements and compliance
This straigthens some mistakes with the outputs of the /add endpoints.

Currently, we had exactly the same output format which:

* was not exactly the ipfs API output format but was sort of similar
* made some weird concessions to be compatible (like having a string-type "size")
* was not aligned with Cluster API conventions (lowercase keys)

This corrects all this:

* The Cluster API /add output format now uses the right types and lowercase keys.
* `Hash` is now `Cid`, because the field carries a Cid.
* We copy error handling with request trailers from IPFS, and avoid carrying the
  errors in the output objects.
* The proxy now returns exactly the types as ipfs would
* We add the X-Chunked-Output: 1 header, which is custom and redundant, but
otherwise breaks js-ipfs-api integrations with the /add endpoint.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-04 00:57:36 +02:00
Hector Sanjuan
dc0b211324 Docs: Fix comment for WaitFor function
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-03 11:04:24 +02:00
Hector Sanjuan
b5631aeb8a Fix IPFS() method missing in Client interface
Seems we forgot about this one.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-01 20:09:26 +02:00
Adrian Lanzafame
000f2b3f6e api/rest: change *cid.Cid to cid.Cid in client iface
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-10-01 12:51:57 +02:00
Joseph Lukasik
b98e549d92 Add Client interface, implemented by defaultClient
License: MIT
Signed-off-by: Joseph Lukasik <joseph@jgl.me>
2018-09-27 16:51:34 -07:00
Adrian Lanzafame
31474f6490
update go-cid and go-libp2p
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-09-24 11:35:38 +10:00
Hector Sanjuan
0c5de3a849 Add/improve godoc descriptions for some modules.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-23 14:14:06 +02:00
Hector Sanjuan
9c3826fdc5 Use cmdkit WebFile.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-21 02:12:36 +02:00
Hector Sanjuan
2061029313 Fix hanging add in sharness.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 12:08:49 +02:00
Hector Sanjuan
b6182e0621 rest/client: Implement .Add(paths), which adds from local files and web files
.Add(paths) will interpret http* paths as WebFiles. These are read performing
a GET request to the location. Otherwise, the path is interpreted as a local
disk file/folder, and read from disk. ipfs-cluster-ctl has been updated
accordingly.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 11:07:23 +02:00
Hector Sanjuan
1b8967aa46 Merge branch 'master' into feat/sharding-v1
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-14 11:40:03 +02:00
Hector Sanjuan
2675bf8040
Merge pull request #489 from ipfs/feat/dht
Feat: Enable DHT-based peer discovery and routing for cluster peers
2018-08-14 11:34:35 +02:00
Hector Sanjuan
c1363f3ad9 rest/client: clarify comment about overriding ssl configs
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-13 10:28:25 +02:00
Hector Sanjuan
522791df8d Feat/rest/client: auto-handling of libp2p endpoints
This removes PeerAddr and uses APIAddr directly, figuring out if it is
a Peer multiaddress or not.

PeerAddr is actually kept for compatiblity.

It also fixes a bad panic when resolving returned 0 results

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 16:29:41 +02:00
Hector Sanjuan
bbf707d61e Fix allocations filter option
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 14:39:44 +02:00
Hector Sanjuan
50fc3c4e95 Address comments from review
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 13:22:47 +02:00
Hector Sanjuan
c2312cbb05 Fix: close files. Close test files. Clean properly. Fix windows tests.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 12:05:27 +02:00
Hector Sanjuan
26cbecca4c Fix tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 03:42:39 +02:00
Hector Sanjuan
0151f5e312 Fixes for adding: set default timeouts to 0. Improve flags and param names.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-08 21:11:26 +02:00
Hector Sanjuan
585cc69eed Put add endpoint in /add
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
6ffc8b8347 Update metrics regularly after a few blockputs
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
3575f05753 Support adding Output (api.AddedOutput)
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.

This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
8f1a15b279 Move adder.Params to api.AddParams. Re-use in other modules
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
aa5589d0d8 codeclimate
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
65dc17a78b testfixing
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Wyatt Daviau
1c91a99492 flag cleanup:
silent flag removed for now
quiet flag renamed 'only-hashes' and implemented

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
2bc295ca9f address comments:
rebase, cleanup and separate tests,
AddParams type replaces map of strings

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
339532b5d1 tests up to date with rebase
sharness tests added

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
c4d1b34810 Tests to front end of adding pipeline
api client test
api test
one sharness test
refactoring of testingData for access from other packages
always wrap files added by cluster
remove unused flag 'progress'
changes to support hidden flag

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
1704295331 Refactor add & general cleanup
addFile function is now a Cluster method accessed by RPC
residue from attempting to stream responses removed
ipfs-cluster-ctl ls bug fixed
problem with importer/add not printing resolved
new test now checks for this

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
32df2ce224 support for replication factors during sharding
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
238f3726f3 Pin datastructure updated to support sharding
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00