Commit Graph

174 Commits

Author SHA1 Message Date
Hector Sanjuan
2061029313 Fix hanging add in sharness.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 12:08:49 +02:00
Hector Sanjuan
b6182e0621 rest/client: Implement .Add(paths), which adds from local files and web files
.Add(paths) will interpret http* paths as WebFiles. These are read performing
a GET request to the location. Otherwise, the path is interpreted as a local
disk file/folder, and read from disk. ipfs-cluster-ctl has been updated
accordingly.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 11:07:23 +02:00
Hector Sanjuan
be651da0d0 Fix: disable KeepAlives in server
See discussion in https://github.com/ipfs/go-ipfs/issues/5168

We cannot stream responses with keep-alives enabled.

I prefer this to not be a client feature, as otherwise users might end up
shooting themselves in the foot.

Note, the price is a corrupted request body which gets added
normally and gives wrong hashes!

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-18 02:48:36 +02:00
Hector Sanjuan
00a757fc0b Proxy/add: flush AddedOutput
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-18 02:48:35 +02:00
Hector Sanjuan
b2914c2516 Fix #495: Hijack proxy /add correctly.
No more hacks around /add. This uses the local adder when hijacking /add.
It supports the parameters and works pretty well with the ipfs CLI, showing
progress and everything.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-18 02:48:35 +02:00
Hector Sanjuan
584f532bb1 Fix tests after rebase to master
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-14 15:18:16 +02:00
Adrian Lanzafame
c6eada9db5 uses new gorpc method to distinguish err type
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-08-14 13:34:02 +02:00
Adrian Lanzafame
df2753dfc6 implements a stateless pintracker
Also updates to the optracker to make retrieving information easier.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-08-14 13:33:36 +02:00
Hector Sanjuan
1b8967aa46 Merge branch 'master' into feat/sharding-v1
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-14 11:40:03 +02:00
Hector Sanjuan
2675bf8040
Merge pull request #489 from ipfs/feat/dht
Feat: Enable DHT-based peer discovery and routing for cluster peers
2018-08-14 11:34:35 +02:00
Hector Sanjuan
f4455d3733 Address comments
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-13 11:56:16 +02:00
Hector Sanjuan
c1363f3ad9 rest/client: clarify comment about overriding ssl configs
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-13 10:28:25 +02:00
Hector Sanjuan
522791df8d Feat/rest/client: auto-handling of libp2p endpoints
This removes PeerAddr and uses APIAddr directly, figuring out if it is
a Peer multiaddress or not.

PeerAddr is actually kept for compatiblity.

It also fixes a bad panic when resolving returned 0 results

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 16:29:41 +02:00
Hector Sanjuan
f2dbd625a3 style
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 14:53:46 +02:00
Hector Sanjuan
bbf707d61e Fix allocations filter option
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 14:39:44 +02:00
Hector Sanjuan
7b826a5335 Dependency upgrades
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 14:39:34 +02:00
Hector Sanjuan
50fc3c4e95 Address comments from review
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 13:22:47 +02:00
Hector Sanjuan
c2312cbb05 Fix: close files. Close test files. Clean properly. Fix windows tests.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 12:05:27 +02:00
Hector Sanjuan
26cbecca4c Fix tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 03:42:39 +02:00
Hector Sanjuan
0151f5e312 Fixes for adding: set default timeouts to 0. Improve flags and param names.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-08 21:11:26 +02:00
Hector Sanjuan
87f4fcf958 Make the adder modules ipld.DAGService modules.
This removes a bunch of the channel dance and block forwarding
by having the adder submodules be DAGServices themselves and take
Add() directly from the ipfsAdder.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
e2e84dfad1 Remove some commented code. Addressing a couple of comments.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
585cc69eed Put add endpoint in /add
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
6ffc8b8347 Update metrics regularly after a few blockputs
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
3575f05753 Support adding Output (api.AddedOutput)
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.

This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
c81f61eeea Make sure sync and recover operations receive all cids in a clusterDAG.
Cleanup some code and fixmes.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
8f1a15b279 Move adder.Params to api.AddParams. Re-use in other modules
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
7c08177d6f Address some comments from lanzafame
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
aa5589d0d8 codeclimate
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
327a81b85a golint govets
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
623120fd50 Start cluster tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
65dc17a78b testfixing
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
db9070e0ca Fix api tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
2bfc8e7c2e Fix restapi tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
f8ff9883b0 wip
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
a96241941e WIP
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
68ec9316f7 Finish rebase of sharding branch. It builds!
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Wyatt Daviau
433928d3df addressing more feedback:
validation of url params in restapi fileadd handler
use --only-hashes in sharness to simplify cid parse

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
1c91a99492 flag cleanup:
silent flag removed for now
quiet flag renamed 'only-hashes' and implemented

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
2bc295ca9f address comments:
rebase, cleanup and separate tests,
AddParams type replaces map of strings

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
339532b5d1 tests up to date with rebase
sharness tests added

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
fabf191883 AddFile tests
tests cover local and sharded adds of files
ipfs mock and ipfs block put/get calls cleaned up

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
fc35bf449c testing directory generated during test
deterministically sample bytes into files of different sizes
reorganized hash storage for easy access from all subpacks
import tests pass with new directory structure

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
c4d1b34810 Tests to front end of adding pipeline
api client test
api test
one sharness test
refactoring of testingData for access from other packages
always wrap files added by cluster
remove unused flag 'progress'
changes to support hidden flag

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
62eb45279b addressing feedback
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
f7c3dcce5b removing pointer across rpc
instead add logic is a submodule imported in api and cluster comp

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
48479a0ca5 removing old code and fixing errors
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
1704295331 Refactor add & general cleanup
addFile function is now a Cluster method accessed by RPC
residue from attempting to stream responses removed
ipfs-cluster-ctl ls bug fixed
problem with importer/add not printing resolved
new test now checks for this

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
32df2ce224 support for replication factors during sharding
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
dcafec24a5 New pin features added
non-recursive cdag pins
priority pins for shards to allocated peer
cdag, meta and shard pin types used

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00