cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
980 Commits 3 Branches 70 Tags 15 MiB
5a0d4ed9a2
Commit Graph

169 Commits

Author SHA1 Message Date
Hector Sanjuan
584f532bb1 Fix tests after rebase to master 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-14 15:18:16 +02:00
Adrian Lanzafame
c6eada9db5 uses new gorpc method to distinguish err type 
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
 2018-08-14 13:34:02 +02:00
Adrian Lanzafame
df2753dfc6 implements a stateless pintracker 
Also updates to the optracker to make retrieving information easier.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
 2018-08-14 13:33:36 +02:00
Hector Sanjuan
1b8967aa46 Merge branch 'master' into feat/sharding-v1 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-14 11:40:03 +02:00
Hector Sanjuan
2675bf8040
Merge pull request #489 from ipfs/feat/dht 
Feat: Enable DHT-based peer discovery and routing for cluster peers
 2018-08-14 11:34:35 +02:00
Hector Sanjuan
f4455d3733 Address comments 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-13 11:56:16 +02:00
Hector Sanjuan
c1363f3ad9 rest/client: clarify comment about overriding ssl configs 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-13 10:28:25 +02:00
Hector Sanjuan
522791df8d Feat/rest/client: auto-handling of libp2p endpoints 
This removes PeerAddr and uses APIAddr directly, figuring out if it is
a Peer multiaddress or not.

PeerAddr is actually kept for compatiblity.

It also fixes a bad panic when resolving returned 0 results

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-10 16:29:41 +02:00
Hector Sanjuan
f2dbd625a3 style 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-10 14:53:46 +02:00
Hector Sanjuan
bbf707d61e Fix allocations filter option 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-10 14:39:44 +02:00
Hector Sanjuan
7b826a5335 Dependency upgrades 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-10 14:39:34 +02:00
Hector Sanjuan
50fc3c4e95 Address comments from review 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-09 13:22:47 +02:00
Hector Sanjuan
c2312cbb05 Fix: close files. Close test files. Clean properly. Fix windows tests. 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-09 12:05:27 +02:00
Hector Sanjuan
26cbecca4c Fix tests 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-09 03:42:39 +02:00
Hector Sanjuan
0151f5e312 Fixes for adding: set default timeouts to 0. Improve flags and param names. 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-08 21:11:26 +02:00
Hector Sanjuan
87f4fcf958 Make the adder modules ipld.DAGService modules. 
This removes a bunch of the channel dance and block forwarding
by having the adder submodules be DAGServices themselves and take
Add() directly from the ipfsAdder.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
e2e84dfad1 Remove some commented code. Addressing a couple of comments. 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
585cc69eed Put add endpoint in /add 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
6ffc8b8347 Update metrics regularly after a few blockputs 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
3575f05753 Support adding Output (api.AddedOutput) 
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.

This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
c81f61eeea Make sure sync and recover operations receive all cids in a clusterDAG. 
Cleanup some code and fixmes.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
8f1a15b279 Move adder.Params to api.AddParams. Re-use in other modules 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
7c08177d6f Address some comments from lanzafame 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
aa5589d0d8 codeclimate 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
327a81b85a golint govets 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
623120fd50 Start cluster tests 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
65dc17a78b testfixing 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
db9070e0ca Fix api tests 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
2bfc8e7c2e Fix restapi tests 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
f8ff9883b0 wip 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
a96241941e WIP 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
68ec9316f7 Finish rebase of sharding branch. It builds! 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Wyatt Daviau
433928d3df addressing more feedback: 
validation of url params in restapi fileadd handler
use --only-hashes in sharness to simplify cid parse

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
1c91a99492 flag cleanup: 
silent flag removed for now
quiet flag renamed 'only-hashes' and implemented

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
2bc295ca9f address comments: 
rebase, cleanup and separate tests,
AddParams type replaces map of strings

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
339532b5d1 tests up to date with rebase 
sharness tests added

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
fabf191883 AddFile tests 
tests cover local and sharded adds of files
ipfs mock and ipfs block put/get calls cleaned up

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
fc35bf449c testing directory generated during test 
deterministically sample bytes into files of different sizes
reorganized hash storage for easy access from all subpacks
import tests pass with new directory structure

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
c4d1b34810 Tests to front end of adding pipeline 
api client test
api test
one sharness test
refactoring of testingData for access from other packages
always wrap files added by cluster
remove unused flag 'progress'
changes to support hidden flag

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
62eb45279b addressing feedback 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
f7c3dcce5b removing pointer across rpc 
instead add logic is a submodule imported in api and cluster comp

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
48479a0ca5 removing old code and fixing errors 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
1704295331 Refactor add & general cleanup 
addFile function is now a Cluster method accessed by RPC
residue from attempting to stream responses removed
ipfs-cluster-ctl ls bug fixed
problem with importer/add not printing resolved
new test now checks for this

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
32df2ce224 support for replication factors during sharding 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
dcafec24a5 New pin features added 
non-recursive cdag pins
priority pins for shards to allocated peer
cdag, meta and shard pin types used

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
238f3726f3 Pin datastructure updated to support sharding 
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
2d47e09d7c Resolving dep conflicts and small rebase inconsistancies 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
6b8bcdfbc3 Finishing feedback and adding new test 
Also updating Makefile to fetch sharness more robustly

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
9f74f6f47d Addressing third round of comments 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
163331d2c1 add with '--shard' now also provides user output 
make check errors addressed
fix importer tests

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00