cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,047 Commits 3 Branches 70 Tags 15 MiB
b98e549d92
Commit Graph

1047 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Hector Sanjuan
e3834cf8ab Force-enable wrap-in-directory 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-08 21:43:20 +02:00
Hector Sanjuan
1228d1cfc6 Golint and improved error messages 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-08 21:29:21 +02:00
Hector Sanjuan
0151f5e312 Fixes for adding: set default timeouts to 0. Improve flags and param names. 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-08 21:11:26 +02:00
Hector Sanjuan
0ed14a4b39 Adder: add context to add functions. Fix allocation everywhere. Helpers. 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-08 21:10:42 +02:00
Hector Sanjuan
7460be32c4 Try more 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:47:47 +02:00
Hector Sanjuan
fc2d7beacc Try syncing test folders 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:31:40 +02:00
Hector Sanjuan
5bb4d1a4bc misc 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:24:33 +02:00
Hector Sanjuan
87f4fcf958 Make the adder modules ipld.DAGService modules. 
This removes a bunch of the channel dance and block forwarding
by having the adder submodules be DAGServices themselves and take
Add() directly from the ipfsAdder.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
e2e84dfad1 Remove some commented code. Addressing a couple of comments. 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
585cc69eed Put add endpoint in /add 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
6ffc8b8347 Update metrics regularly after a few blockputs 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
3575f05753 Support adding Output (api.AddedOutput) 
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.

This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
7ef7a8d158 sync test files after creation 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
c81f61eeea Make sure sync and recover operations receive all cids in a clusterDAG. 
Cleanup some code and fixmes.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
8f1a15b279 Move adder.Params to api.AddParams. Re-use in other modules 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
7c08177d6f Address some comments from lanzafame 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
0893f69607 add fixme 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
16df90effc reduce complexity 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
aa5589d0d8 codeclimate 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
327a81b85a golint govets 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
6b9000fe7b Fix broken test 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
623120fd50 Start cluster tests 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
65dc17a78b testfixing 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
db9070e0ca Fix api tests 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
2bfc8e7c2e Fix restapi tests 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
ffd36afbb6 fix a few TODOs and FIXMEs 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
f8ff9883b0 wip 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
9d89bda2a6 wip 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
a96241941e WIP 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Hector Sanjuan
68ec9316f7 Finish rebase of sharding branch. It builds! 
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
 2018-08-07 20:12:05 +02:00
Wyatt Daviau
463840bab3 more pin tests 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
ecec7c24b0 new pin-test: sharded pins sharing a shard in cdag 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
433928d3df addressing more feedback: 
validation of url params in restapi fileadd handler
use --only-hashes in sharness to simplify cid parse

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
1c91a99492 flag cleanup: 
silent flag removed for now
quiet flag renamed 'only-hashes' and implemented

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
1adc409aa7 refactor import_test 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
82facd3629 fix for pinning same cid sharded and unsharded error 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
2bc295ca9f address comments: 
rebase, cleanup and separate tests,
AddParams type replaces map of strings

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
339532b5d1 tests up to date with rebase 
sharness tests added

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
fabf191883 AddFile tests 
tests cover local and sharded adds of files
ipfs mock and ipfs block put/get calls cleaned up

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
fc35bf449c testing directory generated during test 
deterministically sample bytes into files of different sizes
reorganized hash storage for easy access from all subpacks
import tests pass with new directory structure

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
c4d1b34810 Tests to front end of adding pipeline 
api client test
api test
one sharness test
refactoring of testingData for access from other packages
always wrap files added by cluster
remove unused flag 'progress'
changes to support hidden flag

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
62eb45279b addressing feedback 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
f7c3dcce5b removing pointer across rpc 
instead add logic is a submodule imported in api and cluster comp

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:24 +02:00
Wyatt Daviau
48479a0ca5 removing old code and fixing errors 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
1704295331 Refactor add & general cleanup 
addFile function is now a Cluster method accessed by RPC
residue from attempting to stream responses removed
ipfs-cluster-ctl ls bug fixed
problem with importer/add not printing resolved
new test now checks for this

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
1c8c4604c1 fix sharder tests 
fixed obo error
fix parent cid to be a set per new pin format
finalize now works if there is not data to flush

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
32df2ce224 support for replication factors during sharding 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
dcafec24a5 New pin features added 
non-recursive cdag pins
priority pins for shards to allocated peer
cdag, meta and shard pin types used

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
238f3726f3 Pin datastructure updated to support sharding 
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00
Wyatt Daviau
2d47e09d7c Resolving dep conflicts and small rebase inconsistancies 
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
 2018-08-07 20:11:23 +02:00