Commit Graph

135 Commits

Author SHA1 Message Date
Adrian Lanzafame
a34154ff25
gx update go-dot
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-09-13 13:29:34 +10:00
Hector Sanjuan
2f417eb632
Release 0.5.0
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-24 15:03:22 +02:00
Hector Sanjuan
0c5de3a849 Add/improve godoc descriptions for some modules.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-23 14:14:06 +02:00
Hector Sanjuan
5c4ebe7554 Small adjustment to "pin ls" human output in ctl.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-22 13:06:10 +02:00
Hector Sanjuan
fa2ba13ae2
Release 0.5.0-rc1
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-21 20:50:59 +02:00
Hector Sanjuan
05102c13a0 AddParams: support "cid-version" and "hash" function.
Additionally, remove dependency to go-ipfs after extraction from
"go-mfs".

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-21 19:38:41 +02:00
Hector Sanjuan
2061029313 Fix hanging add in sharness.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 12:08:49 +02:00
Hector Sanjuan
b6182e0621 rest/client: Implement .Add(paths), which adds from local files and web files
.Add(paths) will interpret http* paths as WebFiles. These are read performing
a GET request to the location. Otherwise, the path is interpreted as a local
disk file/folder, and read from disk. ipfs-cluster-ctl has been updated
accordingly.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 11:07:23 +02:00
Hector Sanjuan
1b8967aa46 Merge branch 'master' into feat/sharding-v1
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-14 11:40:03 +02:00
Hector Sanjuan
522791df8d Feat/rest/client: auto-handling of libp2p endpoints
This removes PeerAddr and uses APIAddr directly, figuring out if it is
a Peer multiaddress or not.

PeerAddr is actually kept for compatiblity.

It also fixes a bad panic when resolving returned 0 results

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 16:29:41 +02:00
Hector Sanjuan
f2dbd625a3 style
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 14:53:46 +02:00
Hector Sanjuan
bbf707d61e Fix allocations filter option
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 14:39:44 +02:00
Hector Sanjuan
78b8f47c14 Fix: Wrap-in-directory
Usually we had wrap-in-directory enabled by default because otherwise
we had an error when adding single, non-directory files.

wrap-in-directory happens automatically when adding more than one file
so that was no problem. Thigns also worked when adding a folder and Wrap was
disabled. The only case was adding a single with wrap disabled (a default option).

This patches the ipfsadd/add.go file to remember the last added file so that
we can use it's Cid as the resulting root of the adding process without
having to fetch it from our dummy dagservice.

We have to pass this CID to our Finalize() functions, because it turns out that
in this case (single file without wrap-in-directory), the last block added to the DAG
is not the IPFS root (the ipfsadd/Adder adds the mfs root folder last always).

This was the case when wrap-in-directory was enabled by default.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 01:16:30 +02:00
Hector Sanjuan
e3834cf8ab Force-enable wrap-in-directory
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-08 21:43:20 +02:00
Hector Sanjuan
0151f5e312 Fixes for adding: set default timeouts to 0. Improve flags and param names.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-08 21:11:26 +02:00
Hector Sanjuan
6ffc8b8347 Update metrics regularly after a few blockputs
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
3575f05753 Support adding Output (api.AddedOutput)
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.

This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
8f1a15b279 Move adder.Params to api.AddParams. Re-use in other modules
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
623120fd50 Start cluster tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
65dc17a78b testfixing
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
a96241941e WIP
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Wyatt Daviau
1c91a99492 flag cleanup:
silent flag removed for now
quiet flag renamed 'only-hashes' and implemented

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
c4d1b34810 Tests to front end of adding pipeline
api client test
api test
one sharness test
refactoring of testingData for access from other packages
always wrap files added by cluster
remove unused flag 'progress'
changes to support hidden flag

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
1704295331 Refactor add & general cleanup
addFile function is now a Cluster method accessed by RPC
residue from attempting to stream responses removed
ipfs-cluster-ctl ls bug fixed
problem with importer/add not printing resolved
new test now checks for this

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
32df2ce224 support for replication factors during sharding
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
238f3726f3 Pin datastructure updated to support sharding
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
6b8bcdfbc3 Finishing feedback and adding new test
Also updating Makefile to fetch sharness more robustly

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
9f74f6f47d Addressing third round of comments
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
ae38d09208 Beginning add UX improvements--
1. Refactored importer endpoint, including writing cluster-specific
file adder, to get print info from importer
2. Refactored importer consumption to select equally from
channels of different output signals and manage context
timeouts correctly (only in local add here, sharding to follow)
3. Added output streaming and an error/termination handling protocol
4. Discovered that naive eager response streaming cuts off
reads from request data stream and breaks behavior, for
now all responses come after file ingestion.
5. Added ipfs add style flags (trickle, rawleaves etc.) and
refactored importer endpoint to take in these parameters
to provide identicle behavior to ipfs

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
77a61890ff Sharding rough draft:
sharding passes manual tests on single node cluster,
adding the shards of a directory and pinning the
clusterDAG to cluster/ipfs state

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
5725e34e3d polishing up print on restapi
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
9e3e42b19d client streams files
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
9521885b63 ipfs-cluster-ctl add
parse file path arguments to cmdkit.File objects
no support for directories, symlinks or recursive add

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:13 +02:00
ia
3c1bd32e8b all: gofmt
Run standard gofmt command on project root.

License: MIT
Signed-off-by: Isaac Henri Ardis <isaac.ardis@gmail.com>
2018-06-14 11:55:03 +02:00
Hector Sanjuan
0889302c8d
Release 0.4.0
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-30 13:26:10 +02:00
Hector Sanjuan
5b114383e6
Release 0.4.0-rc2
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-28 12:45:10 +02:00
Michael Muré
047fa48ec9
Makefile: add a source target to properly trigger a build on source change
Fix #425

License: MIT
Signed-off-by: Michael Muré <batolettre@gmail.com>
2018-05-18 01:46:19 +02:00
Hector Sanjuan
16c602e5c7
Release 0.4.0-rc1
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-14 11:56:38 +02:00
Hector Sanjuan
a0a0898719
Merge pull request #393 from ipfs/docs/move-to-website
Docs/move to website
2018-05-01 11:14:17 +02:00
Hector Sanjuan
1d20f3de36 ipfs-cluster-ctl: do not provide "peers add"
This uses the PeerAdd endpoint which should NOT be used as the current
workflow states that the way to adding peers is bootstrapping.

Adding peers manually with this endpoint leads to split-head states very
easily. The fact that this operation is visible in ipfs-cluster-ctl
is only leading the users to bad places.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-27 10:27:52 +02:00
Hector Sanjuan
b08b3aba64 Docs: Move to website.
Updated READMEs, removed docs and point everything to website documentation.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-27 09:05:11 +02:00
Hector Sanjuan
ed55daf2c6
Release 0.3.5
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-29 13:43:36 +02:00
Hector Sanjuan
41433298f5 Fix: --wait flag. Address comments
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-29 11:25:15 +02:00
Hector Sanjuan
018e5dbaea Fix: --wait flag in ipfs-cluster-ctl
The --wait flag was being completely ignored unless --no-status was passed
too, which makes no sense because then it would print the wait status.

This waits when --wait and prints the status when --no-status is not passed.

If we have been waiting, the status comes from that. Otherwise we request it.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-29 11:23:40 +02:00
Adrian Lanzafame
4d48504779 rest/client: Add WaitFor utility function
address feedback and add WaitForPinnedStatus

address feedback and rework WaitFor

implement StatusFilter

address feedback and rework StatusFilter

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-03-27 14:56:49 +10:00
Hector Sanjuan
72876465d6
Release 0.3.5-rc1
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-26 21:50:18 +02:00
Hector Sanjuan
09f4c9fce3 rest/libp2p-http: address lanzafame's review
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-20 19:35:42 +01:00
Hector Sanjuan
23d2db15e7 ipfs-cluster-ctl: fix sharness
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-16 19:37:52 +01:00
Hector Sanjuan
73581b0471 ipfs-cluster-ctl: Allow libp2p multiaddress as --host. Use libp2p in that case.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-15 17:15:23 +01:00
Adrian Lanzafame
7715eff0f6 rest/client: ipfs-cluster-ctl: double default timeout
Change the rest/client's and ipfs-cluster-ctl's default
timeout from 60 to 120.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-03-10 11:20:56 +10:00