Commit Graph

78 Commits

Author SHA1 Message Date
Hector Sanjuan
d63a5e2667 Fix race on ApplyTo
The FSM tries to decode an operation on top of the
*LogOp. We might still be using the *LogOp.Cid.Allocations
slice. We need to make a deep of *LogOp.Cid before
returning from ApplyTo.

This one was tricky...

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-29 12:03:47 +01:00
Hector Sanjuan
19b1124999 Make metrics human
Issue #572 exposes metrics but they carry the peer ID in binary.

This was ok with our internal codecs but it doesn't seem to work
very well with json, and makes the output format unusable.

This makes the Metric.Peer field a string.

Additinoally, fixes calling the command without arguments and displaying
the date in the right format.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-10-26 14:11:30 +02:00
Kishan Sagathiya
2cd4420ee8 Issue #446 Adding peername to PinInfo
Fixing errors and better code placement

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-09-24 23:20:37 +05:30
Kishan Sagathiya
ef85ba8780 Issue #446 Adding peername to PinInfo
This commit adds peername to PinInfo and GlobalPinInfo so that we have
a nicer and more meaningfull output for `ipfs-cluster-ctl` queries like
`status`, `sync` and `recover`

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-09-24 23:04:16 +05:30
Kishan Sagathiya
ff48fb319a Issue #446 Adding peername to PinInfo
This commit adds peername to PinInfo and GlobalPinInfo so that we have
a nicer and more meaningfull output for queries like
`ipfs-cluster-ctl status`

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>
2018-09-24 23:04:16 +05:30
Adrian Lanzafame
31474f6490
update go-cid and go-libp2p
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-09-24 11:35:38 +10:00
Hector Sanjuan
2ffa3d80de Fix 466: Hijack repo/stat in the proxy and return aggregates from the cluster.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-20 20:43:27 +02:00
Hector Sanjuan
584f532bb1 Fix tests after rebase to master
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-14 15:18:16 +02:00
Adrian Lanzafame
c6eada9db5 uses new gorpc method to distinguish err type
License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-08-14 13:34:02 +02:00
Adrian Lanzafame
df2753dfc6 implements a stateless pintracker
Also updates to the optracker to make retrieving information easier.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-08-14 13:33:36 +02:00
Hector Sanjuan
bbf707d61e Fix allocations filter option
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-10 14:39:44 +02:00
Hector Sanjuan
50fc3c4e95 Address comments from review
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-09 13:22:47 +02:00
Hector Sanjuan
e2e84dfad1 Remove some commented code. Addressing a couple of comments.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
6ffc8b8347 Update metrics regularly after a few blockputs
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
3575f05753 Support adding Output (api.AddedOutput)
This was a long FIXME/TODO. Handling adding output and
reporting to the client of the progress of the adding process.

This attempts to do it. It is not sure that it works correctly
(response body being written while the multipart request is still being read)

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
c81f61eeea Make sure sync and recover operations receive all cids in a clusterDAG.
Cleanup some code and fixmes.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
7c08177d6f Address some comments from lanzafame
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
327a81b85a golint govets
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
623120fd50 Start cluster tests
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
f8ff9883b0 wip
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
a96241941e WIP
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Hector Sanjuan
68ec9316f7 Finish rebase of sharding branch. It builds!
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-08-07 20:12:05 +02:00
Wyatt Daviau
1c91a99492 flag cleanup:
silent flag removed for now
quiet flag renamed 'only-hashes' and implemented

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
2bc295ca9f address comments:
rebase, cleanup and separate tests,
AddParams type replaces map of strings

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
62eb45279b addressing feedback
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:24 +02:00
Wyatt Daviau
1704295331 Refactor add & general cleanup
addFile function is now a Cluster method accessed by RPC
residue from attempting to stream responses removed
ipfs-cluster-ctl ls bug fixed
problem with importer/add not printing resolved
new test now checks for this

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
dcafec24a5 New pin features added
non-recursive cdag pins
priority pins for shards to allocated peer
cdag, meta and shard pin types used

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
238f3726f3 Pin datastructure updated to support sharding
4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
2d47e09d7c Resolving dep conflicts and small rebase inconsistancies
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
9f74f6f47d Addressing third round of comments
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
ae38d09208 Beginning add UX improvements--
1. Refactored importer endpoint, including writing cluster-specific
file adder, to get print info from importer
2. Refactored importer consumption to select equally from
channels of different output signals and manage context
timeouts correctly (only in local add here, sharding to follow)
3. Added output streaming and an error/termination handling protocol
4. Discovered that naive eager response streaming cuts off
reads from request data stream and breaks behavior, for
now all responses come after file ingestion.
5. Added ipfs add style flags (trickle, rawleaves etc.) and
refactored importer endpoint to take in these parameters
to provide identicle behavior to ipfs

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
edb38b2830 fixing make check errors
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
92dff3462d Initial unit tests
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
504d08d06c Addressing first round of comments
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
fa74bc230d ipfs block put call now format aware
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
ef3b149419 sharding component rough draft:
A sharder config provides a default shard size
getAllocations calls newly exposed rpcs
ipld cbor cluster dag node construction
logic to handle Flushing final shard
logic to initialize and finalize shard sessions

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Wyatt Daviau
4a5a613d94 rpc call and config added
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-08-07 20:11:23 +02:00
Hector Sanjuan
218ee0260b Fix: API constants should explicit their type
Otherwise, they are int. This forces API users to do manual variable
type declaration when initializing something to one of these constants

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-06-01 17:27:21 +02:00
Adrian Lanzafame
c89508035a Maptracker: extract optracker and make improvements
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-28 11:59:26 +02:00
Laevos
a3c77dafb8 fixed formatting issues
License: MIT
Signed-off-by: Lilith McMullen <iggnsthe@live.com>
2018-05-25 20:38:47 -05:00
Laevos
e62c4171ef fix #428: Use a regexp to match for indirect status in IPFSPinStatusFromString()
Since indirect pins are of the form `indirect through <cid>`, let's use
a regexp to match them instead of an equality operator.

License: MIT
Signed-off-by: Lilith McMullen <iggnsthe@live.com>
2018-05-25 20:08:45 -05:00
Hector Sanjuan
a9d6fe3479 Types: rename metric.SetTTLDuration to metric.SetTTL
GetTTL returns duration. SetTTL should take duration too, not seconds.
This removes the original SetTTL method which used seconds.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-05-07 14:26:06 +02:00
Adrian Lanzafame
f68c7f5354 ipfshttp: add pin/unpin specific timeouts
and get the tests passing and add Pin/UnpinQueued
tracker statuses back in.

License: MIT
Signed-off-by: Adrian Lanzafame <adrianlanzafame92@gmail.com>
2018-05-02 15:24:26 +02:00
Hector Sanjuan
0069c0062f Fix metric expire type. Do not discard metrics in Allocate().
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-04-05 17:57:24 +02:00
Hector Sanjuan
403a4cf6e0 api/types: do not print some errors
The rest/client does things like "return id.ToID(), err" which is nice.
But when there's an error the ID has Cid set to nil etc. This is now
causing error messages on the console which are very misleading because
they do not correspond to the actual error of the request.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-28 19:16:49 +02:00
Hector Sanjuan
5cd048a65d api: move some util functions to util.go (codeclimate file lines)
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-16 13:43:14 +01:00
Hector Sanjuan
a73d7e6f7e Relocate multiaddrJoin and multiaddrSplit to api/types.h
So they can serve as multi-module helpers without having circular deps.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
2018-03-16 13:37:32 +01:00
Wyatt Daviau
1b7b9185e2 support for recursive pins
extension to api types
modifications to ipfsconnector

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-03-12 11:06:42 -04:00
ZenGround0
4b26ccd144
Merge branch 'master' into feat/connectivity-graph 2018-01-23 08:34:43 -05:00
Wyatt Daviau
d2ef32f48f Testing and polishing connection graph
Added go tests
Refactored cluster connect graph to new file
Refactored dot file printing to new repo
Fixed code climate issues
Added sharness test

License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
2018-01-22 10:03:37 -05:00