Commit Graph

411 Commits

Author SHA1 Message Date
Hector Sanjuan
717ed85823 gofmt -s fixes 2020-04-14 23:44:18 +02:00
Hector Sanjuan
97401443f3 Update to newest golang protobuf 2020-04-14 20:41:43 +02:00
Hector Sanjuan
f83ff9b655 staticcheck: fix all staticcheck warnings in the project 2020-04-14 20:16:10 +02:00
Hector Sanjuan
b3853caf36 Dependency ugprade: changes needed
* Libp2p protectors no longer needed, use PSK directly
* Generate cluster 32-byte secret here (helper gone from pnet)
* Switch to go-log/v2 in all places
* DHT bootstrapping not needed. Adjust DHT options for tests.
* Do not rely on dissappeared CidToDsKey and DsKeyToCid functions fro dshelp.
* Disable QUIC (does not support private networks)
* Fix tests: autodiscovery started working properly
2020-03-22 14:50:25 +01:00
Hector Sanjuan
19aff486c4
Merge pull request #1016 from ipfs/fix/1006-blake
Adder: fix #1006: Specify block format, hash and length when adding
2020-03-13 22:12:46 +01:00
Hector Sanjuan
ede1b89252 Adder: fix #1006: Specify block format, hash and length when adding
Currently we were only specifying the block format. When adding with
a custom hash function, even though we produced the right cids, IPFS
did not know the hash function and ended up storing them using SHA256.

Additionally, since NodeWithMeta serializes the CID, we do not need
to carry a Format parameter (which specifies the Codec): it is already
embedded.

Tests have been added and BlockPut in ipfshttp now checks that the
response's CID matches the data sent. This will catch errors like
what was happening, but also any data corruption between cluster and
IPFS during the block upload.
2020-03-13 14:43:58 +01:00
Hector Sanjuan
45ca4305d2 API/client: add context to http requests 2020-03-13 14:26:39 +01:00
Hector Sanjuan
531379b1d9
Feature: Support multiple listeners in configuration
* add ipv6 listening addresses to the default config

* ipfsproxy: support multiple listeners. Add default ipv6.

* mm

* restapi: support multiple listen addresses. enable ipv6

* cluster_config: format default listen addresses

* commands: update for multiple listeners. Fix randomports for udp and ipv6.

* ipfs-cluster-service: fix randomports test

* multiple listeners: fix remaining tests

* golint

* Disable ipv6 in defaults

It is not supported by docker by default. It is not supported in travis-CI
build environments. User can enable it now manually.

* proxy: disable ipv6 in test

* ipfshttp: fix test

Co-authored-by: @RubenKelevra <cyrond@gmail.com>
2020-02-28 11:16:16 -05:00
Yang Hau
7986d94242
fix: Fix typos (#1001)
Fix typos in files
2020-02-03 10:30:04 +01:00
Hector Sanjuan
9c61bf6f26 Fix #983: Correctly print paths when adding 2019-12-17 23:01:25 +01:00
Kishan Sagathiya
5258a4d428 Remove map pintracker (#944)
This removes mappintracker and sets stateless tracker as the default (and only) pintracker component.

Because the stateless tracker matches the cluster state with only ongoing operations being kept on memory, and additional information provided by ipfs-pin-ls, syncing operations are not necessary. Therefore the Sync/SyncAll operations are removed cluster-wide.
2019-12-12 21:22:54 +01:00
Kishan Sagathiya
e1faf12bae ipfsproxy: hijack repo/gc and trigger cluster-wide GC
This adds hijacking of the repo/gc endpoint to the proxy to do cluster-wide gc.
2019-12-06 13:08:57 +01:00
Hector Sanjuan
ba021e8bb4 restapi: fix logging
Logging with Infof when there is nothing to format.
2019-12-02 11:56:25 +01:00
Hector Sanjuan
71217ac5d1 restapi client: support talking to unix sockets
Setting APIAddr to a Unix socket multiaddress will just work.
2019-12-02 11:56:16 +01:00
Kishan Sagathiya
a178e5988a Avoid out of index error in failover (#951)
Fixes panic when using the load balancing client.
2019-11-13 12:15:29 +01:00
Hector Sanjuan
f788a83d23 Fix lbclient test with new multiaddresses 2019-11-07 20:19:07 +01:00
Hector Sanjuan
249d9007d2 Merge branch 'master' into feat/cluster-gc 2019-11-07 18:35:42 +01:00
Kishan Mohanbhai Sagathiya
4d8ef92b3d health/graph: Improve graph
Mark local, trusted peers. Add peernames. Improve display.
2019-11-07 10:47:29 +01:00
Hector Sanjuan
e34bddd3ab Do not start Autonat when not used
Avoid chaining options as not necessary.

Add functions for common options and protector creation.
2019-11-05 12:51:18 +01:00
Kishan Mohanbhai Sagathiya
c7f37ce9f4 Use quic and use tls by default in restapi and restapi client
(cherry picked from commit 7ab1cd66ed17fda923b149628cae18ea96d1cf45)
2019-11-05 12:51:18 +01:00
Hector Sanjuan
af29bbc944 Pin-expiration: error when pinning expired pins 2019-11-05 12:22:52 +01:00
Kishan Mohanbhai Sagathiya
e4e1cbea6e Fix #481: Pin expiration
This adds a new PinOption: ExpireAt.

The StateSync ticker will check and unpin expired pins from the Cluster.

ipfs-cluster-ctl supports an "expire-in" which gives a duration.
2019-11-05 10:40:48 +01:00
Kishan Sagathiya
31534a429b Fix #374: health metrics improvements
- Human-sizes for freespace metrics. Display whether if metric is
expires in something like "expires in 3m".
- When not passing metric name `ipfs-cluster-ctl health metrics` hits
the the metrics endpoint which returns a list of available metrics and
displays to user
- Humanize metrics output
- Sort metrics output
2019-10-24 16:37:26 +02:00
Kishan Mohanbhai Sagathiya
492b5612e7 Add ability to run Garbage Collector on all peers
- cluster method, ipfs connector method, rpc and rest apis,
command, etc for repo gc
    - Remove extra space from policy generator
    - Added special timeout for `/repo/gc` call to IPFS
    - Added `RepoGCLocal` cluster rpc method, which will be used to run gc
    on local IPFS daemon
    - Added peer name to the repo gc struct
    - Sorted with peer ids, while formatting(only affects cli
    results)
    - Special timeout setting where timeout gets checked from last update
    - Added `local` argument, which would run gc only on contacted peer
2019-10-22 11:13:19 +05:30
Hector Sanjuan
582968f44e
Merge pull request #915 from ipfs/feat/proxy-logging
API logs for IPFS Proxy
2019-09-27 12:07:34 +02:00
Hector Sanjuan
ce20e86fdb
Merge pull request #907 from ipfs/feat/local-add
`--local` flag for `ctl add`
2019-09-23 13:13:59 -04:00
Hector Sanjuan
a98292bfa6
Merge pull request #893 from ipfs/feat/recover-all
Pin recover on all peers
2019-09-23 13:13:48 -04:00
Kishan Mohanbhai Sagathiya
adc7c5b131 Merge branch 'master' of github.com:ipfs/ipfs-cluster into test2 2019-09-10 20:03:05 +07:00
Kishan Mohanbhai Sagathiya
7f6c45812f Fix test 2019-09-10 15:27:47 +07:00
Kishan Mohanbhai Sagathiya
bc357400f6 API logs for IPFS Proxy 2019-09-10 13:59:03 +07:00
Hector Sanjuan
12141a562e
Merge pull request #913 from ipfs/fix/add-params
Fix: consolidate parsing of PinOptions
2019-09-09 14:52:26 +02:00
Hector Sanjuan
963bf8877f
Merge pull request #894 from ipfs/feat/http-logging
Logging for Cluster HTTP APIs
2019-09-09 10:30:01 +02:00
Hector Sanjuan
18dad223b2
Merge pull request #912 from ipfs/fix/allocations
Fix: handling allocations
2019-09-09 10:22:14 +02:00
Kishan Mohanbhai Sagathiya
9cb1cdeaff Merge branch 'master' into feat/recover-all 2019-09-08 17:06:43 +07:00
Kishan Mohanbhai Sagathiya
b1748d13c4 Use absolute path 2019-09-07 19:05:57 +07:00
Hector Sanjuan
11d5f6a3f2 Fix: consolidate parsing of PinOptions
This removes duplicated code to parse PinOptions.
2019-09-06 18:38:55 +02:00
Hector Sanjuan
96752e4e58 Fix: handling allocations
* pin() should not allocate if allocations are already provided
* pin() should not skip pinning if the exact same pin exists
  * Additionally this was unreliable as it allocated it before
    so the pin may have existed but the allocations may have been
    artificially changed.
* pin() re-uses existing pin when pin options are the same and thus
  avoids changing the allocations of a pin.

As a side effect, this fixes re-allocations which were broken: peers
called `shouldPeerRepinCid()` and instead of repinning that single
cid proceeded to repin the full state. For every pin.

Additionally tests have been adapted. It may be that some re-alloc tests
were very unreliable for the problems above.
2019-09-06 17:56:00 +02:00
Hector Sanjuan
9eb0deee4e
Merge pull request #891 from ipfs/feat/metadata
Metadata can be set and shown from `ctl`
2019-09-06 15:46:29 +02:00
Hector Sanjuan
2d389428cd Merge branch 'master' into feat/metadata 2019-09-06 15:17:05 +02:00
Hector Sanjuan
81ea1e76bc Merge branch 'master' into feat/sort-responses 2019-09-06 15:13:54 +02:00
Hector Sanjuan
f45c8544a7
Merge pull request #890 from ipfs/feat/allocations-for-add
Added UserAllocations support for `add`
2019-09-06 15:03:22 +02:00
Hector Sanjuan
d63a7fd641
Merge pull request #877 from ipfs/fix/ipfs-to-p2p
Use `p2p` protocol name over `ipfs` for multiaddr
2019-09-06 15:00:36 +02:00
Kishan Mohanbhai Sagathiya
5f57feab54 Addressed reviews on tests 2019-08-31 11:15:16 +05:30
Kishan Mohanbhai Sagathiya
db11f77396 Remove unwanted configs 2019-08-30 09:13:18 +05:30
Kishan Mohanbhai Sagathiya
6c863df9fe Merge branch 'master' of github.com:ipfs/ipfs-cluster into feat/http-logging 2019-08-29 21:37:04 +05:30
Kishan Mohanbhai Sagathiya
d43c8fb4a4 Make api logger work 2019-08-29 20:49:53 +05:30
Kishan Mohanbhai Sagathiya
49af782444 Use custom io.Writer for logging, Use go test instead of sharness tests 2019-08-28 22:13:23 +05:30
Kishan Mohanbhai Sagathiya
a684c9567d Add with option --local
This commit introduces `--local` option for `ctl add` which would add
content only the local ipfs peer and then pin it according to pin
options (fetching from the local peer)
For achieving this, a new local dag service is introduced
2019-08-28 18:39:30 +05:30
Kishan Mohanbhai Sagathiya
62a998a694 Use custom log formatter when logging to stdout
- Instead of using a boolean config element(send logs to std out or
file), use a string element that would store path of the log file.
That way user has freedom to choose the filepath. Use std out if this
path is empty.
2019-08-27 14:41:15 +05:30
Kishan Mohanbhai Sagathiya
7b35524779 Addressed reviews 2019-08-26 14:30:45 +05:30