Commit Graph

146 Commits

Author SHA1 Message Date
Kishan Mohanbhai Sagathiya
1fd4e41c99 Use urfave/cli's environment variable support for loglevels
- introduce a new flag for loglevels for individual components
- handle all log related flags together so that we can maintain priority
among log level flags (loglevel < component-loglevel < debug)
2019-11-16 15:37:57 +05:30
Kishan Mohanbhai Sagathiya
d8929aa5a6 Environment variable to set loglevels for individual identifiers
- Example: export LOG_FACS="cluster:debug,raft:debug"
2019-11-14 22:54:26 +05:30
Hector Sanjuan
7c5d86ee86 Fix graph not displaying all ipfs nodes when all-ipfs-peers enabled 2019-11-08 12:48:02 +01:00
Hector Sanjuan
f9b52e02b9 Fix graph tests failing and randomly failing 2019-11-07 20:18:50 +01:00
Hector Sanjuan
249d9007d2 Merge branch 'master' into feat/cluster-gc 2019-11-07 18:35:42 +01:00
Hector Sanjuan
5f265b017a Change colors and shapes 2019-11-07 12:49:51 +01:00
Kishan Mohanbhai Sagathiya
4d8ef92b3d health/graph: Improve graph
Mark local, trusted peers. Add peernames. Improve display.
2019-11-07 10:47:29 +01:00
Kishan Mohanbhai Sagathiya
56ef75b50c Use TLS instead of secio for security 2019-11-05 12:50:46 +01:00
Hector Sanjuan
dcb6c0805b
Merge pull request #946 from ipfs/dockerfiles-improvs
Dockerfiles: download modules in advance
2019-11-05 12:27:51 +01:00
Hector Sanjuan
af29bbc944 Pin-expiration: error when pinning expired pins 2019-11-05 12:22:52 +01:00
Kishan Mohanbhai Sagathiya
e4e1cbea6e Fix #481: Pin expiration
This adds a new PinOption: ExpireAt.

The StateSync ticker will check and unpin expired pins from the Cluster.

ipfs-cluster-ctl supports an "expire-in" which gives a duration.
2019-11-05 10:40:48 +01:00
Hector Sanjuan
073a0dfa58 Dockerfiles: download modules in advance
This should speed up Dockerfile builds, particularly when building after
updating the code since the modules will not be to be re-downloaded.
2019-10-28 10:23:18 +01:00
Kishan Sagathiya
7b499263af service init --randomports:
Allow initialize cluster with random ports that are ununsed (at the moment of initialization)
2019-10-24 17:54:50 +02:00
Kishan Sagathiya
31534a429b Fix #374: health metrics improvements
- Human-sizes for freespace metrics. Display whether if metric is
expires in something like "expires in 3m".
- When not passing metric name `ipfs-cluster-ctl health metrics` hits
the the metrics endpoint which returns a list of available metrics and
displays to user
- Humanize metrics output
- Sort metrics output
2019-10-24 16:37:26 +02:00
Kishan Mohanbhai Sagathiya
492b5612e7 Add ability to run Garbage Collector on all peers
- cluster method, ipfs connector method, rpc and rest apis,
command, etc for repo gc
    - Remove extra space from policy generator
    - Added special timeout for `/repo/gc` call to IPFS
    - Added `RepoGCLocal` cluster rpc method, which will be used to run gc
    on local IPFS daemon
    - Added peer name to the repo gc struct
    - Sorted with peer ids, while formatting(only affects cli
    results)
    - Special timeout setting where timeout gets checked from last update
    - Added `local` argument, which would run gc only on contacted peer
2019-10-22 11:13:19 +05:30
Hector Sanjuan
ce20e86fdb
Merge pull request #907 from ipfs/feat/local-add
`--local` flag for `ctl add`
2019-09-23 13:13:59 -04:00
Kishan Mohanbhai Sagathiya
fe57e33726 Addressed reviews 2019-09-21 14:07:26 +05:30
Hector Sanjuan
3c4859c74c
Release 0.11.0 2019-09-13 18:57:25 +02:00
Hector Sanjuan
d21dffa923
Release 0.11.0-rc10 2019-09-13 10:52:13 +02:00
Kishan Mohanbhai Sagathiya
adc7c5b131 Merge branch 'master' of github.com:ipfs/ipfs-cluster into test2 2019-09-10 20:03:05 +07:00
Hector Sanjuan
cf189799f2
Release 0.11.0-rc9 2019-09-09 14:54:52 +02:00
Hector Sanjuan
4e75bee03d
Merge pull request #909 from ipfs/fix/rpl-confusion
Explanation of 0 replication factor
2019-09-09 10:31:18 +02:00
Hector Sanjuan
2d389428cd Merge branch 'master' into feat/metadata 2019-09-06 15:17:05 +02:00
Hector Sanjuan
f45c8544a7
Merge pull request #890 from ipfs/feat/allocations-for-add
Added UserAllocations support for `add`
2019-09-06 15:03:22 +02:00
Hector Sanjuan
d63a7fd641
Merge pull request #877 from ipfs/fix/ipfs-to-p2p
Use `p2p` protocol name over `ipfs` for multiaddr
2019-09-06 15:00:36 +02:00
Kishan Mohanbhai Sagathiya
7155fa87c3 Explanation of 0 replication factor 2019-09-02 09:22:50 +05:30
Kishan Mohanbhai Sagathiya
a684c9567d Add with option --local
This commit introduces `--local` option for `ctl add` which would add
content only the local ipfs peer and then pin it according to pin
options (fetching from the local peer)
For achieving this, a new local dag service is introduced
2019-08-28 18:39:30 +05:30
Hector Sanjuan
1cfea47696
Release 0.11.0-rc8 2019-08-27 12:55:41 +02:00
Kishan Mohanbhai Sagathiya
cd86211766 Sharness test, dont print metadata in cmd 2019-08-27 09:32:34 +05:30
Hector Sanjuan
46cd5327fa
Release 0.11.0-rc7 2019-08-26 14:58:49 +02:00
Kishan Mohanbhai Sagathiya
7b35524779 Addressed reviews 2019-08-26 14:30:45 +05:30
Hector Sanjuan
33f111c44d mDNS: attach mDNS inside the Cluster. Allow interval configuration.
Setting up mDNS outside the Cluster is dirtier and allows less configuration.

This adds MDNSInterval to the cluster config options and allow disabling it
when the option is set to 0.
2019-08-24 17:24:18 +02:00
Hector Sanjuan
aab5f9bd0b Enable p2p addresses in a single place 2019-08-23 22:45:32 +02:00
Kishan Mohanbhai Sagathiya
0b6c376bad Metadata can be set and shown from ctl
This commit adds
- `metadata` option to `pin add` and `add` commands
- ability to set metadata from `/add` rest API
2019-08-19 10:29:06 +05:30
Kishan Mohanbhai Sagathiya
6bd897ac47 Added UserAllocations support for add 2019-08-16 18:13:06 +05:30
Kishan Mohanbhai Sagathiya
6656b80a00 Some more occurences of /ipfs
and use  SwapToP2pMultiaddrs (very helpful since ipfs still send
addresses with `/ipfs` tag)
2019-08-16 11:56:09 +05:30
Hector Sanjuan
5e4c8d9418 Ctl: increase host-name spacing in status output
Allow for slightly longer hostnames
2019-08-14 14:10:11 +02:00
Hector Sanjuan
dc2e73dfc2
Release 0.11.0-rc6 2019-08-13 20:25:53 +02:00
Hector Sanjuan
dc3170b1d2 cluster: add mDNS service discovery
I always thought the libp2p node would do this, but it is not the case.
With this, CRDT peers are able to autodiscover on local networks.
2019-08-13 19:32:38 +02:00
Hector Sanjuan
5303a7f721
Merge pull request #874 from ipfs/feat/865-init-consensus
Service: Select consensus on "init" (not on "daemon")
2019-08-13 10:21:45 +02:00
Hector Sanjuan
3b3e3493a8 Service: init: improve log messages
Align with previous behaviour and make sure it is logged that
the configuration file was written.

Do not say "peerstore written with 0 entries" as that might be
taken like an error.
2019-08-12 16:58:06 +02:00
Hector Sanjuan
d8c20adc4e
Merge pull request #869 from ipfs/fix/732-pin-update-the-good-way
Fix #732: Introduce native pin/update
2019-08-12 12:15:30 +02:00
Kishan Mohanbhai Sagathiya
684f58894d Use string flag instead of string slice
Because string slice is not very user friendly. Using string instead
would allow us to pass a comma separated list of arguments in one option
2019-08-12 11:31:26 +02:00
Hector Sanjuan
2311bdac19 daemon: do consensus things inside a switch 2019-08-12 10:39:49 +02:00
Hector Sanjuan
f32a53c4e6 Daemon: "init" do not TrustAll when passing --peers 2019-08-12 10:25:55 +02:00
Kishan Mohanbhai Sagathiya
c6192cebf4 Use p2p protocol name over ipfs for multiaddr 2019-08-11 19:06:45 +05:30
Hector Sanjuan
063c5f1b78 Service: Select consensus on "init" (not on "daemon")
Fixes #865.

This makes the necessary changes so that consensu is selected on "init" with a
flag set, by default, to "crdt". This generates only a "crdt" or a "raft"
section, not both.

If the configuration file has a "raft" section, "raft" will be used to start
the daemon. If it has a "crdt" section, "crdt" will be used. If it has none or
both sections, an error will happen.

This also affects "state *" commands, which will now autoselect how to work
from the existing configuration.
2019-08-09 19:20:53 +02:00
Hector Sanjuan
1eade4ae58 Fix #732: Introduce native pin/update
This introduces a pin/update operation which allows to Pin a new item to
cluster indicating that said pin is an update to an already-existing pin.

When this is the case, all the configuration for the existing pin is copied to
the new one (including allocations). The IPFS connector will then trigger
pin/update directly in IPFS, allowing an efficient pinning based on
DAG-differences. Since the allocations where the same for both pins,
the pin/update can proceed.

PinUpdate does not unpin the previous pin (it is not possible to do this
atomically in cluster like it happens in IPFS). The user can manually do it
after the pin/update is done.

Internally, after a lot of deliberations on what the optimal way for this is,
I opted for adding a `PinUpdate` option to the `PinOptions` type (carries the
CID to update from). In order to carry this option from the REST API to the
IPFS Connector, it is serialized in the Protobuf (and stored in the
datastore). There is no other way to do this in a simple fashion since the Pin
object is piece of information that is sent around.

Additionally, making it a PinOption plays well with the Pin/PinPath APIs which
need little changes. Effectively, you are pinning a new thing. You are just
indicating that it should be configured from an existing one.

Fixes #732
2019-08-09 16:11:52 +02:00
Hector Sanjuan
00e78a6b6d
Daemon: support remote configuration (#868)
* Daemon: support remote configuration

This:

* Adds support for fetching the configuration from a remote HTTP location:

`ipfs-cluster-service init http://localhost:8080/ipfs/Qm...` will instruct
cluster to read the configuration file from ipfs on start (potentially making
use of ipns and dnslink).

This is done by creating a `service.json` like `{ "source": <url> }`.

The source is then read when loading that configuration every time the daemon starts.

This allows to let users always use a mutating remote configuration, potentially
adding/removing trusted peers from the list or adjusting other things.

* Configuration and state helpers from ipfs-cluster-service have been extracted
to its own cmdutils package. This will help supporting something like an
`ipfs-cluster-follow` command in the next releases.

* Allows to disable the rest api by not defining it in the configuration (I thought
this was already so, but apparently only affected the ipfsproxy).

* Removes informer/allocator configurations from the daemon (--alloc). No one used
a non default pair. In fact, it was potentially buggy to use the reposize one.
2019-08-09 12:56:27 +02:00
Hector Sanjuan
eace9031dc
Merge pull request #856 from ipfs/feat/force-consensus-choice
service: Make --consensus a mandatory flag
2019-07-30 11:32:49 +02:00