Commit Graph

254 Commits

Author SHA1 Message Date
Hector Sanjuan
b3853caf36 Dependency ugprade: changes needed
* Libp2p protectors no longer needed, use PSK directly
* Generate cluster 32-byte secret here (helper gone from pnet)
* Switch to go-log/v2 in all places
* DHT bootstrapping not needed. Adjust DHT options for tests.
* Do not rely on dissappeared CidToDsKey and DsKeyToCid functions fro dshelp.
* Disable QUIC (does not support private networks)
* Fix tests: autodiscovery started working properly
2020-03-22 14:50:25 +01:00
Hector Sanjuan
c80a207e41 Fix #1012: Trim paths on Makefile builds
This results in more reproduceable builds.

Additionally, sources command to list all go-files and not just those
one directory below.
2020-03-06 14:49:19 +01:00
Hector Sanjuan
5d44275d0e
Chore: Add license file with URLS (#1014)
* Chore: Add license file with URLS

Source: 59df5e58a2

* Chore: license copyright updates
2020-03-06 13:52:19 +01:00
Hector Sanjuan
531379b1d9
Feature: Support multiple listeners in configuration
* add ipv6 listening addresses to the default config

* ipfsproxy: support multiple listeners. Add default ipv6.

* mm

* restapi: support multiple listen addresses. enable ipv6

* cluster_config: format default listen addresses

* commands: update for multiple listeners. Fix randomports for udp and ipv6.

* ipfs-cluster-service: fix randomports test

* multiple listeners: fix remaining tests

* golint

* Disable ipv6 in defaults

It is not supported by docker by default. It is not supported in travis-CI
build environments. User can enable it now manually.

* proxy: disable ipv6 in test

* ipfshttp: fix test

Co-authored-by: @RubenKelevra <cyrond@gmail.com>
2020-02-28 11:16:16 -05:00
Yang Hau
7986d94242
fix: Fix typos (#1001)
Fix typos in files
2020-02-03 10:30:04 +01:00
Hector Sanjuan
3cdcb44af2
Release 0.12.1 2019-12-24 01:13:23 +01:00
Hector Sanjuan
1accd0e415 follow: close the RepoGCLocal RPC endpoint
Trusted peers should not be able to GC a follower.
2019-12-23 23:45:26 +01:00
Hector Sanjuan
db2ef9231a Fix #991: fix panic on "info" when IPFS not running 2019-12-23 23:45:14 +01:00
Hector Sanjuan
f20a524bbc Fix #992: Ensure the gateway variable is set on the run --init command 2019-12-23 23:32:06 +01:00
Kishan Mohanbhai Sagathiya
68abae9287 Merge branch 'master' into feat/alerts 2019-12-23 12:45:22 +05:30
Hector Sanjuan
09e4007c31
Release 0.12.0 2019-12-20 14:45:59 +01:00
Hector Sanjuan
60ce5b3228 Update copyrights to 2020x 2019-12-20 14:40:34 +01:00
Hector Sanjuan
0fa01d5435 Distribute the correct dual license with ctl and service 2019-12-20 14:04:48 +01:00
Hector Sanjuan
aad179ab29 follow: Missing line break in message 2019-12-19 20:40:06 +01:00
Hector Sanjuan
912a6f43fa follow: improve return error. Remove logger (unused) 2019-12-19 20:39:20 +01:00
Hector Sanjuan
0ae04c42c8 Follow: treat errors from Default() methods 2019-12-19 20:39:20 +01:00
Hector Sanjuan
eef37027e9 follow: Handle IPFS_GATEWAY as a flag option 2019-12-19 20:02:40 +01:00
Hector Sanjuan
274f52b46a follow: do not ignore ShowAppHelp() error 2019-12-19 19:56:49 +01:00
Hector Sanjuan
212b3b00d2 follow: handle error from ApplyEnvVars 2019-12-19 19:55:19 +01:00
Hector Sanjuan
10e90616f5 follow: rename listCmd to listClustersCmd and pinsetCmd to listCmd 2019-12-19 19:53:30 +01:00
Hector Sanjuan
130965d07f follow: show the notInitialized message when not initialized. 2019-12-19 19:50:46 +01:00
Hector Sanjuan
bf2c950016 Fix #986: Ensure ctrl-c always kills ipfs-cluster-follow 2019-12-19 18:34:18 +01:00
Hector Sanjuan
d12bc7daf7 Merge branch 'master' into feat/ipfs-cluster-follow 2019-12-18 13:49:00 +01:00
Hector Sanjuan
66abca4e32
Release 0.12.0-rc1 2019-12-16 17:58:30 +01:00
Hector Sanjuan
2391d2d246
Merge pull request #960 from ipfs/feat/log-env-flags
Environment variable to set loglevels for individual identifiers
2019-12-16 17:20:17 +01:00
Hector Sanjuan
1e47f74ef8 Follow: simplify tracing/metrics [no] setup a bit 2019-12-16 15:07:03 +01:00
Hector Sanjuan
0d5857a95a follow: support custom gateway url 2019-12-16 15:02:07 +01:00
Hector Sanjuan
a79994ed44 follow: improve ipfs connector handling in info 2019-12-16 14:55:36 +01:00
Hector Sanjuan
6aef2e8648 Follow: wait for IPFS to be running 2019-12-16 14:47:14 +01:00
Hector Sanjuan
ad7b57aec0 follow: do not show help when "info" called explicitally 2019-12-16 14:46:24 +01:00
Hector Sanjuan
1dd9207422 follow: support custom restapi endpoint in http client
by using CLUSTER_RESTAPI_HTTPLISTENMULTIADDRESS.
2019-12-16 14:22:45 +01:00
Hector Sanjuan
d2042b8092 follow: set shorhand for setting loglevels 2019-12-16 13:42:35 +01:00
Hector Sanjuan
6a35c19acf follow: better text formatting 2019-12-16 13:42:35 +01:00
Hector Sanjuan
f578145e59 follow: hide blockstore error 2019-12-16 13:42:35 +01:00
Hector Sanjuan
78f160aeb8 follow: better errors when doing "list" 2019-12-16 13:42:35 +01:00
Hector Sanjuan
26f553f3db follow: fix typos in comments/docs 2019-12-13 09:51:15 +01:00
Hector Sanjuan
7646cffa38 follow: update stateless tracker creation to new signature 2019-12-13 09:50:44 +01:00
Hector Sanjuan
c8b7409685 follow: makefile: fix "all" rule 2019-12-13 09:47:01 +01:00
Kishan Mohanbhai Sagathiya
d21860eee7 Merge branch 'master' into feat/alerts 2019-12-13 10:22:06 +05:30
Hector Sanjuan
3bf77055df Merge remote-tracking branch 'origin/master' into feat/ipfs-cluster-follow 2019-12-12 21:32:02 +01:00
Kishan Sagathiya
5258a4d428 Remove map pintracker (#944)
This removes mappintracker and sets stateless tracker as the default (and only) pintracker component.

Because the stateless tracker matches the cluster state with only ongoing operations being kept on memory, and additional information provided by ipfs-pin-ls, syncing operations are not necessary. Therefore the Sync/SyncAll operations are removed cluster-wide.
2019-12-12 21:22:54 +01:00
Kishan Mohanbhai Sagathiya
04069b8c81 Introduction of ctl alerts
- basic version, just alerts if peers are down
2019-12-13 00:21:28 +05:30
Hector Sanjuan
5121a30770
Release 0.12.0-alpha1 2019-12-07 15:46:04 +01:00
Hector Sanjuan
4ea830f74e Feat: ipfs-cluster-follow
This adds a new cluster command: ipfs-cluster-follow.

This command allows initializing and running follower peers as configured by a
remote-source configuration. The command can list configured peers
and obtain information for each of them.

Peers are launched with the rest API listening on a local unix socket. The
command can be run to list the items in the cluster pinset using this
endpoint. Alternatively, if no socket is present, the peer will be assumed to
be offline and the pin list will be directly read from the datastore.

Cluster peers launched with this command (and their configurations) are
compatible with ipfs-cluster-ctl and ipfs-cluster-service. We purposely do not
support most configuration options here. Using ipfs-cluster-ctl or launching
the peers using ipfs-cluster-service is always an option when the usecase
deviates from that supported by ipfs-cluster-follow.

Examples:

$ ipfs-cluster-follow -> list configured peers
$ ipfs-cluster-follow --help
$ ipfs-cluster-follow <clusterName> init <url>
$ ipfs-cluster-follow <clusterName> info
$ ipfs-cluster-follow <clusterName> run
$ ipfs-cluster-follow <clusterName> list
2019-12-07 15:38:59 +01:00
Hector Sanjuan
f587b453c8
Merge pull request #970 from ipfs/feat/service-extract-to-cmdutils
Service: extract some methods to cmdutils
2019-12-06 11:08:58 +01:00
Kishan Mohanbhai Sagathiya
a136509522 No need to verify log levels 2019-12-06 11:22:08 +05:30
Hector Sanjuan
418587a946
Merge pull request #968 from ipfs/fix/libp2p-api-with-crdt
Fix: disable default libp2p API endpoint with CRDT consensus
2019-12-05 16:14:35 +01:00
Kishan Sagathiya
30fd5ee6a0 Feat: Allow and run multiple informer components (#962)
This introduces the possiblity of running Cluster with multiple informer components. The first one in the list is the used for "allocations". This is just groundwork for working with several informers in the future.
2019-12-05 15:08:43 +01:00
Kishan Mohanbhai Sagathiya
df36995213 detail description and example for how to customize logging 2019-12-04 23:22:38 +05:30
Kishan Mohanbhai Sagathiya
794646ff12 Addressed reviews 2019-12-04 23:10:35 +05:30
Hector Sanjuan
f6ea95bccf Service/cmdutils: fix small typos (review) 2019-12-02 16:08:47 +01:00
Hector Sanjuan
5cc9ed0c4f ipfs-cluster-service: extract some helpers to cmdutils
This will allow better re-use from other CLIs.
2019-12-02 11:57:24 +01:00
Hector Sanjuan
454efe83fd ipfs-cluster-service: rename cluster node -> cluster peer in help.
We use "peer" everywhere. This is an old leftover.
2019-12-02 11:57:13 +01:00
Hector Sanjuan
578ee29bdf Fix: disable default libp2p API endpoint with CRDT consensus 2019-12-02 11:54:51 +01:00
Kishan Mohanbhai Sagathiya
e99abf7688 Do not modify global var ipfscluster.LoggingFacilities 2019-11-25 17:18:50 +05:30
Kishan Mohanbhai Sagathiya
f0c83f59b4 Don't error if a loglevel is passed twice, but warn 2019-11-25 16:57:48 +05:30
Kishan Mohanbhai Sagathiya
047d3fe615 Use a single log-level flag 2019-11-25 16:41:49 +05:30
Kishan Mohanbhai Sagathiya
361b819139 Try to set all loglevels with a single flag 2019-11-25 12:57:59 +05:30
Kishan Mohanbhai Sagathiya
1fd4e41c99 Use urfave/cli's environment variable support for loglevels
- introduce a new flag for loglevels for individual components
- handle all log related flags together so that we can maintain priority
among log level flags (loglevel < component-loglevel < debug)
2019-11-16 15:37:57 +05:30
Kishan Mohanbhai Sagathiya
d8929aa5a6 Environment variable to set loglevels for individual identifiers
- Example: export LOG_FACS="cluster:debug,raft:debug"
2019-11-14 22:54:26 +05:30
Hector Sanjuan
7c5d86ee86 Fix graph not displaying all ipfs nodes when all-ipfs-peers enabled 2019-11-08 12:48:02 +01:00
Hector Sanjuan
f9b52e02b9 Fix graph tests failing and randomly failing 2019-11-07 20:18:50 +01:00
Hector Sanjuan
249d9007d2 Merge branch 'master' into feat/cluster-gc 2019-11-07 18:35:42 +01:00
Hector Sanjuan
5f265b017a Change colors and shapes 2019-11-07 12:49:51 +01:00
Kishan Mohanbhai Sagathiya
4d8ef92b3d health/graph: Improve graph
Mark local, trusted peers. Add peernames. Improve display.
2019-11-07 10:47:29 +01:00
Kishan Mohanbhai Sagathiya
56ef75b50c Use TLS instead of secio for security 2019-11-05 12:50:46 +01:00
Hector Sanjuan
dcb6c0805b
Merge pull request #946 from ipfs/dockerfiles-improvs
Dockerfiles: download modules in advance
2019-11-05 12:27:51 +01:00
Hector Sanjuan
af29bbc944 Pin-expiration: error when pinning expired pins 2019-11-05 12:22:52 +01:00
Kishan Mohanbhai Sagathiya
e4e1cbea6e Fix #481: Pin expiration
This adds a new PinOption: ExpireAt.

The StateSync ticker will check and unpin expired pins from the Cluster.

ipfs-cluster-ctl supports an "expire-in" which gives a duration.
2019-11-05 10:40:48 +01:00
Hector Sanjuan
073a0dfa58 Dockerfiles: download modules in advance
This should speed up Dockerfile builds, particularly when building after
updating the code since the modules will not be to be re-downloaded.
2019-10-28 10:23:18 +01:00
Kishan Sagathiya
7b499263af service init --randomports:
Allow initialize cluster with random ports that are ununsed (at the moment of initialization)
2019-10-24 17:54:50 +02:00
Kishan Sagathiya
31534a429b Fix #374: health metrics improvements
- Human-sizes for freespace metrics. Display whether if metric is
expires in something like "expires in 3m".
- When not passing metric name `ipfs-cluster-ctl health metrics` hits
the the metrics endpoint which returns a list of available metrics and
displays to user
- Humanize metrics output
- Sort metrics output
2019-10-24 16:37:26 +02:00
Kishan Mohanbhai Sagathiya
492b5612e7 Add ability to run Garbage Collector on all peers
- cluster method, ipfs connector method, rpc and rest apis,
command, etc for repo gc
    - Remove extra space from policy generator
    - Added special timeout for `/repo/gc` call to IPFS
    - Added `RepoGCLocal` cluster rpc method, which will be used to run gc
    on local IPFS daemon
    - Added peer name to the repo gc struct
    - Sorted with peer ids, while formatting(only affects cli
    results)
    - Special timeout setting where timeout gets checked from last update
    - Added `local` argument, which would run gc only on contacted peer
2019-10-22 11:13:19 +05:30
Hector Sanjuan
ce20e86fdb
Merge pull request #907 from ipfs/feat/local-add
`--local` flag for `ctl add`
2019-09-23 13:13:59 -04:00
Kishan Mohanbhai Sagathiya
fe57e33726 Addressed reviews 2019-09-21 14:07:26 +05:30
Hector Sanjuan
3c4859c74c
Release 0.11.0 2019-09-13 18:57:25 +02:00
Hector Sanjuan
d21dffa923
Release 0.11.0-rc10 2019-09-13 10:52:13 +02:00
Kishan Mohanbhai Sagathiya
adc7c5b131 Merge branch 'master' of github.com:ipfs/ipfs-cluster into test2 2019-09-10 20:03:05 +07:00
Hector Sanjuan
cf189799f2
Release 0.11.0-rc9 2019-09-09 14:54:52 +02:00
Hector Sanjuan
4e75bee03d
Merge pull request #909 from ipfs/fix/rpl-confusion
Explanation of 0 replication factor
2019-09-09 10:31:18 +02:00
Hector Sanjuan
2d389428cd Merge branch 'master' into feat/metadata 2019-09-06 15:17:05 +02:00
Hector Sanjuan
f45c8544a7
Merge pull request #890 from ipfs/feat/allocations-for-add
Added UserAllocations support for `add`
2019-09-06 15:03:22 +02:00
Hector Sanjuan
d63a7fd641
Merge pull request #877 from ipfs/fix/ipfs-to-p2p
Use `p2p` protocol name over `ipfs` for multiaddr
2019-09-06 15:00:36 +02:00
Kishan Mohanbhai Sagathiya
7155fa87c3 Explanation of 0 replication factor 2019-09-02 09:22:50 +05:30
Kishan Mohanbhai Sagathiya
a684c9567d Add with option --local
This commit introduces `--local` option for `ctl add` which would add
content only the local ipfs peer and then pin it according to pin
options (fetching from the local peer)
For achieving this, a new local dag service is introduced
2019-08-28 18:39:30 +05:30
Hector Sanjuan
1cfea47696
Release 0.11.0-rc8 2019-08-27 12:55:41 +02:00
Kishan Mohanbhai Sagathiya
cd86211766 Sharness test, dont print metadata in cmd 2019-08-27 09:32:34 +05:30
Hector Sanjuan
46cd5327fa
Release 0.11.0-rc7 2019-08-26 14:58:49 +02:00
Kishan Mohanbhai Sagathiya
7b35524779 Addressed reviews 2019-08-26 14:30:45 +05:30
Hector Sanjuan
33f111c44d mDNS: attach mDNS inside the Cluster. Allow interval configuration.
Setting up mDNS outside the Cluster is dirtier and allows less configuration.

This adds MDNSInterval to the cluster config options and allow disabling it
when the option is set to 0.
2019-08-24 17:24:18 +02:00
Hector Sanjuan
aab5f9bd0b Enable p2p addresses in a single place 2019-08-23 22:45:32 +02:00
Kishan Mohanbhai Sagathiya
0b6c376bad Metadata can be set and shown from ctl
This commit adds
- `metadata` option to `pin add` and `add` commands
- ability to set metadata from `/add` rest API
2019-08-19 10:29:06 +05:30
Kishan Mohanbhai Sagathiya
6bd897ac47 Added UserAllocations support for add 2019-08-16 18:13:06 +05:30
Kishan Mohanbhai Sagathiya
6656b80a00 Some more occurences of /ipfs
and use  SwapToP2pMultiaddrs (very helpful since ipfs still send
addresses with `/ipfs` tag)
2019-08-16 11:56:09 +05:30
Hector Sanjuan
5e4c8d9418 Ctl: increase host-name spacing in status output
Allow for slightly longer hostnames
2019-08-14 14:10:11 +02:00
Hector Sanjuan
dc2e73dfc2
Release 0.11.0-rc6 2019-08-13 20:25:53 +02:00
Hector Sanjuan
dc3170b1d2 cluster: add mDNS service discovery
I always thought the libp2p node would do this, but it is not the case.
With this, CRDT peers are able to autodiscover on local networks.
2019-08-13 19:32:38 +02:00
Hector Sanjuan
5303a7f721
Merge pull request #874 from ipfs/feat/865-init-consensus
Service: Select consensus on "init" (not on "daemon")
2019-08-13 10:21:45 +02:00
Hector Sanjuan
3b3e3493a8 Service: init: improve log messages
Align with previous behaviour and make sure it is logged that
the configuration file was written.

Do not say "peerstore written with 0 entries" as that might be
taken like an error.
2019-08-12 16:58:06 +02:00
Hector Sanjuan
d8c20adc4e
Merge pull request #869 from ipfs/fix/732-pin-update-the-good-way
Fix #732: Introduce native pin/update
2019-08-12 12:15:30 +02:00