Commit Graph

1966 Commits

Author SHA1 Message Date
Hector Sanjuan
d2042b8092 follow: set shorhand for setting loglevels 2019-12-16 13:42:35 +01:00
Hector Sanjuan
6a35c19acf follow: better text formatting 2019-12-16 13:42:35 +01:00
Hector Sanjuan
f578145e59 follow: hide blockstore error 2019-12-16 13:42:35 +01:00
Hector Sanjuan
0671159bee Crdt: fix OfflineState comment 2019-12-16 13:42:35 +01:00
Hector Sanjuan
78f160aeb8 follow: better errors when doing "list" 2019-12-16 13:42:35 +01:00
Hector Sanjuan
47b2aadbba Cluster config: Slightly increase default pin_recover interval 2019-12-16 13:42:35 +01:00
Hector Sanjuan
f8a1e0f4b9 pintracker: take care of tests
Simplify the tests, remove things that are not used at all, align the
behaviour of the mocks, add methods to test the correct behaviour of Status
etc.
2019-12-16 13:42:35 +01:00
Hector Sanjuan
9f660ba38e Fix: stateless: cluster should pin items that are in the state but not on ipfs
StateSync() used to take care of this by issuing Track() calls. But this
functionality was removed.

This starts returning items that are in the state but not on IPFS as
PIN_ERRORs. It ensures that the Recover methods see them so that they can
trigger repinnings for missing items. This covers cases where the user
modifies the ipfs state manually, or resets the ipfs daemon but keeps the
cluster state, and cases where cluster was stopped half-way through a pinning.
2019-12-16 13:42:35 +01:00
Hector Sanjuan
c5d1de9b85
Merge pull request #979 from ipfs/fix/stateless-sync
Fix: stateless: cluster should pin items that are in the state but not on ipfs
2019-12-16 12:10:19 +01:00
Hector Sanjuan
e3cf51ac0e Cluster config: Slightly increase default pin_recover interval 2019-12-16 12:09:08 +01:00
Hector Sanjuan
09d933fde1 pintracker: take care of tests
Simplify the tests, remove things that are not used at all, align the
behaviour of the mocks, add methods to test the correct behaviour of Status
etc.
2019-12-13 12:03:01 +01:00
Hector Sanjuan
8b6fd1fabe Fix: stateless: cluster should pin items that are in the state but not on ipfs
StateSync() used to take care of this by issuing Track() calls. But this
functionality was removed.

This starts returning items that are in the state but not on IPFS as
PIN_ERRORs. It ensures that the Recover methods see them so that they can
trigger repinnings for missing items. This covers cases where the user
modifies the ipfs state manually, or resets the ipfs daemon but keeps the
cluster state, and cases where cluster was stopped half-way through a pinning.
2019-12-13 12:00:03 +01:00
Hector Sanjuan
26f553f3db follow: fix typos in comments/docs 2019-12-13 09:51:15 +01:00
Hector Sanjuan
7646cffa38 follow: update stateless tracker creation to new signature 2019-12-13 09:50:44 +01:00
Hector Sanjuan
6ad8974bde cluster: Shutdown informer components 2019-12-13 09:48:12 +01:00
Hector Sanjuan
31f4afadea cluster: Improve Shutdown() docstring 2019-12-13 09:47:42 +01:00
Hector Sanjuan
c8b7409685 follow: makefile: fix "all" rule 2019-12-13 09:47:01 +01:00
Hector Sanjuan
3bf77055df Merge remote-tracking branch 'origin/master' into feat/ipfs-cluster-follow 2019-12-12 21:32:02 +01:00
Kishan Sagathiya
5258a4d428 Remove map pintracker (#944)
This removes mappintracker and sets stateless tracker as the default (and only) pintracker component.

Because the stateless tracker matches the cluster state with only ongoing operations being kept on memory, and additional information provided by ipfs-pin-ls, syncing operations are not necessary. Therefore the Sync/SyncAll operations are removed cluster-wide.
2019-12-12 21:22:54 +01:00
Hector Sanjuan
2da2dc8c86
Merge pull request #956 from ipfs/tests/re-enable-quic
Tests: re-enable quic listen address on tests
2019-12-09 09:32:56 +01:00
Hector Sanjuan
5121a30770
Release 0.12.0-alpha1 2019-12-07 15:46:04 +01:00
Hector Sanjuan
4ea830f74e Feat: ipfs-cluster-follow
This adds a new cluster command: ipfs-cluster-follow.

This command allows initializing and running follower peers as configured by a
remote-source configuration. The command can list configured peers
and obtain information for each of them.

Peers are launched with the rest API listening on a local unix socket. The
command can be run to list the items in the cluster pinset using this
endpoint. Alternatively, if no socket is present, the peer will be assumed to
be offline and the pin list will be directly read from the datastore.

Cluster peers launched with this command (and their configurations) are
compatible with ipfs-cluster-ctl and ipfs-cluster-service. We purposely do not
support most configuration options here. Using ipfs-cluster-ctl or launching
the peers using ipfs-cluster-service is always an option when the usecase
deviates from that supported by ipfs-cluster-follow.

Examples:

$ ipfs-cluster-follow -> list configured peers
$ ipfs-cluster-follow --help
$ ipfs-cluster-follow <clusterName> init <url>
$ ipfs-cluster-follow <clusterName> info
$ ipfs-cluster-follow <clusterName> run
$ ipfs-cluster-follow <clusterName> list
2019-12-07 15:38:59 +01:00
Hector Sanjuan
ce3c50187e
config manager: Improve source-fetch error handling (#967)
config manager: Improve source-fetch error handling

* Allow detection of source-download errors by adding IsSourceDownloadError method.
* Treat non-success response codes as a failure to fetch the source.
* Set the Source url before errors happens so it can be consulted even in that case.
2019-12-07 12:32:43 +01:00
Hector Sanjuan
cf0afdcbc3 Tests: re-enable quic listen address on tests 2019-12-07 12:26:55 +01:00
Kishan Sagathiya
e1faf12bae ipfsproxy: hijack repo/gc and trigger cluster-wide GC
This adds hijacking of the repo/gc endpoint to the proxy to do cluster-wide gc.
2019-12-06 13:08:57 +01:00
Hector Sanjuan
db352534df
Merge pull request #975 from ipfs/update/quic2
Update QUIC transport dependency
2019-12-06 11:27:26 +01:00
Hector Sanjuan
f587b453c8
Merge pull request #970 from ipfs/feat/service-extract-to-cmdutils
Service: extract some methods to cmdutils
2019-12-06 11:08:58 +01:00
Hector Sanjuan
61778f50fd
Merge pull request #965 from ipfs/fix/pprof
Use index endpoint with a trailing slash
2019-12-06 11:07:09 +01:00
Hector Sanjuan
a6c83f72ba Fix #973: update quic-transport
Did not build in windows before...
2019-12-06 10:58:30 +01:00
Hector Sanjuan
d0b2c80a15 Dockerfile formatting 2019-12-06 10:56:17 +01:00
Kishan Mohanbhai Sagathiya
a136509522 No need to verify log levels 2019-12-06 11:22:08 +05:30
Hector Sanjuan
418587a946
Merge pull request #968 from ipfs/fix/libp2p-api-with-crdt
Fix: disable default libp2p API endpoint with CRDT consensus
2019-12-05 16:14:35 +01:00
Kishan Sagathiya
30fd5ee6a0 Feat: Allow and run multiple informer components (#962)
This introduces the possiblity of running Cluster with multiple informer components. The first one in the list is the used for "allocations". This is just groundwork for working with several informers in the future.
2019-12-05 15:08:43 +01:00
Kishan Mohanbhai Sagathiya
df36995213 detail description and example for how to customize logging 2019-12-04 23:22:38 +05:30
Kishan Mohanbhai Sagathiya
794646ff12 Addressed reviews 2019-12-04 23:10:35 +05:30
Hector Sanjuan
281653bdbd
Merge pull request #971 from ipfs/feat/peer-addresses
config: Add PeerAddresses
2019-12-03 13:16:37 +01:00
Hector Sanjuan
dc75343dc4
Merge pull request #969 from ipfs/feat/restapi-unix-sockets
RestAPI client: support unix sockets
2019-12-03 13:15:48 +01:00
Hector Sanjuan
f6ea95bccf Service/cmdutils: fix small typos (review) 2019-12-02 16:08:47 +01:00
Hector Sanjuan
d2bf1bc7b6 config: Add PeerAddresses
This adds a PeerAddresses entry to the main cluster configuration.

The peer will ingest and potentially connect to those peer addresses during
the start (similarly to the ones in the peerstore).

This allows to provide "bootstrap" (as in "peers we connect to") addresses
directly in the configuration, which is useful when distributing a single
configuration template that will allow a cluster peer to know where to connect
on the first boot.
2019-12-02 15:08:37 +01:00
Hector Sanjuan
5cc9ed0c4f ipfs-cluster-service: extract some helpers to cmdutils
This will allow better re-use from other CLIs.
2019-12-02 11:57:24 +01:00
Hector Sanjuan
454efe83fd ipfs-cluster-service: rename cluster node -> cluster peer in help.
We use "peer" everywhere. This is an old leftover.
2019-12-02 11:57:13 +01:00
Hector Sanjuan
ba021e8bb4 restapi: fix logging
Logging with Infof when there is nothing to format.
2019-12-02 11:56:25 +01:00
Hector Sanjuan
71217ac5d1 restapi client: support talking to unix sockets
Setting APIAddr to a Unix socket multiaddress will just work.
2019-12-02 11:56:16 +01:00
Hector Sanjuan
578ee29bdf Fix: disable default libp2p API endpoint with CRDT consensus 2019-12-02 11:54:51 +01:00
Kishan Mohanbhai Sagathiya
42b1413e23 Use index endpoint with a trailing slash
On using `/debug/pprof` as index endpoint, profiles contained by it
would point to wrong url. For example, link for `allocs` would be
`/debug/allocs` (results in 404), whereas it should actually be
`/debug/pprof/allocs`.

Use a trailing slash to get correct links. `/debug/pprof` will
automatically be redirected to `/debug/pprof/`
2019-11-28 00:39:35 +05:30
Kishan Mohanbhai Sagathiya
e99abf7688 Do not modify global var ipfscluster.LoggingFacilities 2019-11-25 17:18:50 +05:30
Kishan Mohanbhai Sagathiya
3d625184a3 Keep debug first to avoid iterations 2019-11-25 17:10:46 +05:30
Kishan Mohanbhai Sagathiya
a384057d39 Merge branch 'master' into feat/log-env-flags 2019-11-25 16:58:43 +05:30
Kishan Mohanbhai Sagathiya
f0c83f59b4 Don't error if a loglevel is passed twice, but warn 2019-11-25 16:57:48 +05:30
Kishan Mohanbhai Sagathiya
047d3fe615 Use a single log-level flag 2019-11-25 16:41:49 +05:30