4 PinTypes specify how CID is pinned
Changes to Pin and Unpin to handle different PinTypes
Tests for different PinTypes
Migration for new state format using new Pin datastructures
Visibility of the PinTypes used internally limited by default
License: MIT
Signed-off-by: Wyatt Daviau <wdaviau@cs.stanford.edu>
This adds the pakage api/rest/client which implements a go-client
for the REST API component. It also update the ipfs-cluster-ctl
tool to rely on it.
Originally, I wanted this to live it in it's own separate repository,
but the api client uses /api/types.go, which is part of cluster.
Therefore it would need to import all of cluster as a dependency.
ipfs-cluster-ctl would also need to import go-ipfs-cluster-api-client
as a dependency, creating circular gx deps which would be a mess to
maintain.
Only the splitting of cluster in multiple repositories (at least for
api, rest, ipfs-cluster-ctl, rest/client and test) would allow better
dependency management by allowing rest/client and the ctl tool
to only import what is needed, but this is something which brings
maintenance costs and can probably wait a bit until cluster is more stable.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
I've modified the peer identifier to be 'peername'. I've also
modified the TestLoadJSON to check that it is correctly read from
config and set to a default if empty.
Also added 'peername' fields to configurations for various tests.
I renamed Hostname to simply Name as to not imply relation to DNS.
Removed quotes from formatter, used helper function setting config,
and added defensive error check.
I added an option called --name or -n which specifies the name for
the added pin to be stored in consensus. Also added quotes around
name for allocations formatter.
Added Name field to be a part of the stored state of a Pin. This
allows the name to be stored as a part of the consensus if needed.
This is the bare minimum to use. Some other calls like status need
to be checked out.
This adds API, RPC calls to support RecoverAllLocal() (and expose RecoverLocal()
on the Rest API too). cluster-ctl is updated accordingly.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
I added a "hostname" property to a node's configuration file. Its
value defaults to the hostname provided by the OS, but can be
modified to anything beside an empty string in the config file.
The "hostname" was added to the output of the "id" call. Thus, peer
hostnames are available when listing peers.
The following commit reimplements ipfs-cluster configuration under
the following premises:
* Each component is initialized with a configuration object
defined by its module
* Each component decides how the JSON representation of its
configuration looks like
* Each component parses and validates its own configuration
* Each component exposes its own defaults
* Component configurations are make the sections of a
central JSON configuration file (which replaces the current
JSON format)
* Component configurations implement a common interface
(config.ComponentConfig) with a set of common operations
* The central configuration file is managed by a
config.ConfigManager which:
* Registers ComponentConfigs
* Assigns the correspondent sections from the JSON file to each
component and delegates the parsing
* Delegates the JSON generation for each section
* Can be notified when the configuration is updated and must be
saved to disk
The new service.json would then look as follows:
```json
{
"cluster": {
"id": "QmTVW8NoRxC5wBhV7WtAYtRn7itipEESfozWN5KmXUQnk2",
"private_key": "<...>",
"secret": "00224102ae6aaf94f2606abf69a0e278251ecc1d64815b617ff19d6d2841f786",
"peers": [],
"bootstrap": [],
"leave_on_shutdown": false,
"listen_multiaddress": "/ip4/0.0.0.0/tcp/9096",
"state_sync_interval": "1m0s",
"ipfs_sync_interval": "2m10s",
"replication_factor": -1,
"monitor_ping_interval": "15s"
},
"consensus": {
"raft": {
"heartbeat_timeout": "1s",
"election_timeout": "1s",
"commit_timeout": "50ms",
"max_append_entries": 64,
"trailing_logs": 10240,
"snapshot_interval": "2m0s",
"snapshot_threshold": 8192,
"leader_lease_timeout": "500ms"
}
},
"api": {
"restapi": {
"listen_multiaddress": "/ip4/127.0.0.1/tcp/9094",
"read_timeout": "30s",
"read_header_timeout": "5s",
"write_timeout": "1m0s",
"idle_timeout": "2m0s"
}
},
"ipfs_connector": {
"ipfshttp": {
"proxy_listen_multiaddress": "/ip4/127.0.0.1/tcp/9095",
"node_multiaddress": "/ip4/127.0.0.1/tcp/5001",
"connect_swarms_delay": "7s",
"proxy_read_timeout": "10m0s",
"proxy_read_header_timeout": "5s",
"proxy_write_timeout": "10m0s",
"proxy_idle_timeout": "1m0s"
}
},
"monitor": {
"monbasic": {
"check_interval": "15s"
}
},
"informer": {
"disk": {
"metric_ttl": "30s",
"metric_type": "freespace"
},
"numpin": {
"metric_ttl": "10s"
}
}
}
```
This new format aims to be easily extensible per component. As such,
it already surfaces quite a few new options which were hardcoded
before.
Additionally, since Go API have changed, some redundant methods have been
removed and small refactoring has happened to take advantage of the new
way.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
The ReadTimeout for the API was the same as for RPC. So the error
did not have time to be correctly returned to the client.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
Be more consistent when printing errors (i.e. if json encoding
is requested then it should be used for errors too). Do not
print "Request accepted".
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
This adds a replication_factor query argument to the API
endpoint which allows to set a replication factor per Pin.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
CidArg used to be an internal name for an argument that carried a Cid.
Now it has surfaced to API level and makes no sense. It is a Pin. It
represents a Pin (Cid, Allocations, Replication Factor)
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
New PeerManager, Allocator, Informer components have been added along
with a new "replication_factor" configuration option.
First, cluster peers collect and push metrics (Informer) to the Cluster
leader regularly. The Informer is an interface that can be implemented
in custom wayts to support custom metrics.
Second, on a pin operation, using the information from the collected metrics,
an Allocator can provide a list of preferences as to where the new pin
should be assigned. The Allocator is an interface allowing to provide
different allocation strategies.
Both Allocator and Informer are Cluster Componenets, and have access
to the RPC API.
The allocations are kept in the shared state. Cluster peer failure
detection is still missing and re-allocation is still missing, although
re-pinning something when a node is down/metrics missing does re-allocate
the pin somewhere else.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
