Commit Graph

165 Commits

Author SHA1 Message Date
Hector Sanjuan
8f06baa1bf Issue #162: Rework configuration format
The following commit reimplements ipfs-cluster configuration under
the following premises:

  * Each component is initialized with a configuration object
  defined by its module
  * Each component decides how the JSON representation of its
  configuration looks like
  * Each component parses and validates its own configuration
  * Each component exposes its own defaults
  * Component configurations are make the sections of a
  central JSON configuration file (which replaces the current
  JSON format)
  * Component configurations implement a common interface
  (config.ComponentConfig) with a set of common operations
  * The central configuration file is managed by a
  config.ConfigManager which:
    * Registers ComponentConfigs
    * Assigns the correspondent sections from the JSON file to each
    component and delegates the parsing
    * Delegates the JSON generation for each section
    * Can be notified when the configuration is updated and must be
    saved to disk

The new service.json would then look as follows:

```json
{
  "cluster": {
    "id": "QmTVW8NoRxC5wBhV7WtAYtRn7itipEESfozWN5KmXUQnk2",
    "private_key": "<...>",
    "secret": "00224102ae6aaf94f2606abf69a0e278251ecc1d64815b617ff19d6d2841f786",
    "peers": [],
    "bootstrap": [],
    "leave_on_shutdown": false,
    "listen_multiaddress": "/ip4/0.0.0.0/tcp/9096",
    "state_sync_interval": "1m0s",
    "ipfs_sync_interval": "2m10s",
    "replication_factor": -1,
    "monitor_ping_interval": "15s"
  },
  "consensus": {
    "raft": {
      "heartbeat_timeout": "1s",
      "election_timeout": "1s",
      "commit_timeout": "50ms",
      "max_append_entries": 64,
      "trailing_logs": 10240,
      "snapshot_interval": "2m0s",
      "snapshot_threshold": 8192,
      "leader_lease_timeout": "500ms"
    }
  },
  "api": {
    "restapi": {
      "listen_multiaddress": "/ip4/127.0.0.1/tcp/9094",
      "read_timeout": "30s",
      "read_header_timeout": "5s",
      "write_timeout": "1m0s",
      "idle_timeout": "2m0s"
    }
  },
  "ipfs_connector": {
    "ipfshttp": {
      "proxy_listen_multiaddress": "/ip4/127.0.0.1/tcp/9095",
      "node_multiaddress": "/ip4/127.0.0.1/tcp/5001",
      "connect_swarms_delay": "7s",
      "proxy_read_timeout": "10m0s",
      "proxy_read_header_timeout": "5s",
      "proxy_write_timeout": "10m0s",
      "proxy_idle_timeout": "1m0s"
    }
  },
  "monitor": {
    "monbasic": {
      "check_interval": "15s"
    }
  },
  "informer": {
    "disk": {
      "metric_ttl": "30s",
      "metric_type": "freespace"
    },
    "numpin": {
      "metric_ttl": "10s"
    }
  }
}
```

This new format aims to be easily extensible per component. As such,
it already surfaces quite a few new options which were hardcoded
before.

Additionally, since Go API have changed, some redundant methods have been
removed and small refactoring has happened to take advantage of the new
way.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-10-18 00:00:12 +02:00
Wyatt
598aaf2623 Undoing gx rewrites 2017-10-13 12:44:43 -04:00
Wyatt
c178afe371 Replace get requests with posts 2017-10-13 12:38:18 -04:00
dgrisham
a46ab3dda9 freespace metric partial impl 2017-08-03 11:24:19 -06:00
Hector Sanjuan
bb82c27b25 Fix #87: Implement ipfs-cluster-ctl pin ls <cid>
I have updated API endpoints to be /allocations rather than /pinlinst

It's more self-explanatory.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-04-06 21:12:16 +02:00
Hector Sanjuan
9b3449e3f8 Remove debugging leftovers in ipfshttp
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-30 15:38:41 +02:00
Hector Sanjuan
18034356fa Fix #57: Hijack /add requests and pin items in cluster after ipfs adds them.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-30 11:27:40 +02:00
Hector Sanjuan
30262a2499 Fix ipfs proxy not respecting return status codes
Added extra test for it.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-28 11:09:05 +02:00
Hector Sanjuan
2bbbea79cc Issue #49: Add disk informer
The disk informer uses "ipfs repo stat" to fetch the RepoSize value and
uses it as a metric.

The numpinalloc allocator is now a generalized ascendalloc which
sorts metrics in ascending order and return the ones with lowest
values.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-27 20:40:49 +02:00
Hector Sanjuan
4bb30cd24a Fixes #16: trigger ipfs swarm connect to other ipfs nodes in the cluster.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-27 12:42:54 +02:00
Hector Sanjuan
17796afb2e Use "pin ls --type=recursive $pin" when checking if a pin exists
This is significantly faster for pins that don't exist.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-21 12:57:09 +01:00
Hector Sanjuan
f3c1636062 Take advantage of go-multiaddr-net utilities.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-16 15:51:24 +01:00
Hector Sanjuan
03a931b8df Fix logging
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-14 17:32:00 +01:00
Hector Sanjuan
e2efef8469 go lint, go vet, put the Consensus component behind interface.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-14 16:37:29 +01:00
Hector Sanjuan
37fab27ba6 Move MapPinTracker to its own submodule and add tests for it
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
2017-03-14 16:10:45 +01:00