cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,892 Commits 3 Branches 70 Tags 15 MiB
7c923c32e2
Commit Graph

33 Commits

Author SHA1 Message Date
Hector Sanjuan
ba2308765f Avoid using rand.Seed as it is deprecated 2023-08-10 18:33:30 +02:00
Hector Sanjuan
50015947da
api.go: update comment on auth pass-through 2023-05-11 12:11:57 +02:00
Hector Sanjuan
287f285d79
api.go: only pass through GET requests to /health 2023-05-11 12:11:21 +02:00
Hector Sanjuan
9cb2219636
Add end of line to api.go 2023-05-11 12:10:50 +02:00
Arthur Gavazza
093d234da9
Merge branch 'master' into feat/add-authless-healthcheck 2023-04-20 08:54:13 -03:00
arthurgavazza
ef23442cab feat: add health endpoint 2023-04-20 08:18:38 -03:00
Gus Eggert
c115d29bce chore: upgrade to boxo 2023-04-19 12:59:07 +02:00
Hector Sanjuan
ddd51b08ca
api: fix typo in error message (#1882) 2023-03-06 18:57:24 +01:00
Hector Sanjuan
2286ee73f8 api: return errors on stream response requests with 0 items 
This fixes a bug in API code that made it return 204-No content when the RPC
methods failed with an error before any items were returned on the channel.
 2022-09-15 16:40:34 +02:00
Hector Sanjuan
5452b59a2e
Dependency upgrades (#1755) 
* Update go-libp2p to v0.22.0

* Testing with go1.19

* build(deps): bump github.com/multiformats/go-multicodec

Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/multiformats/go-multicodec/releases)
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipld/go-car from 0.4.0 to 0.5.0

Bumps [github.com/ipld/go-car](https://github.com/ipld/go-car) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/ipld/go-car/releases)
- [Commits](https://github.com/ipld/go-car/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/ipld/go-car
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/prometheus/client_golang

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.2 to 1.13.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.2...v1.13.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/hashicorp/go-hclog from 1.2.1 to 1.3.0

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 1.2.1 to 1.3.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v1.2.1...v1.3.0)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/ipfs/go-ds-crdt from 0.3.6 to 0.3.7

Bumps [github.com/ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/ipfs/go-ds-crdt/releases)
- [Commits](https://github.com/ipfs/go-ds-crdt/compare/v0.3.6...v0.3.7)

---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-crdt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli/v2 from 2.10.2 to 2.14.1

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.10.2 to 2.14.1.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.10.2...v2.14.1)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-http from 0.3.0 to 0.4.0

Bumps [github.com/libp2p/go-libp2p-http](https://github.com/libp2p/go-libp2p-http) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/libp2p/go-libp2p-http/releases)
- [Commits](https://github.com/libp2p/go-libp2p-http/compare/v0.3.0...v0.4.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-gorpc from 0.4.0 to 0.5.0

Bumps [github.com/libp2p/go-libp2p-gorpc](https://github.com/libp2p/go-libp2p-gorpc) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/libp2p/go-libp2p-gorpc/releases)
- [Commits](https://github.com/libp2p/go-libp2p-gorpc/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-gorpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump contrib.go.opencensus.io/exporter/prometheus

Bumps [contrib.go.opencensus.io/exporter/prometheus](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/releases)
- [Commits](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/compare/v0.4.1...v0.4.2)

---
updated-dependencies:
- dependency-name: contrib.go.opencensus.io/exporter/prometheus
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/libp2p/go-libp2p-raft from 0.1.8 to 0.2.0

Bumps [github.com/libp2p/go-libp2p-raft](https://github.com/libp2p/go-libp2p-raft) from 0.1.8 to 0.2.0.
- [Release notes](https://github.com/libp2p/go-libp2p-raft/releases)
- [Commits](https://github.com/libp2p/go-libp2p-raft/compare/v0.1.8...v0.2.0)

---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-raft
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli from 1.22.9 to 1.22.10

Bumps [github.com/urfave/cli](https://github.com/urfave/cli) from 1.22.9 to 1.22.10.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v1.22.9...v1.22.10)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix checker/linter/staticcheck warnings

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-09-06 16:57:17 +02:00
Hector Sanjuan
9f57f7280f api: improve tls setup debug logging 
Currently the debug messages are a bit confusing.
 2022-06-20 21:30:54 +02:00
Hector Sanjuan
57a49fc36f
tls has also been consolidated on go-libp2p (#1712) 
* Depedencies: tls security has also been consolidated on go-libp2p

* build(deps): bump github.com/urfave/cli/v2 from 2.5.1 to 2.10.1

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.5.1 to 2.10.1.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.5.1...v2.10.1)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/multiformats/go-multicodec

Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec) from 0.4.1 to 0.5.0.
- [Release notes](https://github.com/multiformats/go-multicodec/releases)
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.4.1...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/prometheus/client_golang

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.1 to 1.12.2.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.1...v1.12.2)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/hashicorp/go-hclog from 1.2.0 to 1.2.1

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 1.2.0 to 1.2.1.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v1.2.0...v1.2.1)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli/v2 from 2.10.1 to 2.10.2

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.10.1 to 2.10.2.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.10.1...v2.10.2)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-06-20 21:14:45 +02:00
Hector Sanjuan
d7da1b6044 API: Support JWT bearer token authorization 
The Pinning Services API standard mandates Bearer token authentication.

This adds JWT bearer token authentication to the IPFS Cluster REST and PINSVC
APIs.

The basic_auth_credentials configuration option needs to be not null and have
at least one username/passwords entry.

A user authenticated via Basic Auth can then "POST /token" and obtain a json
object:

```json { "token" : "<JWTtoken>" } ```

The JWT token has the "iss" (issuer) field set to the Basic auth user that
authorized its creation and is HMAC-signed with its password.

When basic-auth-credentials are set, the APIs will verify that requests come
with either Basic Auth authorization header or with a Bearer token
authorization header.

Bearer tokens will be decoded and the signature will be verified against the
password of the issuer.

At the moment we provide no support to revoke tokens, set "expiration date",
"not before" etc, but this may come in the future.
 2022-06-20 20:04:39 +02:00
Hector Sanjuan
49e770ce8c
Dependency upgrades (#1711) 
Updates to libp2p and go-cid.

Cluster now conforms with go-ipfs 0.13.0 changes to block/put.
 2022-06-20 18:23:50 +02:00
Hector Sanjuan
755cebbe0d Enable spell checking and fix spelling errors (using US locale) 2022-06-16 17:43:30 +02:00
Hector Sanjuan
508791b547 Migrate from ipfs/ipfs-cluster to ipfs-cluster/ipfs-cluster 
This performs the necessary renamings.
 2022-06-16 17:43:30 +02:00
Hector Sanjuan
e2403c682f
Merge pull request #1626 from ipfs/feat/cid-strings 
Adopt api.Cid type - replaces cid.Cid everwhere.
 2022-04-07 15:05:00 +02:00
Hector Sanjuan
a97ed10d0b Adopt api.Cid type - replaces cid.Cid everwhere. 
This commit introduces an api.Cid type and replaces the usage of cid.Cid
everywhere.

The main motivation here is to override MarshalJSON so that Cids are
JSON-ified as '"Qm...."' instead of '{ "/": "Qm....." }', as this "ipld"
representation of IDs is horrible to work with, and our APIs are not issuing
IPLD objects to start with.

Unfortunately, there is no way to do this cleanly, and the best way is to just
switch everything to our own type.
 2022-04-07 14:27:39 +02:00
Hector Sanjuan
d1ab0bd0ee Adjust test checking for trailer errors 2022-04-04 12:45:34 +02:00
Hector Sanjuan
787aea5f31 Always include the X-Stream-Error trailer (even without errors) 
This is to potentially address things like this:

https://github.com/nodejs/undici/issues/432#issuecomment-1047931107
 2022-04-04 12:09:54 +02:00
Hector Sanjuan
0d73d33ef5 Pintracker: streaming methods 
This commit continues the work of taking advantage of the streaming
capabilities in go-libp2p-gorpc by improving the ipfsconnector and pintracker
components.

StatusAll and RecoverAll methods are now streaming methods, with the REST API
output changing accordingly to produce a stream of GlobalPinInfos rather than
a json array.

pin/ls request to the ipfs daemon now use ?stream=true and avoid having to
load the full pinset map on memory. StatusAllLocal and RecoverAllLocal
requests to the pin tracker stream all the way and no longer store the full
pinset, and the full PinInfo status slice before sending it out.

We have additionally switched to a pattern where streaming methods receive the
channel as an argument, allowing the caller to decide on whether to launch a
goroutine, do buffering etc.
 2022-03-22 15:38:01 +01:00
Hector Sanjuan
9b9d76f92d Pinset streaming and method type revamp 
This commit introduces the new go-libp2p-gorpc streaming capabilities for
Cluster. The main aim is to work towards heavily reducing memory usage when
working with very large pinsets.

As a side-effect, it takes the chance to revampt all types for all public
methods so that pointers to static what should be static objects are not used
anymore. This should heavily reduce heap allocations and GC activity.

The main change is that state.List now returns a channel from which to read
the pins, rather than pins being all loaded into a huge slice.

Things reading pins have been all updated to iterate on the channel rather
than on the slice. The full pinset is no longer fully loaded onto memory for
things that run regularly like StateSync().

Additionally, the /allocations endpoint of the rest API no longer returns an
array of pins, but rather streams json-encoded pin objects directly. This
change has extended to the restapi client (which puts pins into a channel as
they arrive) and to ipfs-cluster-ctl.

There are still pending improvements like StatusAll() calls which should also
stream responses, and specially BlockPut calls which should stream blocks
directly into IPFS on a single call.

These are coming up in future commits.
 2022-03-19 03:02:55 +01:00
Hector Sanjuan
fbc69ee3c6 pinsvcapi: fix several API test failures 2022-03-11 16:18:08 +01:00
Hector Sanjuan
583011c3a4 Fix panic in common api tests 2022-02-02 00:52:39 +01:00
Hector Sanjuan
ea624d50e4 common api: automatically set NotFound errors 2022-02-02 00:44:27 +01:00
Hector Sanjuan
a5eb3f2db0 api: Support a custom error function for custom error messages 2022-02-02 00:43:00 +01:00
Hector Sanjuan
c71e42f5ad API: update basic-auth tests to allow OPTIONS 2021-12-22 14:06:29 +01:00
Hector Sanjuan
bba410af32 API: OPTIONS requests should bypass authentication 
They need to be handled directly by the CORS handler.

Fixes #1512
 2021-12-22 13:26:03 +01:00
Hector Sanjuan
7e85d90bae misc: re-order some imports 2021-12-17 11:47:50 +01:00
Hector Sanjuan
4739ed9210 Changes pertaining to go-libp2p v0.16.0 2021-11-30 06:25:15 +01:00
Hector Sanjuan
fdc3abcb23 api: use default transports for the libp2p host. 2021-11-30 06:06:10 +01:00
Hector Sanjuan
32386d853a Dependency upgrades 2021-10-20 16:56:24 +02:00
Hector Sanjuan
63972f2b2e API: Refactor REST API. Extract all functionality. 
This is a preparatory PR to add additional APIs (Pinning Service API) easily
to cluster.

Instead of copy-pasting most of what the REST API does, I have refactored so
that the whole configuration, routing and request-handling utilities can be
re-used.

The worst part has been to divide the test between tests that test core
(common.API) functionality and tests that test specific REST API endpoint
functionality. I could not get away without an additional common/test package
to provide functions that are used from both places. This is a side effect of
testing both http and libp2p endpoints for every request etc.
 2021-09-16 15:52:25 +02:00