cascade/ipfs-cluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,781 Commits 3 Branches 70 Tags 15 MiB
3e6577c22a
Commit Graph

2781 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Hector Sanjuan
3e6577c22a
Merge pull request #1725 from ipfs-cluster/metrics-freespace 
Metrics freespace
 2022-06-23 14:19:00 +02:00
Hector Sanjuan
c454769887 Informer/disk: record issued metric weights as prometheus metric. 2022-06-23 11:58:35 +02:00
Hector Sanjuan
2aec92301d metrics: set block/added_size unit to bytes 2022-06-23 11:58:07 +02:00
Hector Sanjuan
e8695dc6f3 informer/disk: set repoSize weight to negative 
smaller repositories should have more priority
 2022-06-23 11:49:37 +02:00
Hector Sanjuan
f7e62beee2
Merge pull request #1724 from ipfs-cluster/dependency-upgrades 
Dependency upgrades
 2022-06-23 11:42:45 +02:00
Hector Sanjuan
5ff5cb2e68 Update dependencies 2022-06-23 11:41:57 +02:00
Hector Sanjuan
12490c959a
Merge pull request #1719 from ipfs-cluster/fix/1697-commit-batches 
crdt: Commit batches on shutdown
 2022-06-23 11:03:28 +02:00
Hector Sanjuan
5e7a694cd1 crdt: Implement proper Batch commit on shutdown 
This implements committing batches on shutdown properly.

Now the batchWorker will only finish when there are no more things queued to
be included in the final batch(es).

LogPin/Unpin operations will fail while we are shutting down and they cannot
be included in the batch.
 2022-06-22 20:19:12 +02:00
Hector Sanjuan
a393ebd8d8 crdt: Commit batches on shutdown 
This attempt to commit any pending batches when the crdt component is being
shutudown. A commit should succeed if the new DAG node is created, heads are
replaced and broadcast.

The latest version of CRDT ensure that the datastore does not unnecessarily
gets marked as dirty when a broadcasted head cannot be fetched/processed, so
the side effect of publishing a head before shutting down should be under
control at least.
 2022-06-21 19:11:05 +02:00
Hector Sanjuan
d6166d802b
Merge pull request #1717 from ipfs-cluster/fix/1702-neg-metrics 
pintracker: fix some races resulting in wrong metric counts
 2022-06-20 22:29:51 +02:00
Hector Sanjuan
28c24931b6 pintracker: fix some races resulting in wrong metric counts 
I believe this fixes the issue with some metrics like pinning going into
negative numbers occasionally. Fixes #1702.
 2022-06-20 22:16:36 +02:00
Hector Sanjuan
44c41b2342
Merge pull request #1715 from ipfs-cluster/api/improve-tls-debug-logging 
api: improve tls setup debug logging
 2022-06-20 22:06:29 +02:00
Hector Sanjuan
2ad9e7e71a
Merge pull request #1716 from ipfs-cluster/fix/1666-expire-in 
ctl: Expiration date in "pin ls" was showing wrong field (timestamp)
 2022-06-20 22:05:17 +02:00
Hector Sanjuan
90a3b63d82 ctl: Expiration date in "pin ls" was showing wrong field (timestamp) 
Fixes #1666.
 2022-06-20 21:36:42 +02:00
Hector Sanjuan
9f57f7280f api: improve tls setup debug logging 
Currently the debug messages are a bit confusing.
 2022-06-20 21:30:54 +02:00
Hector Sanjuan
57a49fc36f
tls has also been consolidated on go-libp2p (#1712) 
* Depedencies: tls security has also been consolidated on go-libp2p

* build(deps): bump github.com/urfave/cli/v2 from 2.5.1 to 2.10.1

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.5.1 to 2.10.1.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.5.1...v2.10.1)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/multiformats/go-multicodec

Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec) from 0.4.1 to 0.5.0.
- [Release notes](https://github.com/multiformats/go-multicodec/releases)
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.4.1...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/prometheus/client_golang

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.1 to 1.12.2.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.1...v1.12.2)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/hashicorp/go-hclog from 1.2.0 to 1.2.1

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 1.2.0 to 1.2.1.
- [Release notes](https://github.com/hashicorp/go-hclog/releases)
- [Commits](https://github.com/hashicorp/go-hclog/compare/v1.2.0...v1.2.1)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump github.com/urfave/cli/v2 from 2.10.1 to 2.10.2

Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli) from 2.10.1 to 2.10.2.
- [Release notes](https://github.com/urfave/cli/releases)
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/urfave/cli/compare/v2.10.1...v2.10.2)

---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-06-20 21:14:45 +02:00
Hector Sanjuan
bc54fd60a6
Merge pull request #1705 from ipfs-cluster/pintracker-wrong-status 
pintracker: fix status objects missing or having wrong fields
 2022-06-20 21:12:20 +02:00
Hector Sanjuan
177af57361
Merge pull request #1703 from ipfs-cluster/pinning-api-auth 
API: Support JWT bearer token authorization
 2022-06-20 21:12:06 +02:00
Hector Sanjuan
d7da1b6044 API: Support JWT bearer token authorization 
The Pinning Services API standard mandates Bearer token authentication.

This adds JWT bearer token authentication to the IPFS Cluster REST and PINSVC
APIs.

The basic_auth_credentials configuration option needs to be not null and have
at least one username/passwords entry.

A user authenticated via Basic Auth can then "POST /token" and obtain a json
object:

```json { "token" : "<JWTtoken>" } ```

The JWT token has the "iss" (issuer) field set to the Basic auth user that
authorized its creation and is HMAC-signed with its password.

When basic-auth-credentials are set, the APIs will verify that requests come
with either Basic Auth authorization header or with a Bearer token
authorization header.

Bearer tokens will be decoded and the signature will be verified against the
password of the issuer.

At the moment we provide no support to revoke tokens, set "expiration date",
"not before" etc, but this may come in the future.
 2022-06-20 20:04:39 +02:00
Hector Sanjuan
f4a445e019 pintracker: fix status objects missing or having wrong fields 
The operation tracker was not setting some fields correctly when producing
PinInfo objects. Additionally, recover operations were submitted with empty
pin objects, which resulted in the status for pins sent on recover operations
to be missing fields.
 2022-06-20 20:03:34 +02:00
Hector Sanjuan
c9120aa672
Merge pull request #1704 from ipfs-cluster/pinsvc-compliance 
PinSVC API: fix bugs and increase compliance
 2022-06-20 18:50:16 +02:00
Hector Sanjuan
ff488e1fec pinsvc: do not set error descriptions 
It is duplicated with "reason".
 2022-06-20 18:49:23 +02:00
Hector Sanjuan
49e770ce8c
Dependency upgrades (#1711) 
Updates to libp2p and go-cid.

Cluster now conforms with go-ipfs 0.13.0 changes to block/put.
 2022-06-20 18:23:50 +02:00
Hector Sanjuan
bb8e8725e8 PinSVC API: fix bugs and increase compliance 
These changes fix a few bugs in the PinSVC API and have been discovered by
running the compliance tool at https://github.com/ipfs-shipyard/pinning-service-compliance.

In particular, the error objects did not respect the spec, filters and counts
were not done right. An additional PR will follow with fixes to the pintracker
because some pin status information was not correctly set.
 2022-06-17 17:06:37 +02:00
Hector Sanjuan
57c3b180b5 Additional fixes for security warnings from CodeQL 2022-06-16 17:44:34 +02:00
Hector Sanjuan
d23240f6c8 CodeQL analysis: fix some security warnings 
They are related to logging user-input.
 2022-06-16 17:44:34 +02:00
Hector Sanjuan
b9a4578f0b Create codeql-analysis.yml 2022-06-16 17:44:34 +02:00
Hector Sanjuan
71bbf4ab74 Adder: update error message when doing Gets on the DAGService. 
Indicate that this is not expected and is likely a bug.
 2022-06-16 17:43:30 +02:00
Hector Sanjuan
455aa6def1 Fix #1691: Error when adding directories with many files 
Fixes #1691 by updating to the latest go-unixfs and adding a test.

The test is verified to fail on the previous go-unixfs version.
 2022-06-16 17:43:30 +02:00
Hector Sanjuan
144c66359e Test module: allow the MockDAGService to be write-only. 
The adder needs a write-only DAGService.
 2022-06-16 17:43:30 +02:00
Hector Sanjuan
e0247e4105 README: more improvements to badges 2022-06-16 17:43:30 +02:00
Hector Sanjuan
b705212ac0 Improve badges in README 2022-06-16 17:43:30 +02:00
Hector Sanjuan
bc8b65a099 Rename Cancelled to Canceled in operationtracker 
Part of fixing misspellings.
 2022-06-16 17:43:30 +02:00
Hector Sanjuan
9bd611e7c3 Error on mispellings 2022-06-16 17:43:30 +02:00
Hector Sanjuan
755cebbe0d Enable spell checking and fix spelling errors (using US locale) 2022-06-16 17:43:30 +02:00
Hector Sanjuan
6260b11e8c Use Codecov@v2 github action 2022-06-16 17:43:30 +02:00
Hector Sanjuan
6723acd333 Fix badge image link in readme 2022-06-16 17:43:30 +02:00
Hector Sanjuan
d7107791ed Rename links to use the ipfscluster.io domain 2022-06-16 17:43:30 +02:00
Hector Sanjuan
508791b547 Migrate from ipfs/ipfs-cluster to ipfs-cluster/ipfs-cluster 
This performs the necessary renamings.
 2022-06-16 17:43:30 +02:00
ipfs-mgmt-read-write[bot]
34396ddb2c Add .github/workflows/stale.yml 2022-06-16 17:43:30 +02:00
Sungyub NA
fbdeebc71e Update comment at docker-compose 
License: MIT
Signed-off-by: Sungyub NA <darkerkorean@gmail.com>
 2022-06-16 17:43:30 +02:00
@RubenKelevra
3b51cc39f4 improve language; fix typos; move breaking change under own headline 2022-06-16 17:43:30 +02:00
Hector Sanjuan
3911f8483a Release v1.0.1 2022-06-16 17:43:30 +02:00
Hector Sanjuan
5a3d36b41a Changelog for v1.0.1 2022-06-16 17:43:30 +02:00
Hector Sanjuan
90df0ba35e GetMetadata: ignore linter check 2022-06-16 17:43:29 +02:00
Hector Sanjuan
7a05eeeb60 Fix #1547: Store pin metadata in a sorted array 
This deprecates the Metadata protobuf map and starts serializing metadata as an
array that is always sorted by the metadata key.

This should resolve the issue that an state export file can be imported
resulting in two different CRDT dags because the binary representation of the
pin can arbitrarily change depending on how the keys in the map are listed,
and thus the CID of a delta is impacted.

So with this, a state export should result in exactly the same DAG regardless
of where it is imported.
 2022-06-16 17:43:29 +02:00
Hector Sanjuan
03740e5bf0 ctl: fix "pin ls" hangs for pinsets > 1024 items. 
Seems we forgot to adapt "pin ls" and run it in a background goroutine, so it
just hangs.
 2022-06-16 17:43:29 +02:00
Hector Sanjuan
447b79093c Fix #1655: rest api lbclients panic on error with streaming methods 
I think this should fix the issue. As solution we make every retry with a
temporary channel and copy results to the final channel which is only closed
by us. This only affects streaming methods.
 2022-06-16 17:43:29 +02:00
Hector Sanjuan
bc6fe29d8b Update to go1.18 2022-06-16 17:43:29 +02:00
Hector Sanjuan
72050c1e59 Stay in go1.17 2022-06-16 17:43:29 +02:00