Hector Sanjuan
06f0cac9c0
Merge pull request #1753 from ipfs-cluster/fix/1706-fix-block-warning
...
ipfshttp: Fix "seen blocks" tracking and blockPut metrics
2022-09-09 16:32:09 +02:00
Hector Sanjuan
573d7e8916
ipfsproxy: add tests for block/put and dag/put intercepts
2022-09-09 16:27:09 +02:00
Hector Sanjuan
6ce90dfe47
ipfsproxy: intercept block/put and dag/put and pin to cluster on pin=true
...
This fixes #1738 . Tests still missing
2022-09-09 00:49:50 +02:00
Hector Sanjuan
5452b59a2e
Dependency upgrades ( #1755 )
...
* Update go-libp2p to v0.22.0
* Testing with go1.19
* build(deps): bump github.com/multiformats/go-multicodec
Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec ) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/multiformats/go-multicodec/releases )
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.5.0...v0.6.0 )
---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipld/go-car from 0.4.0 to 0.5.0
Bumps [github.com/ipld/go-car](https://github.com/ipld/go-car ) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/ipld/go-car/releases )
- [Commits](https://github.com/ipld/go-car/compare/v0.4.0...v0.5.0 )
---
updated-dependencies:
- dependency-name: github.com/ipld/go-car
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/prometheus/client_golang
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang ) from 1.12.2 to 1.13.0.
- [Release notes](https://github.com/prometheus/client_golang/releases )
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md )
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.2...v1.13.0 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/hashicorp/go-hclog from 1.2.1 to 1.3.0
Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog ) from 1.2.1 to 1.3.0.
- [Release notes](https://github.com/hashicorp/go-hclog/releases )
- [Commits](https://github.com/hashicorp/go-hclog/compare/v1.2.1...v1.3.0 )
---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/ipfs/go-ds-crdt from 0.3.6 to 0.3.7
Bumps [github.com/ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt ) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/ipfs/go-ds-crdt/releases )
- [Commits](https://github.com/ipfs/go-ds-crdt/compare/v0.3.6...v0.3.7 )
---
updated-dependencies:
- dependency-name: github.com/ipfs/go-ds-crdt
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/urfave/cli/v2 from 2.10.2 to 2.14.1
Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli ) from 2.10.2 to 2.14.1.
- [Release notes](https://github.com/urfave/cli/releases )
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md )
- [Commits](https://github.com/urfave/cli/compare/v2.10.2...v2.14.1 )
---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/libp2p/go-libp2p-http from 0.3.0 to 0.4.0
Bumps [github.com/libp2p/go-libp2p-http](https://github.com/libp2p/go-libp2p-http ) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/libp2p/go-libp2p-http/releases )
- [Commits](https://github.com/libp2p/go-libp2p-http/compare/v0.3.0...v0.4.0 )
---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-http
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/libp2p/go-libp2p-gorpc from 0.4.0 to 0.5.0
Bumps [github.com/libp2p/go-libp2p-gorpc](https://github.com/libp2p/go-libp2p-gorpc ) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/libp2p/go-libp2p-gorpc/releases )
- [Commits](https://github.com/libp2p/go-libp2p-gorpc/compare/v0.4.0...v0.5.0 )
---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-gorpc
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump contrib.go.opencensus.io/exporter/prometheus
Bumps [contrib.go.opencensus.io/exporter/prometheus](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus ) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/releases )
- [Commits](https://github.com/census-ecosystem/opencensus-go-exporter-prometheus/compare/v0.4.1...v0.4.2 )
---
updated-dependencies:
- dependency-name: contrib.go.opencensus.io/exporter/prometheus
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/libp2p/go-libp2p-raft from 0.1.8 to 0.2.0
Bumps [github.com/libp2p/go-libp2p-raft](https://github.com/libp2p/go-libp2p-raft ) from 0.1.8 to 0.2.0.
- [Release notes](https://github.com/libp2p/go-libp2p-raft/releases )
- [Commits](https://github.com/libp2p/go-libp2p-raft/compare/v0.1.8...v0.2.0 )
---
updated-dependencies:
- dependency-name: github.com/libp2p/go-libp2p-raft
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/urfave/cli from 1.22.9 to 1.22.10
Bumps [github.com/urfave/cli](https://github.com/urfave/cli ) from 1.22.9 to 1.22.10.
- [Release notes](https://github.com/urfave/cli/releases )
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md )
- [Commits](https://github.com/urfave/cli/compare/v1.22.9...v1.22.10 )
---
updated-dependencies:
- dependency-name: github.com/urfave/cli
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* Fix checker/linter/staticcheck warnings
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-06 16:57:17 +02:00
Hector Sanjuan
8c93d4cb81
Merge pull request #1754 from ipfs-cluster/struct-align-comment
...
Add comments to struct fields that must be aligned.
2022-09-06 16:23:24 +02:00
Hector Sanjuan
ed54c665b8
ipfshttp: Fix "seen blocks" tracking and blockPut metrics
...
This fixes two bugs. First, the "blockPut response CID does not match the
multihash" warning was coming up when it shouldn't. Particularly, the
multipart reader called Node() several times for the same block, resulting in
CIDs been removed from the Seen set, and causing the warning when there were
several blocks (usually the empty dir block).
This also means we were counting Blockputs (and total data added) wrong in the
metrics, double-counting some blocks as these were recorded in Node() calls.
The fix makes the tracking in Next(), which is only called once for each
block. To avoid timing issues between Block reads from the channel and
blockput responses, the Seen set now stores how many times we have seen a
block. Thus a duplicated block that will get two BlockPut responses will not
trigger a warning regardless of the time when those responses arrive.
Fixes #1706 .
2022-09-05 18:10:36 +02:00
Hector Sanjuan
a84869d3db
Add comments to struct fields that must be aligned.
2022-09-05 17:03:30 +02:00
Hector Sanjuan
71bda2d658
Merge pull request #1736 from cldy309/fix/1735-panic
...
fix panic: unaligned 64-bit atomic operation on Linux@armv7
2022-09-05 16:44:31 +02:00
chenlong348
bd3b88b933
fix panic: unaligned 64-bit atomic operation on Linux@armv7
2022-08-15 10:13:37 +08:00
Hector Sanjuan
c4d78d52f8
Merge pull request #1732 from ipfs-cluster/fix/goroutine-leak-adder
...
Fix: leaking goroutines on aborted /add requests
2022-07-08 17:40:24 +02:00
Hector Sanjuan
d19c7facff
Fix: leaking goroutines on aborted /add requests
...
It has been observed that some peers have a growing number of goroutines,
usually stuck in go-libp2p-gorpc.MultiStream() function, which is waiting to
read items from the arguments channel.
We suspect this is due to aborted /add requests. In situations when the add
request is aborted or fails, Finalize() is never called and the blocks channel
stays open, so MultiStream() can never exit, and the BlockStreamer can never
stop streaming etc.
As a fix, we added the requirement to call Close() when we stop using a
ClusterDAGService (error or not). This should ensure that the blocks channel
is always closed and not just on Finalize().
2022-07-08 17:39:59 +02:00
Hector Sanjuan
b2ce7d916d
Release v1.0.2
2022-07-06 18:26:32 +02:00
Hector Sanjuan
ff50c119e4
Merge pull request #1731 from ipfs-cluster/v1.0.2/changelog
...
Changelog for v1.0.2
2022-07-06 18:25:18 +02:00
Hector Sanjuan
a1ff94c504
Changelog for v1.0.2
2022-07-06 18:25:03 +02:00
Hector Sanjuan
f3662b8e0f
Merge pull request #1730 from ipfs-cluster/update-deps-car
...
Update go-car to v0.4.0
2022-07-06 18:08:05 +02:00
Hector Sanjuan
ad7329f602
Update go-car to v0.4.0
2022-07-06 17:49:44 +02:00
Hector Sanjuan
d2cbd8f910
Merge pull request #1729 from ipfs-cluster/fix/pins-error-negative
...
Fix: operationtracker metrics go negative
2022-07-04 20:20:13 +02:00
Hector Sanjuan
38e3c4a695
Fix: operationtracker metrics go negative
...
By substracing 1 on every cancel we are double-counting.
2022-07-04 20:09:10 +02:00
Hector Sanjuan
04177fa545
Release candidate v1.0.2-rc1
2022-06-30 14:26:54 +02:00
Hector Sanjuan
3e6577c22a
Merge pull request #1725 from ipfs-cluster/metrics-freespace
...
Metrics freespace
2022-06-23 14:19:00 +02:00
Hector Sanjuan
c454769887
Informer/disk: record issued metric weights as prometheus metric.
2022-06-23 11:58:35 +02:00
Hector Sanjuan
2aec92301d
metrics: set block/added_size unit to bytes
2022-06-23 11:58:07 +02:00
Hector Sanjuan
e8695dc6f3
informer/disk: set repoSize weight to negative
...
smaller repositories should have more priority
2022-06-23 11:49:37 +02:00
Hector Sanjuan
f7e62beee2
Merge pull request #1724 from ipfs-cluster/dependency-upgrades
...
Dependency upgrades
2022-06-23 11:42:45 +02:00
Hector Sanjuan
5ff5cb2e68
Update dependencies
2022-06-23 11:41:57 +02:00
Hector Sanjuan
12490c959a
Merge pull request #1719 from ipfs-cluster/fix/1697-commit-batches
...
crdt: Commit batches on shutdown
2022-06-23 11:03:28 +02:00
Hector Sanjuan
5e7a694cd1
crdt: Implement proper Batch commit on shutdown
...
This implements committing batches on shutdown properly.
Now the batchWorker will only finish when there are no more things queued to
be included in the final batch(es).
LogPin/Unpin operations will fail while we are shutting down and they cannot
be included in the batch.
2022-06-22 20:19:12 +02:00
Hector Sanjuan
a393ebd8d8
crdt: Commit batches on shutdown
...
This attempt to commit any pending batches when the crdt component is being
shutudown. A commit should succeed if the new DAG node is created, heads are
replaced and broadcast.
The latest version of CRDT ensure that the datastore does not unnecessarily
gets marked as dirty when a broadcasted head cannot be fetched/processed, so
the side effect of publishing a head before shutting down should be under
control at least.
2022-06-21 19:11:05 +02:00
Hector Sanjuan
d6166d802b
Merge pull request #1717 from ipfs-cluster/fix/1702-neg-metrics
...
pintracker: fix some races resulting in wrong metric counts
2022-06-20 22:29:51 +02:00
Hector Sanjuan
28c24931b6
pintracker: fix some races resulting in wrong metric counts
...
I believe this fixes the issue with some metrics like pinning going into
negative numbers occasionally. Fixes #1702 .
2022-06-20 22:16:36 +02:00
Hector Sanjuan
44c41b2342
Merge pull request #1715 from ipfs-cluster/api/improve-tls-debug-logging
...
api: improve tls setup debug logging
2022-06-20 22:06:29 +02:00
Hector Sanjuan
2ad9e7e71a
Merge pull request #1716 from ipfs-cluster/fix/1666-expire-in
...
ctl: Expiration date in "pin ls" was showing wrong field (timestamp)
2022-06-20 22:05:17 +02:00
Hector Sanjuan
90a3b63d82
ctl: Expiration date in "pin ls" was showing wrong field (timestamp)
...
Fixes #1666 .
2022-06-20 21:36:42 +02:00
Hector Sanjuan
9f57f7280f
api: improve tls setup debug logging
...
Currently the debug messages are a bit confusing.
2022-06-20 21:30:54 +02:00
Hector Sanjuan
57a49fc36f
tls has also been consolidated on go-libp2p ( #1712 )
...
* Depedencies: tls security has also been consolidated on go-libp2p
* build(deps): bump github.com/urfave/cli/v2 from 2.5.1 to 2.10.1
Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli ) from 2.5.1 to 2.10.1.
- [Release notes](https://github.com/urfave/cli/releases )
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md )
- [Commits](https://github.com/urfave/cli/compare/v2.5.1...v2.10.1 )
---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/multiformats/go-multicodec
Bumps [github.com/multiformats/go-multicodec](https://github.com/multiformats/go-multicodec ) from 0.4.1 to 0.5.0.
- [Release notes](https://github.com/multiformats/go-multicodec/releases )
- [Commits](https://github.com/multiformats/go-multicodec/compare/v0.4.1...v0.5.0 )
---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multicodec
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/prometheus/client_golang
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang ) from 1.12.1 to 1.12.2.
- [Release notes](https://github.com/prometheus/client_golang/releases )
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md )
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.1...v1.12.2 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/hashicorp/go-hclog from 1.2.0 to 1.2.1
Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog ) from 1.2.0 to 1.2.1.
- [Release notes](https://github.com/hashicorp/go-hclog/releases )
- [Commits](https://github.com/hashicorp/go-hclog/compare/v1.2.0...v1.2.1 )
---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-hclog
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/urfave/cli/v2 from 2.10.1 to 2.10.2
Bumps [github.com/urfave/cli/v2](https://github.com/urfave/cli ) from 2.10.1 to 2.10.2.
- [Release notes](https://github.com/urfave/cli/releases )
- [Changelog](https://github.com/urfave/cli/blob/main/docs/CHANGELOG.md )
- [Commits](https://github.com/urfave/cli/compare/v2.10.1...v2.10.2 )
---
updated-dependencies:
- dependency-name: github.com/urfave/cli/v2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-20 21:14:45 +02:00
Hector Sanjuan
bc54fd60a6
Merge pull request #1705 from ipfs-cluster/pintracker-wrong-status
...
pintracker: fix status objects missing or having wrong fields
2022-06-20 21:12:20 +02:00
Hector Sanjuan
177af57361
Merge pull request #1703 from ipfs-cluster/pinning-api-auth
...
API: Support JWT bearer token authorization
2022-06-20 21:12:06 +02:00
Hector Sanjuan
d7da1b6044
API: Support JWT bearer token authorization
...
The Pinning Services API standard mandates Bearer token authentication.
This adds JWT bearer token authentication to the IPFS Cluster REST and PINSVC
APIs.
The basic_auth_credentials configuration option needs to be not null and have
at least one username/passwords entry.
A user authenticated via Basic Auth can then "POST /token" and obtain a json
object:
```json { "token" : "<JWTtoken>" } ```
The JWT token has the "iss" (issuer) field set to the Basic auth user that
authorized its creation and is HMAC-signed with its password.
When basic-auth-credentials are set, the APIs will verify that requests come
with either Basic Auth authorization header or with a Bearer token
authorization header.
Bearer tokens will be decoded and the signature will be verified against the
password of the issuer.
At the moment we provide no support to revoke tokens, set "expiration date",
"not before" etc, but this may come in the future.
2022-06-20 20:04:39 +02:00
Hector Sanjuan
f4a445e019
pintracker: fix status objects missing or having wrong fields
...
The operation tracker was not setting some fields correctly when producing
PinInfo objects. Additionally, recover operations were submitted with empty
pin objects, which resulted in the status for pins sent on recover operations
to be missing fields.
2022-06-20 20:03:34 +02:00
Hector Sanjuan
c9120aa672
Merge pull request #1704 from ipfs-cluster/pinsvc-compliance
...
PinSVC API: fix bugs and increase compliance
2022-06-20 18:50:16 +02:00
Hector Sanjuan
ff488e1fec
pinsvc: do not set error descriptions
...
It is duplicated with "reason".
2022-06-20 18:49:23 +02:00
Hector Sanjuan
49e770ce8c
Dependency upgrades ( #1711 )
...
Updates to libp2p and go-cid.
Cluster now conforms with go-ipfs 0.13.0 changes to block/put.
2022-06-20 18:23:50 +02:00
Hector Sanjuan
bb8e8725e8
PinSVC API: fix bugs and increase compliance
...
These changes fix a few bugs in the PinSVC API and have been discovered by
running the compliance tool at https://github.com/ipfs-shipyard/pinning-service-compliance .
In particular, the error objects did not respect the spec, filters and counts
were not done right. An additional PR will follow with fixes to the pintracker
because some pin status information was not correctly set.
2022-06-17 17:06:37 +02:00
Hector Sanjuan
57c3b180b5
Additional fixes for security warnings from CodeQL
2022-06-16 17:44:34 +02:00
Hector Sanjuan
d23240f6c8
CodeQL analysis: fix some security warnings
...
They are related to logging user-input.
2022-06-16 17:44:34 +02:00
Hector Sanjuan
b9a4578f0b
Create codeql-analysis.yml
2022-06-16 17:44:34 +02:00
Hector Sanjuan
71bbf4ab74
Adder: update error message when doing Gets on the DAGService.
...
Indicate that this is not expected and is likely a bug.
2022-06-16 17:43:30 +02:00
Hector Sanjuan
455aa6def1
Fix #1691 : Error when adding directories with many files
...
Fixes #1691 by updating to the latest go-unixfs and adding a test.
The test is verified to fail on the previous go-unixfs version.
2022-06-16 17:43:30 +02:00
Hector Sanjuan
144c66359e
Test module: allow the MockDAGService to be write-only.
...
The adder needs a write-only DAGService.
2022-06-16 17:43:30 +02:00
Hector Sanjuan
e0247e4105
README: more improvements to badges
2022-06-16 17:43:30 +02:00