james/garage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,925 Commits 1 Branch 25 Tags 42 MiB
3844110cd0
Commit Graph

272 Commits

Author SHA1 Message Date
Alex Auvolat
93114a9747 block manager: refactoring 2023-09-06 16:35:28 +02:00
Alex Auvolat
71c0188055 block manager: skeleton for multi-hdd support 2023-09-06 16:35:28 +02:00
Alex Auvolat
4b4f2000f4 lifecycle: fix SkipBucket bug 2023-09-06 16:34:07 +02:00
Alex Auvolat
51eac97260 update version to 0.8.4 2023-09-05 23:28:12 +02:00
Alex Auvolat
8e0c020bb9 lifecycle worker: correct small clippy lints 2023-09-04 11:33:44 +02:00
Alex Auvolat
1cdc321e28 lifecycle worker: don't get stuck on non-existent bucket 2023-08-31 11:36:30 +02:00
Alex Auvolat
f579d6d9b4 lifecycle worker: fix potential inifinite loop 2023-08-31 11:29:54 +02:00
Alex Auvolat
a00a52633f lifecycle worker: add log message when starting 2023-08-31 11:25:14 +02:00
Alex Auvolat
adbf5925de lifecycle worker: use queue_insert and process objects in batches 2023-08-31 11:19:26 +02:00
Alex Auvolat
1cfcc61de8 lifecycle worker: mitigate potential bugs + refactoring 2023-08-31 00:28:37 +02:00
Alex Auvolat
b2f679675e lifecycle worker: take into account disabled rules 2023-08-30 23:52:09 +02:00
Alex Auvolat
01c327a07a lifecycle worker: avoid building chrono's serde feature 2023-08-30 23:46:15 +02:00
Alex Auvolat
a1d57283c0 bucket_table: bucketparams::new doesn't need to be pub 2023-08-30 20:07:14 +02:00
Alex Auvolat
d2e94e36d6 lifecycle config: add missing line in merge() and remove tracing 2023-08-30 20:05:53 +02:00
Alex Auvolat
75ccc5a95c lifecycle config: store date as given, try to debug 2023-08-30 20:02:07 +02:00
Alex Auvolat
7200954318 lifecycle worker: add logging 2023-08-30 14:54:52 +02:00
Alex Auvolat
0f1849e1ac lifecycle worker: launch with the rest of Garage 2023-08-30 14:51:08 +02:00
Alex Auvolat
da8b224e24 lifecycle worker: skip entire bucket when no lifecycle config is set 2023-08-30 14:38:19 +02:00
Alex Auvolat
2996dc875f lifecycle worker: implement main functionality 2023-08-30 14:29:03 +02:00
Alex Auvolat
a2e0e34db5 lifecycle: skeleton for lifecycle worker 2023-08-30 12:41:11 +02:00
Alex Auvolat
f7b409f114 use a NaiveDate in data model, it serializes to string (iso 8601 format) 2023-08-30 11:24:01 +02:00
Alex Auvolat
abf011c290 lifecycle: implement validation into garage's internal data structure 2023-08-29 18:22:03 +02:00
Alex Auvolat
0b83e0558e bucket_table: data model for lifecycle configuration 2023-08-29 17:00:41 +02:00
Alex Auvolat
2e90e1c124 Merge branch 'main' into next 2023-08-29 11:32:42 +02:00
Alex Auvolat
cece1be1bb bump version to 0.8.3 2023-08-28 13:17:26 +02:00
Alex Auvolat
8ef42c9609 admin docs: reformatting, key admin: add check 2023-06-14 17:19:25 +02:00
Alex Auvolat
90b2d43eb4 Merge branch 'main' into next 2023-06-13 17:14:11 +02:00
Alex Auvolat
942c1f1bfe multipart uploads: save timestamp 2023-06-13 10:48:22 +02:00
Alex Auvolat
3d477906d4 properly delete multipart uploads after completion 2023-06-09 17:13:27 +02:00
Alex Auvolat
412ab77b08 comments and clippy lint fixes 2023-06-09 16:23:37 +02:00
Alex Auvolat
511e07ecd4 fix mpu counter (add missing workers) and report info at appropriate places 2023-06-09 16:23:37 +02:00
Alex Auvolat
82e75c0e29 Adapt S3 API code to use new multipart upload models 
- Create and PutPart
- completemultipartupload
- upload part copy
- list_parts
 2023-06-09 16:23:37 +02:00
Alex Auvolat
38d6ac4295 New multipart upload table layout 2023-06-09 16:23:37 +02:00
Alex Auvolat
e7e164a280 Make fsync an option for meta and data 2023-06-09 16:23:21 +02:00
Alex Auvolat
19639705e6 Mark sled as deprecated, make lmdb default, and improve sqlite and lmdb defaults 2023-05-17 14:30:53 +02:00
Jonathan Davies
c783194e8b *: apply clippy recommendations. 2023-05-09 20:49:34 +01:00
Jakub Jirutka
d2deee0b8b Declare garage crates using workspace.dependencies 
This will allow to really disable "sled" feature without declaring
`default-features = false` in every Cargo.toml where garage_db and
garage_model is used.

See https://doc.rust-lang.org/cargo/reference/workspaces.html#the-dependencies-table
 2023-05-09 08:46:15 +00:00
Alex Auvolat
000006d689 obsolete clippy lints 2023-03-13 18:50:07 +01:00
Alex Auvolat
0a1ddcf630 Prepare for v0.8.2 2023-03-13 18:46:31 +01:00
Alex Auvolat
b8123fb6cd Clearer error message when LMDB has oom error (fix #517) 2023-03-06 11:38:49 +01:00
Alex Auvolat
656b8d42de secrets can be passed directly in config, as file, or as env 2023-02-03 15:27:39 +01:00
Alex Auvolat
c56794655e Fix fmt 2023-01-26 17:27:03 +01:00
Alex Auvolat
8e93d69974 More clippy fixes 2023-01-26 17:26:32 +01:00
Alex Auvolat
590a0a8450 Merge branch 'main' into k2v-watch-range-2 2023-01-26 16:46:40 +01:00
Jonathan Davies
20c1cdf662 Cargo.toml: Loosen tracing dependency to just 0.1. 2023-01-26 11:13:11 +00:00
Jonathan Davies
f952e37ba7 {model,util}/Cargo.toml: Updated blake2 from 0.9 to 0.10. 2023-01-26 11:13:11 +00:00
Jonathan Davies
36944f1839 Cargo.toml: Updated base64 from 0.13 to 0.21. 2023-01-26 11:13:07 +00:00
Jonathan Davies
5c3075fe01 Cargo.toml: Updated zstd from 0.9 to 0.12. 2023-01-23 18:08:14 +00:00
Alex Auvolat
dac254a6e7
Merge branch 'main' into k2v-watch-range-2 2023-01-11 17:09:37 +01:00
Alex Auvolat
638c5a3ce0
PollRange: add extra RPC delay after quorum is achieved, 
to give a chance to the 3rd node to respond
 2023-01-11 16:12:07 +01:00
Alex Auvolat
5b5ca63cf6
Poll cleanup 2023-01-11 15:17:27 +01:00
Alex Auvolat
bba13f40fc
Correctly return bad requests when seeh marker is invalid 2023-01-11 12:27:19 +01:00
Alex Auvolat
ba384e61c0
PollRange: return immediately if no seen marker is provided 2023-01-11 12:03:17 +01:00
Alex Auvolat
09a3dad0f2
Lock once for insert_many 2023-01-11 11:35:36 +01:00
Alex Auvolat
b83517d521
Implement PollRange API endpoint 2023-01-10 15:22:25 +01:00
Alex Auvolat
43fd6c1526
PollRange RPC 2023-01-10 12:54:24 +01:00
Alex Auvolat
789540ca37
Type definition for range seen marker 2023-01-10 11:59:57 +01:00
Alex Auvolat
9f5419f465
Make K2V item timestamps globally increasing on each node 2023-01-10 11:03:52 +01:00
Alex Auvolat
a48e2e0cb2
K2V: Subscription to ranges of items 2023-01-10 10:30:59 +01:00
Felix Scheinost
f2106c2733 Implement rpc_secret_file 2023-01-04 18:35:10 +01:00
Alex Auvolat
f3f27293df
Uniform framework for bg variable management 2023-01-04 13:07:13 +01:00
Alex Auvolat
8d5505514f
Make it explicit when using nonversioned encoding 2023-01-03 15:27:36 +01:00
Alex Auvolat
426d8784da
cleanup 2023-01-03 15:08:37 +01:00
Alex Auvolat
cdb2a591e9
Refactor how things are migrated 2023-01-03 14:44:47 +01:00
Alex Auvolat
939a6d67e8
Merge branch 'main' into internals-rework 2023-01-02 15:07:44 +01:00
Alex Auvolat
6775569525
Bump everything to v0.8.1 2023-01-02 14:15:33 +01:00
Alex Auvolat
d56c472712
Refactor background runner and get rid of job worker 2022-12-14 12:51:42 +01:00
Alex Auvolat
2183518edc
Spawn all background workers in a separate step 2022-12-14 12:28:07 +01:00
Alex Auvolat
83c8467e23
Proper queueing for delayed inserts, now backed to disk 2022-12-14 11:58:06 +01:00
Alex Auvolat
f8e528c15d
Small refactor of tables internals 2022-12-14 10:48:49 +01:00
Alex Auvolat
de9d6cddf7
Prettier worker list table; remove useless CLI log messages 2022-12-12 17:17:05 +01:00
Alex Auvolat
280d1be7b1
Refactor health check and add ability to return it in json 2022-12-05 15:28:57 +01:00
Alex Auvolat
2065f011ca
Implement /health admin API endpoint to check node health 2022-12-05 14:59:15 +01:00
Alex Auvolat
c1fb65194c
Add sled default in garage_model also 2022-11-21 14:25:54 +01:00
Alex Auvolat
5b18fd8201
Add garage bucket cleanup-incomplete-uploads command 2022-11-04 11:55:59 +01:00
Alex Auvolat
ded444f6c9
Ability to have custom timeouts in request strategy (not used) 2022-09-20 16:01:41 +02:00
Alex Auvolat
56592e1853
RPC performance changes 
- configurable ping timeout
- single, much higher, configurable RPC timeout
- no more concurrency semaphore
 2022-09-19 20:31:00 +02:00
Alex Auvolat
ab722cb40f
Add checks on replication_factor of layouts we use (fix #363, fix #364) 2022-09-13 16:22:23 +02:00
Alex Auvolat
38be811b1c
Fix clippy lint that says we should implement Eq 2022-09-13 16:08:00 +02:00
Alex Auvolat
07febd3ecd
Ensure data dir is created immediately when Garage starts (fix #349) 2022-09-13 15:57:27 +02:00
Alex Auvolat
28a4af73ca
Use netapp 0.5 published from crates.io 2022-09-13 13:11:44 +02:00
Alex Auvolat
7f54706b95
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-09-08 15:50:56 +02:00
Alex Auvolat
d9d199a6c9
Merge branch 'main' into lx-perf-improvements 2022-09-08 15:49:17 +02:00
Alex Auvolat
ceb1f0229a
Move version back into util 2022-09-07 18:36:46 +02:00
Alex Auvolat
f310fce34b
Inject GIT_VERSION even later 2022-09-07 18:30:15 +02:00
Alex Auvolat
8adc654713
Merge branch 'main' into improve-deps 2022-09-07 18:13:27 +02:00
Alex Auvolat
28d86e7602
Report build features in garage --help 2022-09-07 17:05:21 +02:00
Alex Auvolat
db61f41030
Move GIT_VERSION injection later in build chain to reduce build times 2022-09-07 11:59:56 +02:00
Alex Auvolat
6b958979bd
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-09-06 22:13:01 +02:00
Alex Auvolat
6f02c36a89
cargo fmt 2022-09-06 17:59:41 +02:00
Alex Auvolat
0f5689c169
Include code from v0.5.1 directly to remove dependencies 2022-09-06 17:52:50 +02:00
Alex Auvolat
b886c75450
Make all DB engines optional build features 2022-09-06 17:09:43 +02:00
Alex Auvolat
48ffaaadfc
Bump versions to 0.8.0 (compatibility is broken already) 2022-09-06 16:47:56 +02:00
Alex Auvolat
07e6bcde85
Merge branch 'main' into lx-perf-improvements 2022-09-05 12:40:17 +02:00
Alex Auvolat
943d76c583
Ability to dynamically set resync tranquility 2022-09-02 15:34:21 +02:00
Alex Auvolat
a35d4da721
update netapp to 0.5 2022-07-29 12:25:02 +02:00
Alex Auvolat
8e7e680afe
First adaptation to WIP netapp with streaming body 2022-07-29 12:25:02 +02:00
Alex Auvolat
2f111e6b3d
Performance improvements: 
- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults
 2022-07-29 12:24:48 +02:00
Alex
4f38cadf6e Background task manager (#332) 
- [x] New background worker trait
- [x] Adapt all current workers to use new API
- [x] Command to list currently running workers, and whether they are active, idle, or dead
- [x] Error reporting
- Optimizations
  - [x] Merkle updater: several items per iteration
  - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on
- scrub:
  - [x] have only one worker with a channel to start/pause/cancel
  - [x] automatic scrub
  - [x] ability to view and change tranquility from CLI
  - [x] persistence of a few info
- [ ] Testing

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/332
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-08 13:30:26 +02:00
Alex
77e3fd6db2 improve internal item counter mechanisms and implement bucket quotas (#326) 
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/326
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-15 20:20:28 +02:00