james/garage-cluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9 Commits 1 Branch 0 Tags 43 KiB
Shell 98.8%
HTML 1.2%
main
Go to file
James Andariese 441856198f add generate_external_config 
generates a config based on the configmap and secrets in the cluster.
this config is intended to be used to add an external node to the
cluster.
2025-04-11 14:42:08 -05:00
0ns.yaml initial import 2025-04-10 20:53:23 -05:00
aws updates after learning things 2025-04-10 23:50:51 -05:00
cm.yaml increase replication_factor to 3 (best practices) 2025-04-11 14:40:07 -05:00
direct-hosted-sites.yaml change to regex-type ingress (haproxy-specific) 2025-04-11 14:40:25 -05:00
ds.yaml initial import 2025-04-10 20:53:23 -05:00
index.html add a basic index 2025-04-11 00:36:03 -05:00
ingress.yaml remove web from ingress 2025-04-10 23:59:42 -05:00
README.md add docs 2025-04-10 21:45:33 -05:00
svc.yaml initial import 2025-04-10 20:53:23 -05:00
tools add generate_external_config 2025-04-11 14:42:08 -05:00

README.md

garage-cluster

Get started with Garage in Kubernetes!

Overview

Extremely opinionated tools to deploy Garage to Kubernetes and manage it.

Actually, these are my tools. You're welcome to borrow them.

Creating a Garage Cluster

To create a Garage cluster, you must do the following:

  • prepare your nodes:

    • all nodes should participate even if they don't have storage for maximum flexibility (load balancing, localhost endpoint for Garage). this repo doesn't setup anything ot make use of this flexibility but it may or you may. you'll want to remove capacity (-c 10G) from nodes which won't store and add -g for gateway instead.

    • nodes which will have storage should have that storage added via LVM so that you can add storage in the future. This DaemonSet-based Garage cluster does not account for multiple HDDs even though they're supported by Garage. To expand your storage, you'll need LVM.

    • partition partitions, format them, and mount them

      NVME_VG=nvme-vg

      HDD_VG=hdd-vg lvcreate -L 1G $NVME_VG -n garage-meta mkfs.btrfs /dev/$NVME_VG/garage-meta lvcreate -L 10G $HDD_VG -n garage-data mkfs.xfs /dev/$HDD_VG/garage-data echo "/dev/$NVME_VG/garage-meta /var/lib/garage/meta btrfs defaults 0 2" >> /etc/fstab echo "/dev/$HDD_VG/garage-data /var/lib/garage/data xfs defaults 0 2" >> /etc/fstab mount /var/lib/garage/{meta,data}

  • modify the contents of cm.yaml:

    • The daemonset will translate these entries into a garage.toml file for you. to place something into a section, prefix it with a period.

    • Settings to check: replication_factor, s3_api.root_domain, s3_web.root_domain

    • You may also include bootstrap peers here if you like. if you do not, you'll need to connect them via the tools which may be more convenient anyway. don't worry about this if you're not sure what to do yet.

  • Modify the contents of ingress.yaml to match your controller's requirements and/or your hostname and prefix.

  • Apply the manifests:

    kubectl apply -f . echo "Ctrl-C to exit" kubectl get ds -w

  • Bootstrap the nodes once they've started:

    ./tools connect

  • Alias garage to use your kubernetes pods:

    ./tools env >> ~/.bashrc

  • Generate a sample layout to start:

If you have the node feature discovery plugin, this may expose likely capacity values. If you do not have the plugin, it will default to 10G for every node.

You should almost certainly edit the resulting commands before applying them to your cluster. $PWD/tools is used to make the resulting script relocatable.

$PWD/tools generate-layout -z my-zone > run-layout.sh
$EDITOR run-layout.sh
./run-layout.sh
garage layout show
read -p 'everything look good?  press enter to continue or ctrl-c to abort'
garage layout apply