chore(nixery): Housekeeping for depot compatibility

Cleans up a whole bunch of things I wanted to get out of the door
right away:

* depot internal references to //third_party/nixery have been replaced
  with //tools/nixery
* cleaned up files from Github
* fixed SPDX & Copyright headers
* code formatting and inclusion in //tools/depotfmt checks

Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
This commit is contained in:
Vincent Ambo 2022-04-20 16:41:20 +02:00 committed by clbot
parent 322589f9b8
commit 73f0087be2
27 changed files with 183 additions and 453 deletions

2
.gitattributes vendored
View File

@ -1,2 +0,0 @@
# Ignore stylesheet modifications for the book in Linguist stats
*.css linguist-detectable=false

View File

@ -1,27 +0,0 @@
# Build Nixery, spin up an instance and pull an image from it.
name: "Build and test Nixery"
on:
push:
branches:
- master
pull_request: {}
env:
NIX_PATH: "nixpkgs=https://github.com/NixOS/nixpkgs/archive/4263ba5e133cc3fc699c1152ab5ee46ef668e675.tar.gz"
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Install Nix
uses: cachix/install-nix-action@v13
- name: Checkout
uses: actions/checkout@v2.3.4
- name: Prepare environment
run: nix-env -f '<nixpkgs>' -iA go
- name: Check formatting
run: "test -z $(gofmt -l .)"
- name: Run `go vet`
run: "go vet ./..."
- name: Build Nixery
run: "nix-build --no-out-link"
- name: Run integration test
run: scripts/integration-test.sh

View File

@ -1,35 +0,0 @@
# How to Contribute
We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.
## Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution;
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to <https://cla.developers.google.com/> to see
your current agreements on file or to sign a new one.
You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.
## Commit messages
Commits in this repository follow the [Angular commit message
guidelines][commits].
## Code reviews
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
## Community Guidelines
This project follows [Google's Open Source Community
Guidelines](https://opensource.google.com/conduct/).
[commits]: https://github.com/angular/angular/blob/master/CONTRIBUTING.md#commit

View File

@ -4,7 +4,7 @@
-----------------
[![Build Status](https://github.com/tazjin/nixery/actions/workflows/build-and-test.yaml/badge.svg)](https://github.com/tazjin/nixery/actions/workflows/build-and-test.yaml)
[![Build status](https://badge.buildkite.com/016bff4b8ae2704a3bbbb0a250784e6692007c582983b6dea7.svg?branch=refs/heads/canon)](https://buildkite.com/tvl/depot)
**Nixery** is a Docker-compatible container registry that is capable of
transparently building and serving container images using [Nix][].
@ -24,6 +24,15 @@ You can watch the NixCon 2019 [talk about
Nixery](https://www.youtube.com/watch?v=pOI9H4oeXqA) for more information about
the project and its use-cases.
The canonical location of the Nixery source code is
[`//tools/nixery`][depot-link] in the [TVL](https://tvl.fyi)
monorepository. If cloning the entire repository is not desirable, the
Nixery subtree can be cloned like this:
git clone https://code.tvl.fyi/depot.git:/tools/nixery.git
The subtree is infrequently mirrored to `tazjin/nixery` on Github.
## Demo
Click the image to see an example in which an image containing an interactive
@ -139,8 +148,9 @@ separate Nix function, which will make it possible to build images directly in
Nix builds.
[Nix]: https://nixos.org/
[layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
[layering strategy]: https://tazj.in/blog/nixery-layers
[gist]: https://gist.github.com/tazjin/08f3d37073b3590aacac424303e6f745
[buildLayeredImage]: https://grahamc.com/blog/nix-and-layered-docker-images
[public]: https://nixery.dev
[depot-link]: https://cs.tvl.fyi/depot/-/tree/tools/nixery
[gcs]: https://cloud.google.com/storage/

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package builder
// This file implements logic for walking through a directory and creating a

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Package builder implements the logic for assembling container
// images. It shells out to Nix to retrieve all required Nix-packages

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package builder
import (

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package builder
import (

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// This package reads an export reference graph (i.e. a graph representing the
// runtime dependencies of a set of derivations) created by Nix and groups it in

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Package config implements structures to store Nixery's configuration at
// runtime as well as the logic for instantiating this configuration from the

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package config
import (

View File

@ -1,16 +1,5 @@
# Copyright 2019-2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# This function header aims to provide compatibility between builds of
# Nixery taking place inside/outside of the TVL depot.
@ -19,12 +8,13 @@
# build system and this will need some major adaptations to support
# that.
{ depot ? { nix.readTree.drvTargets = x: x; }
, pkgs ? import <nixpkgs> {}
, pkgs ? import <nixpkgs> { }
, preLaunch ? ""
, extraPackages ? []
, extraPackages ? [ ]
, maxLayers ? 20
, commitHash ? null
, ... }@args:
, ...
}@args:
with pkgs;
@ -54,7 +44,8 @@ let
"-ldflags=-s -w -X main.version=${nixery-commit-hash}"
];
};
in depot.nix.readTree.drvTargets rec {
in
depot.nix.readTree.drvTargets rec {
# Implementation of the Nix image building logic
nixery-prepare-image = import ./prepare-image { inherit pkgs; };
@ -79,55 +70,57 @@ in depot.nix.readTree.drvTargets rec {
# Container image containing Nixery and Nix itself. This image can
# be run on Kubernetes, published on AppEngine or whatever else is
# desired.
nixery-image = let
# Wrapper script for the wrapper script (meta!) which configures
# the container environment appropriately.
#
# Most importantly, sandboxing is disabled to avoid privilege
# issues in containers.
nixery-launch-script = writeShellScriptBin "nixery" ''
set -e
export PATH=${coreutils}/bin:$PATH
export NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
mkdir -p /tmp
# Create the build user/group required by Nix
echo 'nixbld:x:30000:nixbld' >> /etc/group
echo 'nixbld:x:30000:30000:nixbld:/tmp:/bin/bash' >> /etc/passwd
echo 'root:x:0:0:root:/root:/bin/bash' >> /etc/passwd
echo 'root:x:0:' >> /etc/group
# Disable sandboxing to avoid running into privilege issues
mkdir -p /etc/nix
echo 'sandbox = false' >> /etc/nix/nix.conf
# In some cases users building their own image might want to
# customise something on the inside (e.g. set up an environment
# for keys or whatever).
nixery-image =
let
# Wrapper script for the wrapper script (meta!) which configures
# the container environment appropriately.
#
# This can be achieved by setting a 'preLaunch' script.
${preLaunch}
# Most importantly, sandboxing is disabled to avoid privilege
# issues in containers.
nixery-launch-script = writeShellScriptBin "nixery" ''
set -e
export PATH=${coreutils}/bin:$PATH
export NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
mkdir -p /tmp
exec ${nixery-bin}/bin/nixery
'';
in dockerTools.buildLayeredImage {
name = "nixery";
config.Cmd = [ "${nixery-launch-script}/bin/nixery" ];
# Create the build user/group required by Nix
echo 'nixbld:x:30000:nixbld' >> /etc/group
echo 'nixbld:x:30000:30000:nixbld:/tmp:/bin/bash' >> /etc/passwd
echo 'root:x:0:0:root:/root:/bin/bash' >> /etc/passwd
echo 'root:x:0:' >> /etc/group
inherit maxLayers;
contents = [
bashInteractive
cacert
coreutils
git
gnutar
gzip
iana-etc
nix
nixery-prepare-image
nixery-launch-script
openssh
zlib
] ++ extraPackages;
};
# Disable sandboxing to avoid running into privilege issues
mkdir -p /etc/nix
echo 'sandbox = false' >> /etc/nix/nix.conf
# In some cases users building their own image might want to
# customise something on the inside (e.g. set up an environment
# for keys or whatever).
#
# This can be achieved by setting a 'preLaunch' script.
${preLaunch}
exec ${nixery-bin}/bin/nixery
'';
in
dockerTools.buildLayeredImage {
name = "nixery";
config.Cmd = [ "${nixery-launch-script}/bin/nixery" ];
inherit maxLayers;
contents = [
bashInteractive
cacert
coreutils
git
gnutar
gzip
iana-etc
nix
nixery-prepare-image
nixery-launch-script
openssh
zlib
] ++ extraPackages;
};
}

View File

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# Builds the documentation page using the Rust project's 'mdBook'
# tool.
@ -27,7 +16,8 @@ let
rev = "9f0baf5e270128d9101ba4446cf6844889e399a2";
sha256 = "1pf9i90gn98vz67h296w5lnwhssk62dc6pij983dff42dbci7lhj";
};
in runCommand "nixery-book" { } ''
in
runCommand "nixery-book" { } ''
mkdir -p $out
cp -r ${./.}/* .
chmod -R a+w src

View File

@ -68,10 +68,6 @@ production project we recommend setting up a private instance. The public Nixery
at `nixery.dev` is run on a best-effort basis and we make no guarantees about
availability.
### Is this an official Google project?
**No.** Nixery is not officially supported by Google.
### Who made this?
Nixery was written by [tazjin][], but many people have contributed to Nix over
@ -81,4 +77,4 @@ time, maybe you could become one of them?
[Nix]: https://nixos.org/nix
[layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
[layers]: https://grahamc.com/blog/nix-and-layered-docker-images
[tazjin]: https://github.com/tazjin
[tazjin]: https://tazj.in

View File

@ -65,13 +65,17 @@ use it with your own packages. There are three options available:
### 2.1. With a container image
The easiest way to run Nixery is to build a container image.
This section assumes that the container runtime used is Docker,
please modify instructions accordingly if
you are using something else.
The easiest way to run Nixery is to build a container image. This
section assumes that the container runtime used is Docker, please
modify instructions accordingly if you are using something else.
With a working Nix installation, building Nixery is done by invoking `nix-build
-A nixery-image` from a checkout of the [Nixery repository][repo].
With a working Nix installation, you can clone and build the Nixery
image like this:
```
git clone https://code.tvl.fyi/depot.git:/tools/nixery.git
nix-build -A nixery-image
```
This will create a `result`-symlink which points to a tarball containing the
image. In Docker, this tarball can be loaded by using `docker load -i result`.
@ -184,7 +188,6 @@ If the directory doesn't exist, Nixery will run fine but serve 404.
[nixery#4]: https://github.com/tazjin/nixery/issues/4
[Nix]: https://nixos.org/nix
[gcs]: https://cloud.google.com/storage/
[repo]: https://github.com/tazjin/nixery
[signed-urls]: under-the-hood.html#5-image-layers-are-requested
[ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
[nixinstall]: https://nixos.org/manual/nix/stable/installation/installing-binary.html

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package logs
// This file configures different log formatters via logrus. The

15
main.go
View File

@ -1,16 +1,5 @@
// Copyright 2019-2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// The nixery server implements a container registry that transparently builds
// container images based on Nix derivations.

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Package image implements logic for creating the image metadata
// (such as the image manifest and configuration).

View File

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
{ buildGoPackage }:

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Popcount fetches popularity information for each store path in a
// given Nix channel from the upstream binary cache.

View File

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# This file builds a wrapper script called by Nixery to ask for the
# content information for a given image.
@ -18,7 +7,7 @@
# The purpose of using a wrapper script is to ensure that the paths to
# all required Nix files are set correctly at runtime.
{ pkgs ? import <nixpkgs> {} }:
{ pkgs ? import <nixpkgs> { } }:
pkgs.writeShellScriptBin "nixery-prepare-image" ''
exec ${pkgs.nix}/bin/nix-build \

View File

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# Load a Nix package set from one of the supported source types
# (nixpkgs, git, path).
@ -24,7 +13,8 @@ let
let
url =
"https://github.com/NixOS/nixpkgs/archive/${channel}.tar.gz";
in import (fetchTarball url) importArgs;
in
import (fetchTarball url) importArgs;
# If a git repository is requested, it is retrieved via
# builtins.fetchGit which defaults to the git configuration of the
@ -35,7 +25,8 @@ let
# No special handling is used for paths, so users are expected to pass one
# that will work natively with Nix.
importPath = path: import (toPath path) importArgs;
in if srcType == "nixpkgs" then
in
if srcType == "nixpkgs" then
fetchImportChannel srcArgs
else if srcType == "git" then
fetchImportGit (fromJSON srcArgs)

View File

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# This file contains a derivation that outputs structured information
# about the runtime dependencies of an image with a given set of
@ -23,13 +12,13 @@
{
# Description of the package set to be used (will be loaded by load-pkgs.nix)
srcType ? "nixpkgs",
srcArgs ? "nixos-20.09",
system ? "x86_64-linux",
importArgs ? { },
# Path to load-pkgs.nix
loadPkgs ? ./load-pkgs.nix,
# Packages to install by name (which must refer to top-level attributes of
srcType ? "nixpkgs"
, srcArgs ? "nixos-20.09"
, system ? "x86_64-linux"
, importArgs ? { }
, # Path to load-pkgs.nix
loadPkgs ? ./load-pkgs.nix
, # Packages to install by name (which must refer to top-level attributes of
# nixpkgs). This is passed in as a JSON-array in string form.
packages ? "[]"
}:
@ -77,24 +66,28 @@ let
# `deepFetch haskellpackages.stylish-haskell` retrieves
# `haskellPackages.stylish-haskell`.
deepFetch = with lib; s: n:
let path = splitString "." n;
err = { error = "not_found"; pkg = n; };
# The most efficient way I've found to do a lookup against
# case-differing versions of an attribute is to first construct a
# mapping of all lowercased attribute names to their differently cased
# equivalents.
#
# This map is then used for a second lookup if the top-level
# (case-sensitive) one does not yield a result.
hasUpper = str: (match ".*[A-Z].*" str) != null;
allUpperKeys = filter hasUpper (attrNames s);
lowercased = listToAttrs (map (k: {
let
path = splitString "." n;
err = { error = "not_found"; pkg = n; };
# The most efficient way I've found to do a lookup against
# case-differing versions of an attribute is to first construct a
# mapping of all lowercased attribute names to their differently cased
# equivalents.
#
# This map is then used for a second lookup if the top-level
# (case-sensitive) one does not yield a result.
hasUpper = str: (match ".*[A-Z].*" str) != null;
allUpperKeys = filter hasUpper (attrNames s);
lowercased = listToAttrs (map
(k: {
name = toLower k;
value = k;
}) allUpperKeys);
caseAmendedPath = map (v: if hasAttr v lowercased then lowercased."${v}" else v) path;
fetchLower = attrByPath caseAmendedPath err s;
in attrByPath path fetchLower s;
})
allUpperKeys);
caseAmendedPath = map (v: if hasAttr v lowercased then lowercased."${v}" else v) path;
fetchLower = attrByPath caseAmendedPath err s;
in
attrByPath path fetchLower s;
# allContents contains all packages successfully retrieved by name
# from the package set, as well as any errors encountered while
@ -105,27 +98,30 @@ let
# Folds over the results of 'deepFetch' on all requested packages to
# separate them into errors and content. This allows the program to
# terminate early and return only the errors if any are encountered.
let splitter = attrs: res:
if hasAttr "error" res
then attrs // { errors = attrs.errors ++ [ res ]; }
else attrs // { contents = attrs.contents ++ [ res ]; };
init = { contents = []; errors = []; };
fetched = (map (deepFetch pkgs) (fromJSON packages));
in foldl' splitter init fetched;
let
splitter = attrs: res:
if hasAttr "error" res
then attrs // { errors = attrs.errors ++ [ res ]; }
else attrs // { contents = attrs.contents ++ [ res ]; };
init = { contents = [ ]; errors = [ ]; };
fetched = (map (deepFetch pkgs) (fromJSON packages));
in
foldl' splitter init fetched;
# Contains the export references graph of all retrieved packages,
# which has information about all runtime dependencies of the image.
#
# This is used by Nixery to group closures into image layers.
runtimeGraph = runCommand "runtime-graph.json" {
__structuredAttrs = true;
exportReferencesGraph.graph = allContents.contents;
PATH = "${coreutils}/bin";
builder = toFile "builder" ''
. .attrs.sh
cp .attrs.json ''${outputs[out]}
'';
} "";
runtimeGraph = runCommand "runtime-graph.json"
{
__structuredAttrs = true;
exportReferencesGraph.graph = allContents.contents;
PATH = "${coreutils}/bin";
builder = toFile "builder" ''
. .attrs.sh
cp .attrs.json ''${outputs[out]}
'';
} "";
# Create a symlink forest into all top-level store paths of the
# image contents.
@ -151,7 +147,7 @@ let
# Image layer that contains the symlink forest created above. This
# must be included in the image to ensure that the filesystem has a
# useful layout at runtime.
symlinkLayer = runCommand "symlink-layer.tar" {} ''
symlinkLayer = runCommand "symlink-layer.tar" { } ''
cp -r ${contentsEnv}/ ./layer
tar --transform='s|^\./||' -C layer --sort=name --mtime="@$SOURCE_DATE_EPOCH" --owner=0 --group=0 -cf $out .
'';
@ -159,9 +155,10 @@ let
# Metadata about the symlink layer which is required for serving it.
# Two different hashes are computed for different usages (inclusion
# in manifest vs. content-checking in the layer cache).
symlinkLayerMeta = fromJSON (readFile (runCommand "symlink-layer-meta.json" {
buildInputs = [ coreutils jq openssl ];
}''
symlinkLayerMeta = fromJSON (readFile (runCommand "symlink-layer-meta.json"
{
buildInputs = [ coreutils jq openssl ];
} ''
tarHash=$(sha256sum ${symlinkLayer} | cut -d ' ' -f1)
layerSize=$(stat --printf '%s' ${symlinkLayer})
@ -181,7 +178,8 @@ let
error = "not_found";
pkgs = map (err: err.pkg) allContents.errors;
};
in writeText "build-output.json" (if (length allContents.errors) == 0
then toJSON buildOutput
else toJSON errorOutput
in
writeText "build-output.json" (if (length allContents.errors) == 0
then toJSON buildOutput
else toJSON errorOutput
)

View File

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# Configures a shell environment that builds required local packages to
# run Nixery.

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Filesystem storage backend for Nixery.
package storage

View File

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Google Cloud Storage backend for Nixery.
package storage

View File

@ -1,16 +1,5 @@
// Copyright 2019-2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Package storage implements an interface that can be implemented by
// storage backends, such as Google Cloud Storage or the local