[ notes to myself ]

Build and sign container images with Gitlab CI, Cosign, and Vault

3 min read

Gitlab has built-in support for keyless signing with Sigstore - if you’re using a hosted version of Gitlab. If you’re self-hosting, this requires a bit more effort to setup.

Here’s how I set up a Gitlab CI workflow to build an image with kaniko, push it to a self-hosted Harbor registry, and sign the digest with Cosign and Vault.

Vault setup

Enable transit secrets engine, create the cosign key, and create a policy for access:

vault secrets enable transit
vault write -f transit/keys/cosign type=ecdsa-p256
vault policy write cosign - '
  path "transit/keys/cosign" {
    capabilities = ["read"]
  }
  path "transit/hmac/cosign/*" {
    capabilities = ["update"]
  }
  path "transit/sign/cosign/*" {
    capabilities = ["update"]
  }
  path "transit/verify/cosign" {
    capabilities = ["create"]
  }
  path "transit/verify/cosign/*" {
    capabilities = ["update"]
  }
'

Set up JWT auth:

vault auth enable jwt
vault write auth/jwt/role/cosign - '
  {
    "bound_audiences": [
      "https://gitlab.example.com"
    ],
    "bound_claims": {
      "iss": [
        "gitlab.example.com",
        "https:/gitlab.example.com"
      ],
      "project_path": [
        "container-images/*"
      ]
    },
    "bound_claims_type": "glob",
    "policies": [
      "cosign"
    ],
    "role_type": "jwt",
    "user_claim": "user_email"
  }
'
vault write auth/jwt/config \
  jwks_url="https://gitlab.example.com/oauth/discovery/keys"

kaniko image

Kaniko is now maintained by Chainguard, but they only provide the source code - not the images or binaries for actually doing things with kaniko. Fortunately there is an image available from gitlab.com/gitlab-ci-utils/container-images/kaniko. I’m going to build a custom image based on that and my own alpine image (as an easy way to bring in my internal CA certs):

ARG ALPINE_VERSION=3.21.3
ARG KANIKO_VERSION=1.25.1
 
FROM registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v${KANIKO_VERSION}-debug AS kaniko
FROM harbor.example.com/library/alpine:${ALPINE_VERSION}
 
COPY --from=kaniko /kaniko /kaniko 
ENV PATH=$PATH:/kaniko \
  DOCKER_CONFIG=/kaniko/.docker
 
CMD ["/bin/sh"]

cosign image

To be able to handle interacting with Vault and signing the digest in a single step, I’m using a customized image which contains both of those the needed tools.

ARG ALPINE_VERSION=3.21.3
ARG COSIGN_VERSION=2.5.3
ARG VAULT_VERSION=1.20.2
 
FROM ghcr.io/sigstore/cosign/cosign:v${COSIGN_VERSION} AS cosign
FROM hashicorp/vault:${VAULT_VERSION} AS vault
FROM harbor.example.com/library/alpine:${ALPINE_VERSION}
COPY --from=cosign /ko-app/cosign /cosign/cosign 
COPY --from=vault /bin/vault /cosign/vault
 
RUN mkdir -p /cosign/.docker
ENV PATH=$PATH:/cosign \
  DOCKER_CONFIG=/cosign/.docker
 
CMD ["/bin/sh"]

Gitlab pipeline

# .gitlab-ci.yml
 
stages:
- build
 
build-image:
  stage: build
  image: harbor.example.com/library/kaniko:latest
  before_script:
    - echo "{\"auths\":{\"${HARBOR_HOST}\":{\"auth\":\"$(echo -n ${HARBOR_USERNAME}:${HARBOR_PASSWORD} | base64 -w 0)\"}}}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${HARBOR_HOST}/${HARBOR_PROJECT}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA}"
      --digest-file "./digest"
  artifacts:
    paths:
      - digest
 
sign-image:
  stage: build
  image: harbor.example.com/library/cosign:latest
  needs:
    - build-image
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.example.com
  variables:
    VAULT_ADDR: https://vault.example.com
    VAULT_JWT_ROLE: cosign
    COSIGN_KEY: hashivault://cosign
  before_script:
  - export VAULT_TOKEN="$(vault write --field=token auth/jwt/login role=$VAULT_JWT_ROLE jwt=$VAULT_ID_TOKEN)"
  - echo "{\"auths\":{\"${HARBOR_HOST}\":{\"auth\":\"$(echo -n ${HARBOR_USERNAME}:${HARBOR_PASSWORD} | base64 -w 0)\"}}}" > /cosign/.docker/config.json
  script:
  - /cosign/cosign sign "${HARBOR_HOST}/${HARBOR_PROJECT}/${CI_PROJECT_NAME}@$(busybox cat digest)" --tlog-upload=false

See also:

Refs:

Backlinks