Kustomize

Configuration

Kubernetes

Kustomize

Since there is so much overlap between the deployment of my various Apps using Kustomize simplified the entire process by allowing for consistency. There is still an unfortunate amount of duplication due to replacements and labels needing to be added at the app level (since the name needs to be the final name) but even those sections are mostly copy-pasted to maintain consistency.

Apps do not mention the patches/variations from their Kustomizations, only the base model.

Namespace Kustomization

Each namespace has a kustomization.yaml that includes the appropriate namespace.yaml to deploy the resource as well as any namespace level resources (smtp-secrets.yaml since this is shared between resources) and links to the nested folders that run individual app kustomization. This is done to allow full namespace/stack deployment by deploying the appropriate level kustomization file rather than going through each individually.

Backups

Cron job for Longhorn backups. Originally there was the potential for additional backups so it was created as a template.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - batch.yaml

labels:
  - includeSelectors: true
    pairs:
      app.kubernetes.io/managed-by: kustomize

Deployment

Generic deployment template.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - pvc.yaml
  - secret.yaml          # Infisical Secret
  - httproute-http.yaml
  - httproute-https.yaml

labels:
  - includeSelectors: true
    pairs:
      app.kubernetes.io/managed-by: kustomize

DeployService

Deployment plus a service, but no connectivity outside of Kubernetes

kuztomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Define resources:
resources:
  - deployment.yaml
  - service.yaml

labels:
  - includeSelectors: true
    pairs:
      app.kubernetes.io/managed-by: kustomize

HelmChart

Mirrors Deployment but uses the HelmChart custom resource for the primary deployment while adding HTTPRoutes and Infisical secrets.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - helmchart.yaml
  - httproute-http.yaml
  - httproute-https.yaml
  - secret.yaml

labels:
  - includeSelectors: true
    pairs:
      app.kubernetes.io/managed-by: kustomize

HelmChart-NoAddons

Only a HelmChart to allow for deploying one that needs additional resources that are not part of the standard. If the only thing being deployed is the chart itself it will probably just be deployed as part of a namespace level kustomization.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - helmchart.yaml

labels:
  - includeSelectors: true
    pairs:
      app.kubernetes.io/managed-by: kustomize

NoService

Deployments that do not need any services at all

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Define resources:
resources:
  - deployment.yaml
  - pvc.yaml
  - secret.yaml

labels:
  - includeSelectors: true
    pairs:
      app.kubernetes.io/managed-by: kustomize

Public-OIDC

Built from Deployment but adds the X-Forwarded-Proto: https header on HTTP route calls going to port 80.

StatfulSet

Used for stateful workloads but mirrors a deployment for all other relevant resources.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - statefulset.yaml
  - httproute-http.yaml
  - httproute-https.yaml
  - secret.yaml
  - service.yaml

labels:
  - includeSelectors: true
    pairs:
      app.kubernetes.io/managed-by: kustomize

Common Replacements and Labels

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namePrefix: example-             # Target app
resources:
  - ../../../kustomize/example/  # Kustomize baseline
# Don't override existing namespaces for Infisical secrets
transformers:
  - |-
    apiVersion: builtin
    kind: NamespaceTransformer
    metadata:
      name: notImportantHere
      namespace: example         # Target namespace
    unsetOnly: true

replacements:
  - source:
      kind: Service
      name: svc
    targets:
      - select:
          kind: HTTPRoute
        options:
          create: true
        fieldPaths:
          - spec.rules.0.backendRefs.0.name
  - source:
      kind: Deployment
      name: app
      fieldPath: metadata.labels.[app.kubernetes.io/appName]
    targets:
      - select:
          kind: HTTPRoute
        options:
          create: true
          delimiter: "."
          index: 0
        fieldPaths:
          - spec.hostnames.0
      - select:
          kind: InfisicalSecret
        options:
          delimiter: "-"
          index: 0
        fieldPaths:
          - spec.managedSecretReference.secretName
      - select:
          kind: InfisicalSecret
        options:
          delimiter: "/"
          index: 2
        fieldPaths:
          - spec.authentication.universalAuth.secretsScope.secretsPath
      - select:
          kind: Service
        fieldPaths:
          - spec.ports.0.name
          - spec.ports.0.targetPort
  - source:
      kind: Deployment
      name: app
      fieldPath: metadata.labels.[app.kubernetes.io/appNamespace]
    targets:
      - select:
          kind: InfisicalSecret
        fieldPaths:
          - spec.managedSecretReference.secretNamespace
      - select:
          kind: InfisicalSecret
        options:
          delimiter: "/"
          index: 1
        fieldPaths:
          - spec.authentication.universalAuth.secretsScope.secretsPath
  # With `app.kubernetes.io/routePrefix` label
  - source:
      kind: HTTPRoute
      name: http
      fieldPath: metadata.labels.[app.kubernetes.io/routePrefix]
    targets:
      - select:
          kind: HTTPRoute
        options:
          create: true
          delimiter: "."
          index: 0
        fieldPaths:
          - spec.hostnames.0

labels:
  - includeSelectors: true
    pairs:
      app.kubernetes.io/appName: example         # app name
  - pairs:
      app.kubernetes.io/appNamespace: example    # target namespace
  # Override HTTPRoute match with custom prefix
  - pairs:
      app.kubernetes.io/routePrefix: ex          # Target prefix

K3s Longhorn