Heyal Solutions blog

Bootstrapping a private EKS cluster with Argo CD

sales@heyal.co.uk (Alistair Hey) — Sat, 17 Oct 2026 00:00:00 GMT

A Kubernetes API server on the public internet is an attack surface most teams don’t need. In regulated environments it’s often a non-starter. EKS lets you turn the public endpoint off entirely, so the API is only reachable from inside your VPC.

That’s great for security. It’s less great for your pipeline. GitHub Actions can create the cluster, because Terraform only talks to the AWS API, which is public. But the moment you want to install something into the cluster, the runner has no route to the API server.

This post walks through building a private EKS cluster with Terraform and GitHub Actions, then getting Argo CD installed so it can take over from there. All of the code is in HeyalSolutions/eks-private-argocd-bootstrap.

The key insight: you only need to get in once

Argo CD is pull-based. It runs inside the cluster and pulls its desired state from Git. Once it’s installed, your pipeline never needs to talk to the Kubernetes API again. Changes go through pull requests, and Argo CD applies them.

So the private endpoint problem isn’t “how does CI deploy to the cluster?”. It’s “how do we install Argo CD once?”. That’s a much smaller problem.

GitHub Actions authenticates to AWS with OIDC and runs Terraform. Everything is created through the public AWS API.
The pipeline starts a CodeBuild build. CodeBuild runs inside the VPC.
The build installs Argo CD and one root Application.
Argo CD pulls everything else from Git, including its own configuration.

Step 1: let GitHub Actions into AWS (without keys)

GitHub Actions can exchange a short-lived OIDC token for AWS credentials, so there are no access keys to store or rotate. This needs an IAM OIDC provider and a role that trusts it. Both have to exist before the pipeline can run, so they live in a small bootstrap/ stack you apply once from your own machine. It also creates the S3 bucket for Terraform state.

The important part is the trust policy. Only this repository’s main branch and its pull requests can assume the role:

resource "aws_iam_openid_connect_provider" "github" {
  url            = "https://token.actions.githubusercontent.com"
  client_id_list = ["sts.amazonaws.com"]
}

data "aws_iam_policy_document" "github_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values = [
        "repo:${local.github_repo}:ref:refs/heads/main",
        "repo:${local.github_repo}:pull_request",
      ]
    }
  }
}

The example attaches AdministratorAccess to keep things short. In a real setup, scope it down and use a separate read-only role for pull request plans.

Step 2: the VPC

The VPC module does the heavy lifting. Three tiers of subnets:

Private subnets for nodes, internal load balancers and the CodeBuild bootstrap
Public subnets for the NAT gateway and internet-facing load balancers
Intra subnets for the EKS control plane’s network interfaces, with no route to the internet at all

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.7"

  name = local.name
  cidr = local.vpc_cidr
  azs  = local.azs

  private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
  intra_subnets   = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 52)]

  enable_nat_gateway = true
  single_nat_gateway = true # one per AZ in production

  public_subnet_tags  = { "kubernetes.io/role/elb" = 1 }
  private_subnet_tags = { "kubernetes.io/role/internal-elb" = 1 }
}

The NAT gateway matters here. Nodes need it to pull images, and the bootstrap build needs it to download Helm charts.

Step 3: a private EKS cluster

Version 21 of the EKS module makes the private endpoint explicit:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.29"

  name               = local.name
  kubernetes_version = "1.35"

  endpoint_private_access = true
  endpoint_public_access  = false

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.intra_subnets

  addons = {
    coredns                = {}
    kube-proxy             = {}
    eks-pod-identity-agent = { before_compute = true }
    vpc-cni                = { before_compute = true }
  }

  eks_managed_node_groups = {
    default = {
      ami_type       = "AL2023_x86_64_STANDARD"
      instance_types = ["m7i.large"]
      min_size       = 2
      max_size       = 5
      desired_size   = 2
    }
  }

  # ...access entries and security group rules, below
}

Two more pieces wire up the bootstrap. Cluster access uses access entries rather than the old aws-auth ConfigMap. The GitHub Actions role created the cluster, but it couldn’t reach it anyway, so it gets no Kubernetes permissions. Only the bootstrap build’s role does:

  enable_cluster_creator_admin_permissions = false

  access_entries = {
    bootstrap = {
      principal_arn = aws_iam_role.bootstrap.arn
      policy_associations = {
        admin = {
          policy_arn   = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
          access_scope = { type = "cluster" }
        }
      }
    }
  }

  security_group_additional_rules = {
    bootstrap_https = {
      description              = "CodeBuild bootstrap to API server"
      protocol                 = "tcp"
      from_port                = 443
      to_port                  = 443
      type                     = "ingress"
      source_security_group_id = aws_security_group.bootstrap.id
    }
  }

Step 4: the pipeline

The workflow plans on pull requests and applies on main. id-token: write is what lets the job request the OIDC token:

permissions:
  id-token: write
  contents: read

jobs:
  terraform:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: infra
    steps:
      - uses: actions/checkout@v7

      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
          aws-region: eu-west-2

      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: 1.16.6
          terraform_wrapper: false

      - run: terraform init -input=false
      - run: terraform plan -input=false -out=tfplan

      - name: Apply
        if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
        run: terraform apply -input=false tfplan

At this point we have a private cluster with nodes and nothing running on it. The pipeline can’t go any further on its own.

Step 5: getting in. The options

There are plenty of ways to get from a GitHub-hosted runner to a private API endpoint. Approximate us-east-1 costs, not counting the NAT gateway every option needs anyway:

Option	How it works	Pros	Cons	Rough cost
SSH jumpbox	Public EC2 instance with port 22 open; tunnel kubectl through it	Familiar, simple	Public SSH attack surface, key management, always on	~$7/month
AWS Client VPN	Runner connects to a managed VPN endpoint in the VPC	Fully AWS-managed; good for human access too	Expensive; certificate setup; an OpenVPN client in CI is fiddly	~$73/month per subnet association, plus connection hours
Tailscale (or similar)	Subnet router in the VPC; the runner joins your tailnet	Very easy; great for humans; no inbound ports	Third party in your access path; another account and auth key	Free tier, plus a ~$3/month router instance
SSM port forwarding	Instance with no inbound ports; tunnel to the API endpoint via Session Manager	No open ports; IAM-controlled; audited in CloudTrail	Standing instance; Session Manager plugin in CI; kubeconfig needs a `tls-server-name` override	~$3/month
Lambda in the VPC	Terraform invokes a function that runs `helm install`	Effectively free; pure Terraform; nothing left running	You have to package `helm` into the function; harder to debug; 15-minute limit	~$0
CodeBuild in the VPC	The pipeline starts a build; the buildspec runs `helm` inside the VPC	Effectively free; nothing left running; plain shell script; logs in CloudWatch	Another AWS service; a minute or so of start-up	~$0.02 per run

There’s also a way to skip the problem altogether: EKS Capabilities can run a managed Argo CD for you, operated by AWS and able to reach a private cluster. It’s a good fit if you’re happy for AWS to own Argo CD. It wasn’t what we wanted here, because we want Argo CD to manage itself from Git like every other component. It also needs IAM Identity Center for sign-in.

We went with CodeBuild. It ties with Lambda on cost, but there’s nothing to package: the bootstrap is a readable shell script, and the logs land in CloudWatch.

Step 6: the CodeBuild bootstrap

The CodeBuild project has no source. Its buildspec is passed in from Terraform, and it runs in the private subnets with its own security group (the one the EKS rule above allows in):

resource "aws_codebuild_project" "bootstrap" {
  name          = "${local.name}-bootstrap"
  service_role  = aws_iam_role.bootstrap.arn
  build_timeout = 20

  source {
    type      = "NO_SOURCE"
    buildspec = file("${path.module}/buildspec.yml")
  }

  artifacts {
    type = "NO_ARTIFACTS"
  }

  environment {
    type         = "LINUX_CONTAINER"
    compute_type = "BUILD_GENERAL1_SMALL"
    image        = "aws/codebuild/amazonlinux-x86_64-standard:6.0"

    environment_variable {
      name  = "CLUSTER_NAME"
      value = module.eks.cluster_name
    }
    environment_variable {
      name  = "ARGOCD_CHART_VERSION"
      value = "10.10.5"
    }
    environment_variable {
      name  = "CLUSTER_RESOURCES_REPO"
      value = local.cluster_resources_repo
    }
    environment_variable {
      name  = "DEPLOY_KEY"
      type  = "SECRETS_MANAGER"
      value = data.aws_secretsmanager_secret.deploy_key.arn
    }
  }

  vpc_config {
    vpc_id             = module.vpc.vpc_id
    subnets            = module.vpc.private_subnets
    security_group_ids = [aws_security_group.bootstrap.id]
  }
}

Its IAM role needs eks:DescribeCluster (for the kubeconfig), read access to the deploy key secret, CloudWatch Logs, and the EC2 network interface permissions CodeBuild uses to attach to the VPC. The full policy is in the repo.

The buildspec does four things: writes a kubeconfig, creates the repo credentials so Argo CD can read the private cluster-resources repo, installs Argo CD with Helm, and applies one root Application. It only does this once. If the argocd Helm release already exists, Argo CD is managing itself and a default install would fight with it:

version: 0.2

env:
  shell: bash
  variables:
    HELM_VERSION: v4.3.1
    KUBECTL_VERSION: v1.35.9

phases:
  install:
    commands:
      - curl -fsSL "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" | tar -xz -C /tmp
      - install /tmp/linux-amd64/helm /usr/local/bin/helm
      - curl -fsSLo /usr/local/bin/kubectl "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl"
      - chmod +x /usr/local/bin/kubectl

  build:
    commands:
      - aws eks update-kubeconfig --name "$CLUSTER_NAME"
      - kubectl get nodes

      # (writes root-app.yaml, the Application in step 7)

      - |
        set -eo pipefail
        if helm status argocd --namespace argocd >/dev/null 2>&1; then
          echo "Argo CD already installed; nothing to do."
        else
          kubectl create namespace argocd

          kubectl create secret generic cluster-resources-repo \
            --namespace argocd \
            --from-literal=type=git \
            --from-literal=url="$CLUSTER_RESOURCES_REPO" \
            --from-literal=sshPrivateKey="$DEPLOY_KEY" \
            --dry-run=client -o yaml \
            | kubectl label --local -f - argocd.argoproj.io/secret-type=repository -o yaml \
            | kubectl apply -f -

          helm upgrade --install argocd argo-cd \
            --repo https://argoproj.github.io/argo-helm \
            --version "$ARGOCD_CHART_VERSION" \
            --namespace argocd \
            --wait --timeout 10m

          kubectl apply -f root-app.yaml
        fi

Back in the workflow, a second job starts the build through the AWS API and waits for it:

  bootstrap:
    needs: terraform
    if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
          aws-region: eu-west-2

      - name: Bootstrap Argo CD
        env:
          PROJECT: ${{ needs.terraform.outputs.bootstrap_project }}
          LOG_GROUP: ${{ needs.terraform.outputs.bootstrap_log_group }}
        run: |
          build_id=$(aws codebuild start-build --project-name "$PROJECT" --query 'build.id' --output text)

          while true; do
            status=$(aws codebuild batch-get-builds --ids "$build_id" --query 'builds[0].buildStatus' --output text)
            [ "$status" != "IN_PROGRESS" ] && break
            sleep 15
          done

          # CodeBuild names the log stream after the build ID (the part after the colon)
          aws logs tail "$LOG_GROUP" --log-stream-name-prefix "${build_id#*:}" --since 1h || true

          if [ "$status" != "SUCCEEDED" ]; then
            echo "::error::Bootstrap build finished with status $status"
            exit 1
          fi

It runs on every push to main, but thanks to the guard it’s a no-op once Argo CD is installed. It costs a couple of cents, and a brand new cluster gets bootstrapped automatically.

Step 7: Argo CD manages itself

The root Application points at the cluster-resources repo, a Helm chart whose templates are themselves Argo CD Applications (the “app of apps” pattern):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-resources
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@github.com:my-org/cluster-resources.git
    targetRevision: main
    path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

One of those templates is an Application for Argo CD itself. It uses the same chart and the same release name (argocd) as the bootstrap install, so it adopts the resources Helm created rather than creating new ones:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://argoproj.github.io/argo-helm
    chart: argo-cd
    targetRevision: {{ .Values.argocd.chartVersion | quote }}
    helm:
      releaseName: argocd
      valuesObject:
        {{- toYaml .Values.argocd.values | nindent 8 }}
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true # Argo CD's CRDs are too big for client-side apply

From here on, Argo CD’s configuration lives in cluster-resources/values.yaml. Upgrading Argo CD is a pull request that bumps chartVersion. Every other add-on (cert-manager, ingress controllers, external-secrets, monitoring) is another template in the same chart, following the same pattern.

Things to watch out for

Keep the bootstrap chart version in step. The bootstrap installs ARGOCD_CHART_VERSION, and Git then says what it should be. If they drift, a new cluster briefly runs the older version until Argo CD syncs itself.
The bootstrap guard is load-bearing. Without it, every pipeline run would reinstall Argo CD with default values and fight with the self-managed configuration.
Humans still need a way in. CI doesn’t need cluster access any more, but people sometimes do. SSM port forwarding or a VPN (from the table above) works well for break-glass kubectl, with its own access entry.
Use a deploy key or GitHub App for Argo CD. It should have read-only access to cluster-resources. The example stores a deploy key in Secrets Manager, created by the one-time bootstrap/ stack so it exists before the first run.

Wrapping up

A private EKS endpoint doesn’t have to mean self-hosted runners, VPNs in CI or a bastion you have to babysit. Because GitOps is pull-based, you only need to get inside the VPC once, and a CodeBuild build started from the pipeline does that for a couple of cents. After that, Git is the only way changes reach the cluster.

The full example, including the IAM policies and the cluster-resources chart, is at HeyalSolutions/eks-private-argocd-bootstrap.

My Home Office Build, Part 4: Finishing Off, and Four Years On

sales@heyal.co.uk (Alistair Hey) — Fri, 16 Oct 2026 00:00:00 GMT

A note from the future: parts 1 to 3 were written in 2022, while the build was still going on. This part was written in October 2026, after four years of working in the office, so it covers how it has held up as well as how it was finished.

At the end of part 3 the office had walls, insulation, plasterboard and doors. It looked like a building, but it definitely wasn’t somewhere you’d want to sit for eight hours a day. This part covers everything it took to get from there to moving in.

Plastering

I fitted all of the plasterboard myself, but I got a plasterer in to skim it. Plastering is HARD!

TIP: Unless you’ve plastered before, this is one job worth paying for. Every imperfection shows once it’s painted.

Painting

Once the plaster had dried out properly, everything got painted white. No feature walls, nothing fancy.

Outside: render, soffits and gutters

The outside walls were rendered in a cream colour to match our house.

The roof slopes towards the back (the 1:40 pitch from part 2), so the gutter and downpipe are at the back. It’s finished with soffit boards, which you can see in the photo at the top of part 1.

Nothing on the outside has been treated or sealed, and four years on it’s all still good with no issues.

Second fix electrics

Second fix is when the sockets, switches and lights actually go on the ends of all the cables from first fix (part 3). It went smoothly. I got an electrician to come and check and certify all of the work, because electrics are a safety thing and not something to guess at.

As for those 10 network points: I use two or three of them. I probably didn’t need them all, but I don’t regret it. Having them everywhere means I can put the desk or the WiFi access point wherever I like.

Underfloor heating and flooring

I installed electric underfloor heating mats myself, then laid laminate on top: large, wide planks. I’ve laid laminate in four or five rooms before, and the floor was completely flat and level because it was a brand new build, so this was one of the easiest jobs of the whole project.

TIP: Lay the flooring before the skirting boards, so the skirting covers the expansion gap around the edges.

Skirting boards

Standard skirting, nothing special. A mitre saw makes the corners easy.

Heating and cooling

Winter: the underfloor heating plus all that insulation from part 3 keeps the office warm. It holds the heat well. On the very coldest days I add an oil-filled radiator.

Summer: this is the one thing I got wrong. The bifold doors face south, and there’s no cooling, so on a sunny day the office gets STUPIDLY hot. Two things help:

Reflective film on the glass. It made a real difference to how much heat comes in through the doors.
Opening the bifolds right up to let the hot air out.

Moving in

The office is set up with a standing desk, an office chair, a sofa and a rug. Behind me there’s a sideboard with photos and trinkets, including photos of places I’ve been on holiday.

Four years on

I’ve worked in the office for four years now, and the only real maintenance has been:

Clearing leaves off the roof.
Clearing the gutter and downpipe. It backed up once because of leaves. Now I clear it every six months or so, and it hasn’t happened again.
Trimming back the bushes behind the office so they don’t rub against the roof.

That’s it!

What would I do differently?

Not a lot. The big one is that I wouldn’t make it south facing. Big glass doors facing south look great, but in summer the office is so, so hot. If I built another one, I’d face the glass away from the midday sun or plan for cooling from the start.

Time, cost and tools

Time: about two months from start to finish. That was evenings, weekends and a week off work.

Cost: about £15,000 in total. That’s not cheap! The biggest costs were:

The electrician, including getting power down to the bottom of the garden
The plasterer
The bifold doors and the window, which were expensive
The concrete pump. Looking back, the base was probably overkill!

Tools: if I could recommend one thing, it’s hiring a trench digger. Digging the 20m cable trench by hand (part 1) was the hardest part of the whole build.

Thanks!

A big thank you to my electrician, the plasterer, and the window and door fitters. And to the friends and family who helped out along the way: all the “we” in this series.

My Home Office Build, Part 3: Electrics, Insulation and Plasterboard

sales@heyal.co.uk (Alistair Hey) — Thu, 30 Jun 2022 00:00:00 GMT

Once the shell was watertight, the work moved inside: cables, insulation, plasterboard, and finally the windows and doors.

First Fix Electrics

Less than 1 day (cable, internet cable)

I went a bit mad with sockets. There are 10 double sockets around the 3 walls (back and 2 sides). It’s easy to add them now and impossible to add them later. I also added 10 internet points for the same reason. I’m currently using 2. One for the WIFI router and one for my desktop.

Hint: Get your electrician to do this.

I’m not going to show pictures (as I don’t have any) and didn’t do the work.

We went with 5 down-lights on the front (outside), 2 up/down lights and 6 lights inside.

Insulation

1-2 days (saw, insulation, tape measure)

There are a lot of ways to insulate a building. It took a lot of reading to come up with the decision of using 75mm rigid foam insulation in the walls and 100mm of the same in the floor/ceiling.

This was picked because of its good “R” value (how good it is at keeping heat in/out) and the fact that the wood used for the wall construction was about 90mm deep and we couldn’t fit 100mm.

There are cheaper options! We spent about £1000 on insulation. You could use the fiber stuff on a roll and pay less than £200 for the room but you would boil in summer and freeze in winter!

As you can see from the picture you cut the big sheets into sections that snugly fit into the gaps between the frame.

Once you have completed the insulation blocks you need to cut out around your lights and sockets.

Then you tape over the joints, this creates a vapour barrier (keeps moisture from condensing in the wooden frame) and helps with keeping the heat in/out.

Hint: Don’t do what I did and push the insulation to the back of the holes, it should be flush with the front of the frame! I created a section there behind the plasterboard for moisture to condense! I hope it doesn’t!

Plasterboard

1-2 days (Plasterboard, plasterboard fixings, saw, stanley knife)

I went for 9.5mm plasterboard with a 12.5mm insulation backing, this is to reduce thermal bridging through the wall timbers.

Your plasterboard needs cutting to size. For the height use the full height of the room. Width needs to end exactly half way over a wall joist. This is so you can get a good fixing onto the board and wall!

The process is just cut, poke wires through holes and then screw on. The ceiling is hard because those boards are heavy!

Windows & Doors

Less than 1 day (Window, Door, screws?)

It’s a good idea to get these into the building as soon as the frame is up (or even as part of building the frame). However, the window and door I ordered had a 3 week lead time and we had to wait.

Once they arrived they were professionally installed (I think most competent DIYers could do this, but a 3m bifold door was a bit daunting for me!)

I don’t have any hints/tips on fitting these as I didn’t get involved. Here’s a picture of the doors fitted

And the window.

My Home Office Build, Part 2: Frame, Walls and Roof

sales@heyal.co.uk (Alistair Hey) — Thu, 30 Jun 2022 00:00:00 GMT

With the concrete base cured, it was time to build upwards and get the building watertight as quickly as possible.

Wooden Frame

1-2 days (screws, wood, screwdrivers, packers, bolts, socket sets, saw)

Once the concrete was dry a 4”x3” wooden frame was bolted down into the concrete. We used long concrete bolts and drilled down into the base. The picture shows this without the second damp proof course (large plastic sheet) we added. This wooden base was leveled using the long spirit level and placing small plastic packers under the wood to get a perfectly level and secure base.

Now we have a base, we can start to build the walls, we worked out what length to cut the uprights in the wall and then cut them all to the same length, once we had everything we used a long beam along the top (called a wall plate) to start to build the first wall.

You can see the second damp proof course here (black plastic sheet) that went on top of the concrete.

The corners were braced with those large diagonal lengths of wood to stop the frame falling over while we built it. Once you start to get the wall timbers screwed onto the frame you no longer need to brace the frame (you will see these later!)

You can also see 3 sheets of the rigid foam insulation we are going to use for the floor. These were put into the frame to give us somewhere to stand so we didn’t rip the plastic sheet (you need to keep it intact to stop any water getting into the building).

Once you have done all 4 walls (leaving a gap for your door and window) you can start to attach the wall panels.

Because I had decided to have a 3m bifold door on the front of the building I used 2 9”x2” timbers screwed together to form the top of the doorway. Bifold doors are HEAVY and they hang from above, so big timbers were used, bolted together with coach bolts and we also added an angle iron above the door. You will see these details in the next few photos because we moved quickly and I didn’t get time to take photos of these specifically.

Walls

1-2 days (screws, wall panels 8x4’ 22mm OSB boards, screwdriver )

We now have wall timbers we can start to screw in the large 8ft by 4ft boards onto the frame. These give good support to the walls and we made sure the room was square before screwing these up. The picture shows a few being added with overhang at the top, we cut the overhang off once the roof joists were added (next section) as there is no need to cut them before.

It’s just a case of using a lot of screws here to make sure every board is securely fixed onto the timbers.

Roof

1 day (wood, screws, screwdriver, step ladder, roofing material)

I decided to use a 1:40 pitched roof, this has a slight runoff and I picked heavy duty EPDM membrane (thick rubber sheet) for the roof material. It’s easy to install yourself and you can just lay it out without fixing it down to make your roof watertight very quickly!

This was important because it rained…. A LOT the day we installed the roof so we worked very quickly to get the roof joists, boards and the membrane on. Unfortunately I didn’t get any other photos, so this one shows all those things done!

My Home Office Build, Part 1: Groundwork

sales@heyal.co.uk (Alistair Hey) — Thu, 30 Jun 2022 00:00:00 GMT

Update, October 2026: four years after starting this post, I’ve finally finished writing it up! The build is now split into four parts, from digging the first trench to what it’s like to work in the finished office.

This wasn’t my first DIY project. Let’s not pretend that I suddenly knew how to build this without several years of increasingly more complex projects.

It all started when we moved house in August 2020. After 9 months waiting to move in (and therefore not doing any work on the house we were selling) we needed to “redecorate” our new house. This involved laying some new wooden flooring, painting and re-finishing a fireplace. These are fairly simple tasks for a “keen DIYer” and I have no doubt that anyone reading this could have a good go at these and be happy with the results.

Things got increasingly more complex from there. First it was building a window seat:

This really opened my eyes to how even more complex projects were within my skill level.

In the following months I re-built our fish pond (bricklaying) and totally refitted our bathroom. Took everything out, including the flooring, walls, shower and then re-fitted everything.

The Office

I was already very privileged to have a dedicated room inside the house we purpose fitted to be my office.

However, I like to have a project on the go!

My general plan was to build a 4m x 3m office at the end of the garden. I didn’t really have more plans than this when I started and knew I needed to lay a base as the first step.

This is the spot I picked, to the left of the row of sleepers (in the gravel). Before anyone asks, the tree had unfortunately died from a fungus and was cut down.

Getting started

Pick a spot large enough to house your building!

I’m based in the UK and the rules around garden buildings are complicated. Make sure you know what permits/permissions you need before you start to avoid having to tear down your new building!

Power & Internet

2-4 days (spade, cables, a professional electrician)

I decided to build the office at the bottom of the garden. I got my electrician to run a large armoured cable from the fuse-board in the house down to the garden

Make sure to get a professional to install any electrical cables, they need to be rated for outdoors and you may need to have your house electrics upgraded to be safe too!

Once the electrician had connected the cable to the house (and terminated it at the end of the garden in an outdoor socket so I could easily get power while doing the work) the cable needed burying!

The UK requires that external cables that are buried are around 500mm + deep and have a yellow warning tape on top with a final layer of gravel to make extra sure someone won’t accidentally dig into the cable! Here’s a picture.

I had to run the cable about 20m from the end of that building (in the photo) to the spot where the office was. THIS TOOK AGES!

Digging is hard, it probably took 2.5 full days digging this by hand. I couldn’t get any machines into the garden because I only have access to the back through a single doorway in the garage.

Along with the power cable I ran 6 CAT6E (internet) cables from the garage, through the building into the trench. This could be done by an electrician if you don’t know how! I saved a few £££ by doing it myself as I work in technology and know what I’m doing.

TIP: If you can hire a trench digging machine DO! Unless your distance is short it will save you A LOT of time.

Ground preparation

2-5 days (spade, wheelbarrow, pickaxe)

Once I had picked a spot and run the power cable the next job was preparing the ground to pour a concrete base. I scraped back all the gravel, piled it up on the other half of the gravel section and removed a layer of hardcore (which I kept for later!). You can see from the picture below that I already had a good solid edge on the back, left and front of the space, so I only needed to screw a plank to the sleepers that run towards the back. (pictured)

Once that was removed I dug about 30cm (12”) of soil out of the hole and piled it under a tree that was close by.

When the soil was removed I put out 4 corner posts and screwed 6” x 1” shuttering (thin wooden boards to hold back the concrete) using a long 6ft level to make sure it was level! The concrete base was about 20cm larger than the planned size for the building, so 4.2m x 3.2m.

Once the shuttering was level and square(ish) we put back the hardcore into the wood and roughly leveled it with a rake. Once level we used a whacker plate (a machine with a big weight on a flat base plate that bashes down) to compact the hardcore to make a really solid base.

We bought some metal mesh to reinforce the concrete to reduce the chance of cracking once dry.

Concrete

less than 1 day (concrete delivered by lorry. 2 days for self-mix )

We bought a large plastic sheet to go under the concrete as an initial damp proof course (concrete can hold a lot of water so you need to keep as much moisture away as possible or you risk getting damp in your building)

Next job was pouring concrete. There are a few ways to do this, ranging from self-mixing using a cement mixer to getting concrete delivered on a lorry and pumped into place. We had concrete delivered and pumped because we needed about 2.5 cubic meters (probably 1.5-2 days mixing with a cement mixer) We used a pump (expensive! cost about 2x the price of the concrete we pumped) because we had to move the concrete 45m from the road to the site. Wheelbarrowing this would have taken several people a few hours and I’m not strong enough for that!

Once the concrete is poured and leveled you need to wait about 3-5 days for it to set.

This is a good time to finalise your wood order and buy insulation.

First look at loft.sh using GitOps

sales@heyal.co.uk (Alistair Hey) — Tue, 15 Mar 2022 00:00:00 GMT

The problem

I spend my days building bespoke kubernetes platforms for companies. They aim for consolidation and increased standardisation. Often the companies are transitioning from on-prem to cloud and the first step on this journey for many organisations is to give every team its own cloud account.

This leads to every team choosing different technology to deploy, build, run, monitor and debug their systems. These usually range from click-ops (using the console) to fully automated systems using in-house tooling and scripts, and everything between.

You can already see the problem… Nobody can move teams without re-learning how to build, ship and run their systems. There is no standard documentation. Infosec can’t easily integrate and provide good-practice patterns and alerting across the organisation.

This then leads to a period of consolidation and building “Standards”. These unwieldy documents describe what and why things should be done. They often don’t specify HOW they should be done. The reason? Every team’s setup is different. There are 50, 100, different ways things are being done.

The Idea

Wouldn’t it be better to provide one way to do something, that one way already has the security, logging and monitoring tools built-in? In comes the need for a consolidated platform. Companies usually land on Kubernetes because it can be “Multi-cloud” (A topic for another day).

The problem with this is building it. Unlike with the cloud where your provider manages the day-to-day operations of your underlying platform (from managing disk damage with servers to managing the entire platform like with serverless), Kubernetes requires you to build, customize and run it.

The additional challenge is that Kubernetes as a platform is complex, infinitely customizable and expensive (because the people who know how to build and run these systems are in high demand!)

The solution

I have spent several years building bespoke PaaS offerings on kubernetes for large organisations, think 100s-1000s developers and 10s-100s of teams.

Every time there is a base level of “stuff” that needs building/configuring to run even a basic multi-tenant cluster.

Network Isolation
Authentication
RBAC
Tenant Isolation
Policy Enforcement
Resource limits
Financial reporting (cost allocation)

The list goes on… but you get the idea. These things take time (and therefore money) to build and configure.

As Kubernetes use becomes more widespread there needs to be a better way to lay down these building blocks. I’ve been thinking for a long time that this should be packaged up and productised. We shouldn’t be building the same things every time we want multi-tenant kubernetes!

Looks like loft.sh have done just that. I’m going to go through installing loft.sh on a new kubernetes cluster with Flux using GitOps techniques, just like I would do for work.

Getting started

Get authenticated with a new Kubernetes cluster, I’m using a Civo k3s cluster.

Install the Flux CLI using their instructions, and set up your GitHub token and user.

Once configured this should show a similar output

$ flux check --pre
► checking prerequisites
✔ Kubernetes 1.22.2+k3s1 >=1.20.6-0
✔ prerequisites checks passed

Then we can install Flux2, which should create a github repo, install the flux controllers and start reconciling against that repo. This means changes we make in that repository will be reflected in the cluster.

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=loft-resources \
  --branch=main \
  --path=./clusters/dev \
  --personal

The bootstrap command above does the following:

Creates a git repository loft-resources on your GitHub account
Adds Flux component manifests to the repository
Deploys Flux Components to your Kubernetes Cluster
Configures Flux components to track the path /clusters/dev/ in the repository

Now we can clone that repository

git clone https://github.com/$GITHUB_USER/loft-resources
cd loft-resources

Once inside that repository we can create a Helm source to add the loft Helm repo to our cluster.

flux create source helm loft \
  --url=https://charts.loft.sh/ \
  --export > clusters/dev/loft-source.yaml

Next we need to set some Helm values and install the loft chart, create the following file at loft-values.yaml replacing your email appropriately.

admin:
  email: demo@example.com

flux create hr loft \
  --target-namespace=loft \
  --create-target-namespace=true \
  --source=HelmRepository/loft \
  --chart=loft \
  --values=loft-values.yaml \
  --export > clusters/dev/loft-release.yaml

If we add everything to git and commit


git add -A && git commit -m "Add Loft chart" && git push

we should then see that loft is installed!

Install the loft cli from the downloads page and re-set the admin password. We need to do this because we didn’t pass in a password when generating the release, the reason is that we would have to check-in the password.

We could have created a secret with the password and passed it into flux’s helm release as a “values-from” argument, but for simplicity we didn’t.

# run this and follow the prompt
loft reset password

We can login by port-forwarding the loft service

# port forward, then go to localhost:9898 and login with your email and password
kubectl port-forward service/loft 9898:443 --namespace loft

navigate to https://localhost:9898 and login

Conclusion

This was a basic installation of loft.sh using Flux, we could now use the loft provided CRDs to build a more complex multi-tenant system. All our config is stored in Github and we could apply this same installation to another cluster or another 10 clusters very quickly!

Next steps

Start to build out templates for teams to get pre-configured environments at the click of a button! Loft provides a set of CRDs to customize every aspect of the installation, a good start would be creating some app templates to deliver pre-configured applications into the cluster.

How to Unit Test Helm Charts with Go

sales@heyal.co.uk (Alistair Hey) — Sat, 23 Jan 2021 00:00:00 GMT

Learn how to write Golang unit tests for your Helm charts to keep quality high and make changes with confidence.

In both my open source and commercial work I have needed to write helm charts. These charts are a definition of how to install an application on kubernetes. They often include configuration options like if we should enable auth, clustering or specific database options.

These options usually boil down to some “logic” or code within the helm chart. Here is an example of a configuration option being used to set a flag.

{{- if .Values.auth.enabled }}
  - --enable-auth
{{- end }}

This is a basic example and often these logic statements can be much more complex and have larger impacts to many different parts of the chart.

How do we test this?

The software developer in me wants to write tests for this. If I pass the auth.enabled=true flag when calling helm install I want to be sure that my above logic generates the expected outcome.

I did a lot of searching before embarking on my own solution for testing this type of logic at a unit test level. Most of the options I found were for an integration level test, such as the official helm website. This level of testing is great, but can be slow to test out the potentially hundreds of options and flags we may want to set in our chart.

So, I set out in search of a way to define my unit tests in code. I wanted to define what I expected the helm command to generate and then compare what was actually generated.

The code

I chose Golang - partly because I use golang every day, and partly because Helm is written in go. This means we can potentially import parts of the helm codebase into our test library and avoid “re inventing the wheel”.

The first place to look is the code for the helm CLI binary itself; we want to be able to leverage the existing implementation so our tests match our real world use.

This snippet, adapted from the helm template command, is a good place to start:


func Template(name, namespace, chartPath, filePath string, valueFilePaths, overrideValues []string, output interface{}) error  {
    // Some setup code omitted for readability
    ...

    // This is where the magic happens, This takes the values and the template and runs the helm logic
	release, err := client.Run(chart, values)
	if err != nil {
		return err
	}

    // A function to split the output of the template, (A single array of all files in 1 variable)
    // it returns a map of filename to  contents
	manifests := splitChart(release.Manifest)

	if _, exists := manifests[filePath]; !exists {
		return fmt.Errorf("no file found at path %s", filePath)
	}
    
    // convert the returned YAML to JSON, all internal kubernetes communication uses JSON
	jsonBytes, err := yaml.YAMLToJSON(manifests[filePath])
	if err != nil {
		return err
	}
    // Unmarshal the file (as json now) into a struct, We will see how we use k8s definitions in the test to reduce 
    // code even more!
	if err = json.Unmarshal(jsonBytes, &output); err != nil {
		return err
	}

	return nil
}

This allows us to write tests where we define the struct we expect our file to match, and then compare that expected value with what we get. Below is an example using a simple ingress record.

func Test_IngressWithTLS(t *testing.T) {
	want := v1.Ingress{
		TypeMeta:   metav1.TypeMeta{
			Kind:       "Ingress",
			APIVersion: "networking.k8s.io/v1beta1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:                       "helmChart-nginx-example",
			Labels:                   map[string]string{"app.kubernetes.io/instance": "helmChart", "app.kubernetes.io/name": "nginx-example"} ,
		},
		Spec:       v1.IngressSpec{
			Rules: []v1.IngressRule{
				{Host: "chart-example.local" , IngressRuleValue: v1.IngressRuleValue{HTTP: &v1.HTTPIngressRuleValue{Paths: nil}}},
			},
		},
	}
	var got v1.Ingress

	if err := Template("helmChart",
		"default",
		"./chart",
		"template/ingress.yaml",
		[]string{"./chart/values.yaml"},
		nil,
		&got,
	); err != nil {
		t.Fatalf("got an error templating chart: %v", err)
	}

	if !reflect.DeepEqual(want, got) {
		t.Fatalf("difference in chart and expected. Got:\n%+v\nWant:\n%+v\n", got, want)
	}
}

We can now describe what we expect our chart to produce in our code, we can add logic to generate sections, like if we enabled TLS in the values with --set then we could add a section in our want struct to match what the helm template logic should generate.

When we make changes to our chart we can then use our tests to validate that we don’t make any other unintended changes. For example we could update an image version in our base values.yaml, run our tests and see that our deployment now fails because we didn’t update our tests to use the new image.

The Upsides

With a good suite of unit tests written we can be confident that when we make changes to our chart that they have the desired effect. It’s hard having to manually verify the changes each time and this is time we could better spend developing more features!

This type of testing is particularly helpful for new members of the team, they can make changes with confidence and start having an immediate impact. Additionally, when reviewing helm chart changes we get a version of the change written in a more familiar and strongly typed language. This should reduce the time taken to digest the changes and make informed decisions when reviewing your team’s PRs.

The downsides?

It wouldn’t be fair to not talk about a few possible downsides when showing off this technique. Using this approach (using go code from helm) means vendoring large chunks of the k8s.io repos and pulling in helm. This is a lot of dependencies we have just added to our codebase. The other alternative is to write a go wrapper that executes the helm template binary with the desired values, this removes the need to vendor k8s/helm code. However, it does mean we would need to write our own version of the v1.Ingress structs. I have chosen to vendor the code because it makes writing tests quicker as we don’t need to keep adding to our own type definitions of the kubernetes objects.

Full example

I have written a full example with a basic helm chart in this repo

This is my second attempt at unit testing Helm charts. I took a similar approach when writing the OpenFaaS Cloud helm chart. Last time I went with the “DIY” approach and wrote the type definitions and a wrapper around the helm binary. You can check out the implementation at the link.

Let me know what you think on LinkedIn. I’d be keen to hear whether you’re unit testing your Helm charts, and whether you see the value in this approach.

OpenFaaS, what's it all about?

sales@heyal.co.uk (Alistair Hey) — Tue, 12 May 2020 00:00:00 GMT

Serverless should let you focus on delivering features into production fast not fiddling with infrastructure.

If you spend some time on the internet you should have seen that kubernetes is everywhere. It’s a great platform for abstracting your infrastructure away from your developers and letting them focus on code.

When it comes to deploying serverless on Kubernetes I am of two minds. Part of me thinks it’s not “serverless” because you have to (at the very least) manage worker nodes (This is not strictly true, you could use AWS Fargate backed EKS Clusters?).

The other part of me thinks, well you don’t have to care about the nodes. We could call it serverless. For example using managed Kubernetes with autoscaling worker groups using the cloud’s default VM images? You probably would never need to bother looking at the instances, let alone connecting to them.

Why bring in another abstraction with OpenFaaS

OpenFaaS abstracts away some of the complexity of running applications on kubernetes if you want to focus on delivering value to your paying customers, who don’t care how your platform is run as long as it works.

If you need to run application code, you can run it here. There are community maintained templates for languages from Rust and Go to COBOL. If you can’t find a template that quite hits the spot you can build and customise your own to provide exactly the base application configuration that works for you.

Let’s look at this from a couple of angles. I’m going to put my Developer and Platform engineer hats on for a minute…

Developer

You probably don’t really mind where your code gets run, as long as you can iterate on features quickly and deploy new services without fuss.

I’m going to let you into a secret, you need to write A LOT of YAML to deploy a service into kubernetes, most of it will be the same, or very similar.

In this case using plain old YAML manifests hand crafted for each service probably isn’t your bag.

It takes 2 commands to deploy a service into OpenFaaS.

$ faas-cli new --lang python3 python3-fn --prefix <dockerhub username>

$ faas-cli up -f python3-fn.yml

To get the same container deployed into Kubernetes using YAML manifests you would need:

A Service (If you are going to scale your deployment) (20+ lines of YAML)
A Deployment (50+ lines of YAML)
An Ingress Definition (20+ lines of yaml)
A Dockerfile (20+ Lines of Dockerfile)

Do you really want to get stuck maintaining all of these for each service you are running?

In addition to this you can easily install and run OpenFaaS on your laptop using docker swarm or a local Kubernetes cluster.

I like using k3d for local testing. I can spin up a single node cluster inside docker install OpenFaaS and route traffic to my local machine in a few commands.

$ k3d create
# Wait for that to finish, It will output a command to export KUBECONFIG

$ export KUBECONFIG="$(/usr/local/bin/k3d get-kubeconfig --name='k3s-default')"


# I'm using arkade to install openfaas with default config https://github.com/alexellis/arkade
$ arkade install openfaas


# When thats deployed (all pods running, check with kubectl get pods -A)
$ PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" \ 
    | base64 --decode; echo)

# Port forward the "gateway" to your localhost
$ kubectl port-forward -n openfaas svc/gateway 8080:8080 &

# Login with the CLI
$ echo -n $PASSWORD | faas-cli login --username admin --password-stdin


# Deploy your stack, which by default should be in stack.yml
$ faas-cli up

It would be easy to include this in your CI pipeline. For example the Kubernetes provider for OpenFaaS faas-netes uses minikube to run OpenFaaS during CI.

Platform/Ops engineer

You want to provide a stable, simple to use and easy to debug platform for developers to maximise productivity.

If you are on a small team or don’t have bags of kubernetes experience then leveraging a platform like OpenFaaS that abstracts away some of the complexity for you and your users seems like a good idea.

If you want to focus on delivering a stable scalable platform for your services you can leverage the many thousands of hours of community knowledge and experience that has gone into building OpenFaaS. If you wanted to build a custom solution you would no doubt end up hitting some of the same problems that have already been overcome by the OpenFaaS contributors. Why go to all that effort yourself?

My journey with OpenFaaS

First look

My first experience with OpenFaaS was during a company “hack day” where you put aside your normal work and get into new teams and experiment with new technologies and processes. Our team decided to look into open source serverless technology as an alternative to your favourite cloud vendor serverless technology.

We were trying to find a way of overcoming the difficult challenge of testing serverless deployments End to End in a cost effective, fast and easy to test way.

I have always found services like AWS Lambda too hard to test, run locally and debug without deploying the whole stack into Amazon and trying things until it works. Our feedback loop for an End to End test on a AWS based serverless stack was in excess of 30 minutes! Not to mention the costs for deploying and running on the cloud.

Needless to say, within a few days we had written a serverless application on OpenFaaS. We had a range of languages in use for our functions (we all picked our favourites!), and we could all run the stack locally for testing.

Where am I now?

I’m part of the community and a regular contributor to OpenFaaS in my spare time.

(sorry everyone else who didn’t fit onto that screenshot!)

I love OpenFaaS! I think it’s the perfect platform for developing cloud native applications. It keeps its focus firmly on providing developers with the best tools and workflows to get things done.

I may be slightly biased: I contribute a lot of code to the OpenFaaS project! At the time of writing I was the 9th highest contributor by commits.

Come and join the party!

At the time of writing there were more than 2,500 of us in the OpenFaaS community, and 20K+ stars on GitHub!

What can you do to get involved? Well lots of things!

OpenFaaS has a great community - find out how to join here
Check out the OpenFaaS Website and blog
Head over to Github and run through the getting started Workshop
Follow OpenFaaS on Twitter
Support the project’s maintainer Alex Ellis by sponsoring him on Github!

Introducing Arkade - The Kubernetes app installer

sales@heyal.co.uk (Alistair Hey) — Fri, 28 Feb 2020 00:00:00 GMT

Arkade! What is it?

Arkade (ark for short) provides a clean CLI with strongly-typed flags to install charts and apps to your cluster in one command.

It’s written in Golang and supported by Alex Ellis and the OpenFaaS Community.

I added the cert-manager, openfaas-ingress, kubernetes dashboard and docker-registry-ingress apps! Big thanks to Alex and the rest of the community for contributing all the other amazing apps.

What’s it for?

Installing apps on Kubernetes, this is a list of the currently supported apps:

$ arkade install 

You can install: 
 - openfaas
 - nginx-ingress
 - cert-manager
 - openfaas-ingress
 - inlets-operator
 - metrics-server
 - chart
 - tiller
 - linkerd
 - cron-connector
 - kafka-connector
 - minio
 - postgresql
 - kubernetes-dashboard
 - istio
 - crosspane
 - mongodb
 - docker-registry
 - docker-registry-ingress
Run arkade install NAME --help to see configuration options.

Wow! That’s a lot of apps!

Let’s take a look

The arkade cli is the easiest installation method for OpenFaaS on kubernetes. I am going to show you just how easy it is to get OpenFaaS installed and ready to work using this cli.

I have a Kubernetes cluster on DigitalOcean ready to go, with kubectl configured on my laptop.

We could install the basic default configuration like this:

$ arkade install openfaas

However, I’m going to set an override for the Helm installation, so our functions scale down to 0 when they are idle, saving cluster resources. In the full helm command we would use --set faasIdler.dryRun=false, as you can see from the command below we just pass the same flag in. Easy as that!

$ arkade install openfaas --set faasIdler.dryRun=false

That should output the below message, showing it worked!

=======================================================================
= OpenFaaS has been installed.                                        =
=======================================================================

# Get the faas-cli
curl -SLsf https://cli.openfaas.com | sudo sh

# Forward the gateway to your machine
kubectl rollout status -n openfaas deploy/gateway
kubectl port-forward -n openfaas svc/gateway 8080:8080 &

# If basic auth is enabled, you can now log into your gateway:
PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
echo -n $PASSWORD | faas-cli login --username admin --password-stdin

faas-cli store deploy figlet
faas-cli list

# For Raspberry Pi
faas-cli store list \
 --platform armhf

faas-cli store deploy figlet \
 --platform armhf

# Find out more at:
# https://github.com/openfaas/faas

Thanks for using arkade!

That’s a lot deployed in one small command. We now have OpenFaaS installed in our cluster, ready for us to deploy serverless functions, microservices or containers.

Note: You don’t even need to remember all that information. Arkade has a command to print the info for any app again!
$ arkade info openfaas

With two more arkade commands we can set up TLS and ingress from the internet:

$ arkade install cert-manager --wait
$ arkade install openfaas-ingress \
    --email example@example.com \
    --domain openfaas.example.com

The wait flag can be used to wait for the helm chart to install and be ready before moving on to the next command.

So now we have OpenFaaS, cert-manager and some ingress records set up. Once DNS points at this cluster, we get a Let’s Encrypt TLS certificate, so our installation is now secured on the https://openfaas.example.com domain.

Now that is actually it. We have OpenFaaS installed and secured with TLS in three commands, but let’s go through an example function to validate the installation.

Let’s grab the OpenFaaS CLI:

$ curl -sSL https://cli.openfaas.com | sudo sh

# We need to login, so let's grab our password and login
$ export OPENFAAS_URL="https://openfaas.example.com"
$ PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
$ echo $PASSWORD | faas-cli login --username admin --password-stdin

Now we can deploy an example function

$ faas-cli store deploy figlet

# And test it out with

$ echo "IT'S WORKING" | faas-cli invoke figlet

 ___ _____ _ ____   __        _____  ____  _  _____ _   _  ____ 
|_ _|_   _( ) ___|  \ \      / / _ \|  _ \| |/ /_ _| \ | |/ ___|
 | |  | | |/\___ \   \ \ /\ / / | | | |_) | ' / | ||  \| | |  _ 
 | |  | |    ___) |   \ V  V /| |_| |  _ <| . \ | || |\  | |_| |
|___| |_|   |____/     \_/\_/  \___/|_| \_\_|\_\___|_| \_|\____|

Where has it come from?

It was spun out of the k3sup codebase to allow the k3sup cli to be lean and focused on creating k3s clusters.

While the k3sup app install commands work on any Kubernetes cluster, there were some people who wouldn’t even look at its functionality because they thought it was just for k3s clusters. It’s not! It installs helm charts and applications onto any kubernetes cluster using strongly typed flags and sensible defaults, while still allowing users to set the more obscure helm flags when they need to.

This new direction will allow the arkade cli to be more focused on providing cluster admins with the abstractions to let them move faster with more safety.

Get involved

Arkade is on github, so pop over and give it a star to show your support.

You can find the developers through the OpenFaaS community to talk about new features, how you can contribute and more.

Exploring NAT Traversal and Tunnels with Inlets and Inlets Pro

sales@heyal.co.uk (Alistair Hey) — Mon, 20 Jan 2020 00:00:00 GMT

Inlets! What is it?

inlets is a reverse proxy and service tunnel written in Go.

This is the overview from the inlets GitHub README

inlets PRO replaces SSH tunnels, VPNs, SaaS tunnels, port-forwarding and dedicated connections.

It’s compatible with any HTTP (L7) or TCP (L4) software and can work through the most challenging network conditions like captive portals, HTTP proxies, firewalls and NAT to give you access to services.

You can run the tunnel as a process, container or Kubernetes Pod, and it’s easy to configure or automate it.

This means you can use it to create inbound network tunnels to computers that don’t have a public IP, are behind firewalls or get assigned new IPs frequently. Things like your laptop when you move from the office to a coffee shop. While writing this post I have probably been on 5 different networks, but still been able to connect to my laptop’s k3d cluster using the same public IP address.

Why inlets

inlets has a place on the Cloud Native Computing Foundation (CNCF)’s technology landscape. It’s in the Service Proxy category along with MetalLB, Traefik, HAProxy and Nginx.

There are plenty of freemium services out there that will give you inbound network access to processes that don’t have inbound connectivity, but most of these services are in control of your data, rate limit and are closed source. Inlets gives you back control of your data, privacy and lets you run the service in the way you want.

Services like ngrok and Cloudflare’s Argo Tunnel are closed source and have service restrictions, and you are not fully in control of them.

With cloud VMs starting at $5 with heaps of inbound data transfer there’s no need to pay 10s of dollars a month for someone else to run a limited service for you.

How it works

Inlets creates a tunnel from your local computer to a public cloud instance (or anything that can accept inbound traffic). I have been using DigitalOcean’s smallest VMs as my exit nodes. However, you can set up inlets using anything that can be accessed from both the internet, and the application or process you want to expose.

inletsctl

There is a handy CLI for managing your inlets services called inletsctl. I will be using this cli throughout the post as it simplifies the process of getting things set up and configured.

 _       _      _            _   _ 
(_)_ __ | | ___| |_ ___  ___| |_| |
| | '_ \| |/ _ \ __/ __|/ __| __| |
| | | | | |  __/ |_\__ \ (__| |_| |
|_|_| |_|_|\___|\__|___/\___|\__|_|


inletsctl automates the task of creating an exit-node on cloud infrastructure.
Once provisioned, you'll receive a command to connect with. You can use this 
tool whether you want to use inlets or inlets-pro for L4 TCP.

See also: inlets-operator which does the same job, but for Kubernetes services.

Usage:
  inletsctl [flags]
  inletsctl [command]

Available Commands:
  create      Create an exit-server with inlets PRO preinstalled.
  delete      Delete an exit node
  download    Downloads the inlets PRO binary
  help        Help about any command
  kfwd        Forward a Kubernetes service to the local machine
  update      Print update instructions
  version     Display the clients version information.

Flags:
  -h, --help   help for inletsctl

Use "inletsctl [command] --help" for more information about a command.

The create command automates the provisioning, setup and configuration of the publicly accessible exit-node using cloud instances.

Currently supported providers:

Digital Ocean
Equinix metal
AWS EC2
Scaleway
Civo
Google Cloud
Linode
Hetzner

You will need your own account and will incur costs with whichever provider you use if you are following along.

There are some great blog posts already about setting up inlets without the CLI, such as:

Setting up an EC2 Instance as an Inlets Exit Node - Matt Bacchi

Therefore I won’t be focusing on this, instead we will focus on the why and the what of inlets.

Before you get started

It’s probably best to say that if you are using a corporate network, or any network you aren’t responsible for, you should ensure you have permission to start routing inbound internet traffic. It goes without saying that this exposes the network to extra risk and could drastically increase the network attack surface.

inlets-operator

I’m using Kubernetes and can’t forward every deployment, that would take ages…

Not to worry, the inlets-operator has your back.

The operator automates the creation of an inlets exit-node on public cloud, and runs the client as a Pod inside your cluster. Your Kubernetes Service will be updated with the public IP of the exit-node and you can start receiving incoming traffic immediately.

The operator has a more specific use case: running Kubernetes somewhere without inbound network traffic, like k3s, k3d, minikube, MicroK8s, Docker Desktop or KinD.

It’s worth exploring if you need to expose your local kubernetes services, for example to give a demo or to collaborate.

People LOVE reverse proxies

My most viewed tweet so far is this

I’m not entirely sure why people love tunnelling traffic into networks so much, especially where it’s specifically disallowed (corporate networks!).

Why Inlets Pro?

According to the GitHub README inlets-pro is an L4 TCP load-balancer.

OK, thanks.

For those of us that can’t remember the entire network stack and what each layer does or is (humans?), this roughly translates as delivers the correct data to the correct application process over the network.

So putting that in the context of inlets-pro, it forwards TCP traffic over an encrypted websocket secured with TLS to your application process. Cool!

These are some of the features

Support for any TCP protocol
Automatic TLS encryption for tunnel and control-port
Pass-through L4 proxy
Automatic port detection, as announced by client
systemd support and automatic retries
Kubernetes compatible

Enough talk - time for action

Setup

Inlets

We have already briefly looked at inletsctl, but we didn’t discuss installation or setup.

All of the commands shown are for linux and will probably work on Mac too. If you are using Windows then you should probably use WSL.

Download and install inlets, inlets pro and inletsctl

# Install Inletsctl to /usr/local/bin
curl -sLSf https://inletsctl.inlets.dev | sudo sh

# Install Inlets Pro
sudo inletsctl download

Get your Inlets license by buying a subscription here

I’m using Digital Ocean for my exit nodes, you need to grab your Digital Ocean API Key if you are following along. I have saved mine in a file called do-token in my home directory.

If you want to use one of the other providers you can check the examples in the inletsctl github repo.

Then we can create an exit node


inletsctl create --access-token-file ~/do-token

You should have some similar output to above, this is showing that we created a node and it’s ready to be configured to forward traffic. It’s even helpfully given us the commands we need to run to get something configured.

If you have docker installed you can do a quick test to see if it’s working.

docker run --name nginx-test -d -p 8080:80 --rm nginx:latest 

export LICENSE="your inlets pro license"

export UPSTREAM=http://127.0.0.1:8080
inlets-pro tcp client --remote "ws://157.245.37.54:8080" \
      --token "a-long-random-string" \
      --upstream $UPSTREAM \
      --license $LICENSE

You should have some output in your terminal that shows inlets client connecting to your exit server. You can now navigate to the public IP of your exit node. If you see the nginx splash page then we are all set up!

If that worked you can stop the docker container

docker stop nginx-test

Now we have validated our setup we can get into some real use cases!

Broadly my use cases are split into HTTP traffic, and non HTTP traffic. The example commands to forward these two categories of traffic are the same, so I will show some example uses and then the commands needed to get inlets or inlets-pro working.

HTTP based services

Webhooks

While developing OpenFaaS Cloud we relied on webhooks to let us know when new code has been pushed, and therefore when to run new builds.

This is easy with inlets, we just forward the webhooks through an inlets exit-node to our local machine, this means we can build our services locally without having to push containers up to registries and pull them back down in a remote kubernetes cluster. This saves us loads of time and bandwidth.

Exposing temporary services

Let’s suppose you are working remotely on a set of microservices, your colleague wants to test their new front-end against your backend service, this means committing, raising a PR and getting code into a development or staging environment to properly test. Alternatively they could pull your code, build and deploy that service themselves.

Problem is, both of these are a hassle. We like things to be easy! You get some free user testing, potentially catch a bug before your code even makes it into PR, your colleague gets a service provided by the person building it, without having to do anything. Sounds good.

All you need to do is expose your service using inlets. The commands to do this are later in the section.

Demos

It’s always cool to give access to the demo services and websites you are making to the “users”, this could be the product owner, client or your business partner.

Without something like inlets you would need to deploy your application up to a machine that had a public IP, configure the service and then remember to keep that up-to-date. You would also need to go off to that machine to get logs. Having this instance running costs you money and takes away valuable time during the early product development stages.

Using inlets we can expose a local service to the internet in less than a minute, and when we are done we can tear everything down again, meaning we can create on-demand tunnels, only pay for what we use and spend more time where it matters, creating value for our customers.

I can deploy a local copy of my blog using docker-compose, that updates when files change and allows me to see what my changes will look like without publishing to the rest of the world.

The commands

Let’s look at the last example above, my personal blog in development mode.

I can expose my service to http://127.0.0.1:4000 by running docker-compose up. This stands up a container that publishes to port 4000 on my local machine.

Using the exit-node we created earlier we can set up an inbound network route to my blog like this

export UPSTREAM=http://127.0.0.1:4000

inlets client --remote "ws://157.245.37.54:8080" \
      --token "a-long-random-string" \
      --upstream $UPSTREAM \
      --license $LICENSE

My local service will be accessible at the remote ip. That’s it, all that’s needed.

You can substitute the port on the UPSTREAM environment variable if your service is using a different port, 5000 for example:

export UPSTREAM=http://127.0.0.1:5000

Then we just run the inlets command again.

Non-HTTP based services (inlets pro)

Home Lab

I didn’t know if I should have put Home Lab under HTTP services, but I decided it probably lives under the inlets-pro section, I personally use mainly HTTP, but do have an SSH Tunnel open from time-to-time to allow configuration of the resources from outside my home network.

You might be collaborating with someone who doesn’t have their own resources, or computer with particular chipset. So it’s useful to be able to open these up to trusted parties via the internet.

Here’s an example from the inlets author

IoT

You could open a reverse tunnel to configure, update and change the software running on IoT Devices no-matter where they are, provided they have outbound network connectivity.

This could be a really good way to deliver updates to IoT devices.

Bastion hosts

Sometimes you need to get access to resources in private networks. This could be a database cluster or your kubernetes nodes for example. A common pattern is to have “SSH Bastions”. These are machines that live within the network but have an inbound network route. This allows you to ssh onto these machines and then jump into internal parts of your network.

Access to these machines is usually controlled with IP allow-lists and SSH keys.

We could add an extra layer of security into the mix here by not having any inbound network access to the machine. This limits the discoverability of the service. If you have ever run SSH on a public IP you will be inundated by bots trying weak username/password combinations.

This would look like this:

An example, SSH

So to set up TCP forwarding we can use inlets-pro and inletsctl. I will be running through a simple SSH example, but this could be applied to any of the non-HTTP examples in the same way.

I’m going to run the client on one of my Raspberry Pi computers with ssh enabled using key pairs only (here is a Digital Ocean post on setting this up)

Once we are logged into our Pi on our local network we can set up the inlets-pro client to enable ssh access from the IP of our exit node. We also need to change the default SSH port away from 22, as our exit-node will have ssh mapped to this port and we won’t be able to use it.

Here is an article about changing the default ssh port I used 2222, and will refer to that from now on.

# Set this to the IP of the machine you want to expose on the web
export RPI_IP=192.168.0.99 

inletsctl create --access-token-file ~/do-token --remote-tcp $RPI_IP

This gives you a command that looks like this to run on our client to forward the traffic

export PORTS="2222"
export LICENSE="YOUR INLETS_PRO_LICENSE"
inlets-pro tcp client --connect "wss://178.62.88.252:8123/connect" \
    --token "a-long-random-string" \
    --license "$LICENSE" \
    --ports $PORTS

We need to forward port 2222, so set this in the PORTS variable. Don’t forget to set your inlets-pro licence key too.

The last step is starting the client on our Raspberry pi. SSH on there from your computer and run the command!

If we have set up SSH correctly, and everything works as expected we should be able to ssh onto our Pi using the public IP

ssh -p 2222 pi@178.62.88.252

This example should work for any service that uses TCP. You will just need to edit the PORTS environment variable and re-run the client command to forward different ports.

Have a go yourself

Check out each of the tools we have looked at on github

inlets-pro - the TCP tunnel
inletsctl - The cli tool we use to speed up provisioning servers

I’m sure there are loads of other things we could do with this. I would love to hear from you if you have any more great use cases. Get in touch or find me on LinkedIn.

Docker Birmingham January 2020: Multi-Architecture Images with buildx

sales@heyal.co.uk (Alistair Hey) — Wed, 08 Jan 2020 00:00:00 GMT

Every two months a group of us host a meetup focused on Docker. You can find the meetup page here.

Seeing as I have your attention

We are hosting February’s CNCF Birmingham meetup on Wednesday 5th Feb and whilst we haven’t finalised the speakers yet, it looks like it will be a good event! Hopefully we will see you there too.

Building Docker images for multiple architectures

"Raspberry Pi Compute Module" by lespounder is licensed under CC BY-SA 2.0

This month, Docker Birmingham’s own Matt Todd was presenting about building docker images for multiple cpu architectures.

Matt had received a Jetson Nano by Nvidia over the holidays and wanted to get Elastic’s packetbeat set up on his home network to understand what data was flowing about.

When he went to run the docker container on his new kit it wouldn’t start, the process kept dying. If you have ever tried running containers on a Raspberry Pi, for example, you might have found that some containers just don’t work, even though running the same command on your laptop, desktop or cloud instance works first time. This is most likely due to the different CPU instruction sets, and code has to be compiled differently for the different architectures.

Docker images and CPU Architectures

Let’s take the nginx container for example. If we look on Dockerhub at the latest tag we can see that it is published with multiple different images all pointing at the “latest” tag.

This is because the maintainers of nginx have built the image for these different architectures all of these are the same version, same source code, but they have produced different images (the digests are different).

Docker cleverness

So why does it “just work” when we type docker run --rm -p 8080:80 nginx:latest on a Pi4(arm64) , our laptops(amd64) or other computer.

The docker daemon basically does a “best effort” to go off to your registry, finds the image you want and matches it to your specific CPU architecture. So in my case it went off and found this image

If you were running the same command on a Pi, you would have got a different image. The one published for the specific ARM cpu in your Pi. This image would (well, should) be functionally equivalent to the image pulled by an amd64 computer.

How can we build these images?

One common way to support multiple architectures when using docker containers is to publish images with different names depending on which architecture they are targeting, this is a decent approach if you are using a language which you can compile, using flags or settings, to create a binary that is specifically tied to one platform. The language I use that does this is Go. You give it some flags and can build for a whole host of OS and CPU types.

Matt introduced us to an experimental feature in docker called buildx which, through the docker cli, provides full access to the features of Moby Buildkit.

Matt then showed us how, using Docker for Mac, you can enable this experimental feature and use it to create build environments for a selection of supported architectures. This is provided through the use of an emulator called QEMU.

This emulator provides the environments for buildkit to build these different images, all under the same tag, rather than the first example where we create different image names or tags.

Using buildx

I’m not going to show you how to enable buildx, that’s a bit out of scope for this post. We will explore how it can be used though.

If we wanted to build this snippet of Go code, which uses the GOOS and GOARCH values, which are set at compile time, to print out the cpu arch and OS we used to compile we could use the commands below.

package main

import (
	"fmt"
	"os"
	"runtime"
)

func main() {
	hostname, _ := os.Hostname()
	fmt.Printf("Hello World, I'm %s running on %s/%s\n", hostname, runtime.GOOS, runtime.GOARCH)
}

To build this using the buildx feature

# assuming contexts node-amd64 and node-arm64 exist in "docker context ls"
$ docker buildx create --use --name mybuild node-amd64
$ docker buildx create --append --name mybuild node-arm64


# We now have the buildx contexts setup, we can build multiple architectures like this!
$ docker buildx build --platform linux/amd64,linux/arm64 -t dockerbirmingham/hello-world:latest .

Matt did this live, and pushed the following to Dockerhub

If you ran docker run --rm dockerbirmingham/hello-world:latest on your arm64 computer, you should see it pulled the arm64 image, and amd64 should have pulled on an amd64 computer!

Wrapping up

That’s quite neat. It should allow projects building for multiple architectures to build container images more quickly and with easier use. It’s annoying to have to keep 2 or 3 copies of docker-compose files, or helm charts just because we pushed images with different names, or tags, for these differences.

Let’s just hope it makes it into stable, and is well supported on linux (docker for Windows & Mac have it packaged in the VM they use to run docker). Matt said it was a bit harder to use on Linux because docker runs natively rather than in that VM.

Matt will be publishing his slides and dockerfiles etc. here so you can run through them for yourself.

Thanks Matt!

Civo Kubernetes KUBE100: First Look at Managed k3s

sales@heyal.co.uk (Alistair Hey) — Tue, 07 Jan 2020 00:00:00 GMT

The World’s First Managed K3s

When Civo announced they were looking for Beta testers for their new managed Kubernetes offering I obviously jumped at the chance. Free compute - never turn that down! (Thanks Civo)

KUBE100

Civo have been running a beta for their offering, it’s called KUBE100, you can request to join here

It’s been great so far, there have been a few teething problems. They have all been resolved well, that’s the point of a Beta, and Civo have been managing the process really well through a community slack group that you get access to when you are accepted for the beta.

Everyone is having a blast, or at least it appears that way, so well done & thanks Civo!

Civo cli

Civo provide a cli to manage clusters (as well as their other services). It’s great, works well and is even packaged in a docker container for those of us who don’t have Ruby installed. With this it is easy to get the status of clusters, create new ones and even install marketplace apps.

What is K3s?

According to Rancher, who took Kubernetes and made it smaller, it’s a “Certified Kubernetes distribution built for IoT & Edge computing”. You can check out the k3s web page here for more information.

What does this actually mean though? Well, it is small, really small. The master node process runs in under 500mb ram, and worker nodes about 50mb. It’s packaged for ARM64 and ARMv7 (think Raspberry Pi) and you can easily build installation scripts (or tools) that can detect the CPU architecture. This allows you to spin up clusters on anything from your home raspberry pis to large bare-metal servers.

It’s fast. Civo can spin up a k3s cluster for you in under 2 minutes. We will explore more on this later.

Rancher do a better job at explaining all of the features and benefits in the docs. We won’t dwell on this much more here; there are lots of great resources on this already.

How is k3s different from k8s?

That’s a good question. Don’t immediately expect that you can deploy your application onto k3s without having to do some extra work. If you’re coming from a managed kubernetes service provided by one of the large cloud providers you get a lot of integrations out of the box.

For example, the first thing I did was to install the gitlab helm chart, which sat in pending forever. This is because there was no default storage provider for the persistent volumes required by many apps. Not to worry though, Civo provide a 1 click Longhorn installation which can provide the persistent volumes required.

Andy Jeffries, the Civo CTO wrote a good blog post on K8s vs k3s which is well worth a read if you have the time.

The Civo Community

I mentioned the Civo slack earlier, which is great for asking questions about their k3s offering, but it has delivered much more than just a beta support channel to Civo. I have seen the group help each other out more times than I could count. Civo have done a great job of engaging the developers that have offered to beta test the service.

This makes sense, there’s a large section on their website about how they are community focused. This doesn’t always mean that companies with this attitude actually follow through on this and deliver their vision.

Civo have definitely focused on this, and you can tell.

Features provided by Civo

The managed k3s service provided by Civo is great, here are my highlights:

Marketplace apps
DNS entry for master node, for when your nodes are recycled
Civo CLI
Speed of cluster creation
Clean and friendly Web UI
Their constant stream of blog posts
Their promotion of community content

OpenFaaS Cloud

My main use for Civo’s managed k3s is a platform to deploy and develop OpenFaaS Cloud

OpenFaaS cloud is a self-hosted PaaS for serverless, with built in CI&CD, dashboards and Git Flow. Here are the main Features:

Portable - self-host or use the hosted Community Cluster (SaaS)
Multi-user - use your GitHub/GitLab identity to log into your personal dashboard
Automates CI/CD triggered by git push (also known as GitOps)
Onboard new git repos with a single click by adding the GitHub App or a repository tag in GitLab
Immediate feedback on your personal dashboard and through GitHub Checks or GitLab Statuses
Sub-domain per user or organization with HTTPS
Runtime-logs for your functions
Fast, non-root image builds using buildkit from Docker

I have been contributing to OpenFaaS Cloud and needed a place to run, test and develop my new features. I used Civo’s service to deploy my changes when I implemented runtime logs in OpenFaaS Cloud, you can see them here

Get involved!

The dashboard is written in React, if anyone is interested in contributing you can reach out through the OpenFaaS community. There’s also lots of Golang code involved, if you are more comfortable with that.

I contributed to their App installers

Their kubernetes app installers are open source. When someone on slack noticed that the postgres app wasn’t working I pulled down the github repository that contains the apps and committed a fix. This was swiftly merged and released to production, where several other people created clusters and verified that the app worked.

I didn’t write the initial implementation but it was easy enough to get involved and fix the issue. Thanks to the Civo team for getting it merged and released so quickly!

That issue Andy fixed while at Kubecon

There was an issue that seemed to be affecting my account’s ability to spin up new clusters. I mentioned that I was having problems on the Civo slack. I needed a cluster to deploy a new OpenFaaS Cloud installation that I was testing. I was deep in flow and wanted a cluster quickly, I turned to one of the other providers of managed kubernetes (who shall remain nameless).

Before that new cluster had finished building Andy from Civo had, from his chair at Kubecon, tracked down the issue, added code, tested his fix and deployed it into production. I was even able to spin up a new cluster and connect using kubectl before my backup cluster was ready. That’s fast… real fast. I owe Andy a drink for that.

The magic recipe?

People

From where I sit, as a user, Civo have an amazing team. Everyone is helpful, knowledgeable and committed. They also have a great sense of humour, take the title image for this blog for example.

They got those cards printed for me and mailed them to me after the Kubecon fix. We were having a joke about a job title with as many toxic recruiter buzzwords as possible. They seem to have jammed a lot onto that small business card. Well done…

If a team are working together in this way then they can do great things! Good luck guys.

Community

Civo, as I have mentioned, are smashing this community thing. Keep it up!

It’s great to see their engagement with the community. This is a great way for them to improve the state of Cloud Native, a space that is rapidly expanding and where they are providing some great content, resources and products.

Other cool k3s projects and tooling

It wouldn’t be right not to mention some of the other tools I’m finding really useful in this space.

k3sup

If you want to quickly spin up k3s clusters on your own hardware you can use k3sup. It’s maintained by Alex Ellis and has contributions from loads of other people (including myself).

I can create a new k3s cluster with just 3 commands:

First install a master k3s node like this:

k3sup install --ip 192.168.0.10 --user pi

Then join the two workers:

k3sup join --ip 192.168.0.20 --server-ip 192.168.0.10 --user pi
k3sup join --ip 192.168.0.30 --server-ip 192.168.0.10 --user pi

Arkade

arkade is a kubernetes app installer, it can be used to quickly install apps with a set of default config, maintained by Alex Ellis

Install it by following the instructions in the GitHub README, then install some applications on your new cluster:

arkade install openfaas
arkade install cert-manager
arkade install postgresql

It’s that easy!

k3d

Rancher love containers, and k3s is no exception. There is a tool called k3d that builds k3s clusters in docker containers. It’s great for testing out stuff, then deleting the clusters once you’re done. Quick, easy, no fuss.

# Create a cluster
k3d create

# Connect to it
export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
kubectl get nodes 


# Remove your cluster 
k3d delete

I used k3d a lot. If you have not used it then give it a try!

Thanks for reading!

If you need help with a cloud native platform or migration, get in touch.

How To Get Started Contributing to OpenFaaS

sales@heyal.co.uk (Alistair Hey) — Sat, 04 Jan 2020 00:00:00 GMT

New Year New You?

Don’t bother with the Gym. We all know you won’t keep it up.

Why not give back to the programming community by contributing to an open source project that you use?

When I first started out it took me a while to get my IDE set up, all the dependencies installed and then once I had this set up, picking a github issue and getting through the Pull Request (PR) process also seemed daunting. This still felt like a big deal even though I raise many PRs a day at work and push code into production frequently. For some reason this “open” and new space felt very exposed.

Fortunately, I was helped through the process by Alex, who was always helpful and patient with my early contributions.

This post focuses mainly on code contributions. There are other types of contributions that are just as important and would be welcomed. Things like documentation, user guides, blog posts and helping members of the community on slack.

There’s a good section in the OpenFaaS docs on contributing too.

Maintainers are amazing!

Maintaining an open source project is hard, time consuming and often done for free. There are a load of great resources on the internet about the things maintainers face.

It might be worth you doing a small amount of reading into this, so you know what to expect from a maintainer, and what they expect from your contribution(s).

Here are some examples:

My condolences, you’re now the maintainer of a popular open source project - Daniel Bachhuber
The Five Pressures of Leadership - Alex Ellis
This twitter thread - Benjamin Lupton

Setting up Golang

OpenFaaS is almost exclusively written in Golang. If you’re not a strong Golang developer don’t worry, it’s quite easy to pick up. However, there are other contributions you could make to get started, then ease yourself into Golang at a later date.

I like using the Goland IDE by JetBrains, but you can use VSCode or any other IDE that has Golang support, or even Vim/Emacs.

To install Golang

Install Go from go.dev or with your package manager, which should sort everything out for you.

You probably want to add these lines to your .bashrc or .zshrc so that go is available in new terminals and after you reboot.

export GOPATH=$HOME/go/
export PATH=$PATH:/usr/local/go/bin/

If go version prints a version, Go is set up correctly.

Docker & Kubernetes

You will probably need access to Docker and Kubernetes while writing contributions to OpenFaaS. You can install Docker by following one of the many online guides.

If you are using Kubernetes you will need to install kubectl. Best check online for the most up to date installation method for your Operating System.

I usually use a tool called k3d for creating Kubernetes environments, it creates a single node k3s cluster (Kubernetes compliant) inside a docker container.

Once you have these tools installed you can create, connect to and delete kubernetes clusters like this

# Create a cluster
k3d create

# Connect to it
export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
kubectl get nodes 


# Remove your cluster 
k3d delete

You should now have the tools to deploy an OpenFaaS installation quickly, and reset things when you need a fresh start.

Finding an issue to work on

The best way to find something to work on is to look through the various OpenFaaS Repositories for issues that have the help wanted or good first issue type labels.

This search should show Issues in the main openfaas/faas repository that have the help wanted label.

There are loads of other moving parts of OpenFaaS ecosystem and projects maintained by Alex from OpenFaaS that have various open issues, some good places to look are:

faas-cli The OpenFaaS CLI
OpenFaaS Cloud (OFC)
ofc-bootstrap a tool for provisioning OFC Clusters
openfaas.github.io The OpenFaaS Website
k3sup A tool by Alex for bootstrapping k3s clusters
faas-netes The Kubernetes specific components to OpenFaaS

And many, many more…

You can look in the contributing guide at what types of things you can get involved with

If in doubt, reach out to the OpenFaaS community.

Forking the repository

To get started contributing you need to create your own copy of the repository you want to contribute to.

So, login to Github, and find the repository you are looking to contribute to then click the “fork” button.

Creating a branch and committing your code

Once you have cloned your fork to your computer you need to create a branch for you to work on.

You can do this by typing git checkout -b <branch name>. This creates your local branch to work on.

Once you have written your contribution and written tests (if a code change) and tested the change manually you should commit your code,

git add <files you have changed or added>
# then 
git commit -s

This should open a commit window in your chosen editor (Vim by default) In this file you should write the title, leave a blank line and then a description. This is a good blog about commit messages

Your IDE may also provide a mechanism to commit not using the command line, so feel free to use whatever you’re most comfortable with

Opening your PR

Before you open a PR on github you should read the OpenFaaS Contributing guidelines. This is a requirement for opening a PR against OpenFaaS, and you have to check a box to confirm you have read it.

Once you have finished, you need to open a Pull Request (PR) on github.

Find your fork (copy of the repository) and then go to branches page. From here there’s a “New pull request” button.

You will probably find a “pull request template”, a list of questions you need to answer, like why you have made this change, how it’s been tested and the impact of the change on existing users.

If you are creating a PR that “closes” an issue then you can add Closes #<Issue Number> in the box and github will close that issue when the PR is merged!

Fill in the questions inside the pull request box and submit!

That’s it

After you have raised your PR someone should take a look at it, provide feedback or suggestions and let you know the next steps. You may need to make a change, update some docs or re-word commit messages.

You might also want to let the OpenFaaS community know about it.

Check this out

Alex Ellis, the maintainer of OpenFaaS did a call in early 2019 about contributing, you can find the recording here

My 2019 In Review - Hello Open Source!

sales@heyal.co.uk (Alistair Hey) — Mon, 30 Dec 2019 00:00:00 GMT

OpenFaaS and OpenFaaS Cloud Contributions

In October I started contributing to the OpenFaaS project. It started out as a bugfix here, a documentation change there and it has expanded into 75+ PRs. Not all my contributions were in the OpenFaaS org, but I still managed to get to 15th most contributions in the 12 weeks I have been involved.

I was even given the chance to present some of my work on the OpenFaaS community call which you can watch on Youtube (I’m presenting from about 50 minutes in).

Probably my largest impact has been in adding features and fixing issues in the OpenFaaS Cloud project (OFC).

If I had to pick one feature that I implemented to show you it would be the addition of Runtime logs to the OpenFaaS Cloud dashboard. This means users can view their logs directly from the UI, reducing the time taken to debug and making it easier to focus on their code. It looks a little like this:

Other things I have done:

Upgraded OFC cert-manager version to 0.11.0
Added a splash screen to the OFC dashboard for new users to learn more about OpenFaaS and OpenFaaS Cloud
Added validation for the Registry config file and a command to generate it (In the ofc-bootstrap tool)
Users can now only see their own OFC dashboards & function information (validating a user has claims for the information they are trying to view)
Migrated Kubernetes YAML manifests to 1.16
Added frontend (Gateway) support for the new multi-namespace feature in OpenFaaS

There’s a great community around OpenFaaS; find out how to join here.

k3sup

Alex Ellis has created this amazing tool to provision k3s clusters and manage the installation of various Kubernetes apps and tools. With a few commands you can provision a new cluster, install OpenFaaS and set up TLS ingress. There are a load of other supported apps. I made a few small contributions to k3sup, which was fun!

I had a few spare RPi boards (Or other compute) with IPs of

192.168.0.10
192.168.0.20
192.168.0.30

I created a new k3s cluster with just 3 commands:

First install a master k3s node like this:

k3sup install --ip 192.168.0.10 --user pi

Then join the two workers:

k3sup join --ip 192.168.0.20 --server-ip 192.168.0.10 --user pi
k3sup join --ip 192.168.0.30 --server-ip 192.168.0.10 --user pi

This leaves me with a 3 node cluster

It’s really cool and you can check it out here.

Birmingham (UK) Meetups and Events

Docker & Cloud Native

This year I co-organised monthly Docker & Cloud native meetups. We had some great speakers come along, as well as talks from the organisers. You can check out the past meetups by following the links.

CoffeeOps

In September a group of us from Birmingham went to DevOpsDays London 2019. On the morning of the Friday some people attended a one-off CoffeeOps organised and run by Jennifer Davis and they loved the format so much that we decided to run a weekly meetup on Tuesdays from 0730 to 0900 in Urban Coffee.

Every week a group of us get together to talk about Dev, Ops and Technology in general. We have seen a good turnout (14 at max).

DevOpsDays Birmingham 2021

We loved DevOpsDays London SO MUCH that we decided that Birmingham needed to organise its own.

So five of us have got together and booked a venue for Birmingham’s first DevOpsDays, in early 2021.

Tech Birmingham

As we found ourselves in the business of organising a large multi-day conference, we thought it would be a good idea to get some practice in.

Between us we organise half a dozen meetups in Birmingham. We usually all individually host our own “Christmas” event, but this year we decided that we would try and raise some sponsorship and put on a larger event with the idea of bringing people together who might not normally mingle.

We were able to provide a free alcoholic drink (or 2 non-alcoholic) and mountains of pizza. Over 90 people were scanned in by us at the venue!

We hosted on the 12th December in a Pinball bar in Birmingham:

We got sponsorship from these amazing companies:

Other Things

Civo cloud sent me some personalised business cards as a joke

I have been using Civo Cloud’s Kubernetes service as a Beta tester, and have been finding a few little issues here and there. They thought it would be funny to send me some business cards with a made-up job title.

Alex Ellis sent me some swag

After my first few contributions to OpenFaaS Alex sent me a personalised note and an OpenFaaS Mug!

I wrote a tweet, people seemed to like it!

My 2020 Plans

Work

I left my job at the end of this year to start my own consulting company. This means my January is going to be very interesting!

I’m focusing on Cloud Native platform engineering, building DevOps cultures and empowering teams to deliver more value to their customers!

If you think we might be able to work together to build great things then get in touch.

Tech Community

I’m going to be mentoring for the School of Code, a free 16 week bootcamp run in Birmingham. I think they are always on the lookout for mentors and guest speakers. If you are interested you can contact them via their website.

I’m going to be continuing to run the Docker, CNCF and CoffeeOps meetups in birmingham. Alongside these I’m looking to write more talks and then give these outside Birmingham. I have given talks in Nottingham and Bristol so far.

Open Source

I have no plans to scale back my Open Source contributions. I have really enjoyed being part of the OpenFaaS community and being able to see my efforts have an impact.

If you were considering getting into OpenSource but are not sure where to start I would be happy to have a chat, and guide you through your first contributions! It’s great fun and very rewarding!

Have a great 2020

Thanks for taking the time to read my blog. Hope you had a good year too!

It would be great to connect with more people, so feel free to get in touch!