Are you tired of manually deploying new versions of your website or docker image after completing a new feature or fixing a bug?

Wouldn't it be great if your code could automatically build and publish a semantically versioned docker image and deploy itself to your server? This is where GitHub Actions and Webhooks come in. They allow you to automate your CI/CD pipelines, making your deployment workflow more efficient and less error-prone.

In this guide, I'll walk you through setting up a local CI/CD pipeline using GitHub Actions and Webhooks. The big advantage to doing this locally is that it's free, even beyond the first 2000 minutes of GitHub Actions usage per month, and besides, the build process can be a lot faster if you have proper hardware.

You will learn to:

• Use self-hosted GitHub Actions runners responsibly
• Set up a system that is able to run webhooks
• Create a GitHub Actions workflow to build and publish your Docker image
• Configure Webhooks to trigger a deployment after a new image is published
• Configure your GitHub repository to use GitHub Actions
• Create a GitHub Actions workflow

Prerequisites and assumptions

To follow along with this guide, you'll need to know your way around a Linux shell and have some basic knowledge of Docker and GitHub. You'll also need access to a GitHub repository and a server where you can deploy your software and webhooks, and ideally a separate server where you can run your GitHub Actions runner.

I'm also assuming you use Traefik reverse proxy as your ingress controller to serve web pages and to expose webhooks. Traefik and your web pages run in Docker and are managed by Docker Compose, but you'll learn that we run the webhooks outside of Docker.

The more similar your environment, the easier to follow the guide, but the principles are easy enough to replicate on any other platform you might be using.

Let's get started!

Preparing your environment

First of all, you need to set up some basic components to prepare your system for the CI/CD pipeline. This includes setting up a system that is able to run webhooks and a dedicated VM for your GitHub Actions runner.

Set up a dedicated VM for your self hosted GitHub Actions runner

This step isn't strictly necessary, but I recommend using a separate VM for your GitHub Actions runner. This is not only out of security concerns, but also because the runner will be consuming some resources from time to time, and you don't want to risk it being affected by or affect other processes running in your cluster too much. Having it in a separate VM makes it easier to both manage the packages you need to install and to control the resources it can use.

As a side note, it's recommended to even use a separate "throw-away" VM for each run to guarantee isolation and avoid cross contamination in case of security breaches coming from the build, but for your private hobby projects and repos where you control who can commit code, I think that is a little over the top.

Setting up an extra VM for your GitHub actions runner is really no sweat if you regularly use a virtualization platform like Proxmox, VirtualBox, or VMware for these things. I'm using Proxmox myself, so I quickly created a CT (Container) with Ubuntu 24.04 and 2 vCPU, 4096 RAM and 10 GB disk. This lightweight "VM" doesn't need any special privileges, just use the default setup. Make sure you give the VM a recognizable name, for example <my-github-runners>.

Install the necessary packages

The VM needs to have docker installed, and you will need a non-root user to run the runners.

Log in to your new VM as root, and install the necessary packages. You will need curl and libdigest-sha-perl in order to install the github runner, and you'll want to install Docker from the source.

Start by installing curl and a tool to check the SHA256 hash of the downloaded GitHub runner tarball. Also, jq comes in handy when reading GitHub API responses:

apt update
apt install curl libdigest-sha-perl jq

Now install docker, following the instructions from the official Docker documentation.

Set up a non-root user for the GitHub runner

At this point, create a new user for the GitHub runner and give it the necessary permissions:

adduser github-runner
usermod -aG sudo github-runner
usermod -aG docker github-runner

You may now want to restart the server to make sure all old Docker installations are removed and the new one is properly loaded.

Now your VM is ready to be used as a GitHub runner, and we'll come back to actually configuring a runner on it. First, we need to make sure webhooks are available for your future workflow.

Set up webhooks for your pipeline to trigger

Webhooks are essentially a way for anyone to send a request to a URL to make something happen. It's dangerous to expose webhooks without any form for authentication, and you may want to even use whitelisting of IPs if you experience unexpected traffic.

With GitHub Actions you can have GitHub run the webhook, or in our case, we'll call the webhook ourselves from our deployment script on our GitHub self-hosted runner. We'll make sure the webhook calls are authenticated with a WEBHOOK_SECRET variable we set on the repo level in GitHub. This secret is safely included in the build script sent from GitHub to our runner.

For now though, we'll just set up the webhooks server and lay the grounds for our future pipeline with secrets and all.

Install Webhook on your web server

Webhook is a lightweight tool that allows you to easily create webhooks for your applications. It's written in Go and is very easy to install, especially on a Ubuntu server.

This installs the necessary packages to run webhooks locally:

apt install webhook

Note that we can't practically run the webhook package as a Docker container, because we want the webhook to run scripts on the server itself. This means we need to install it on the server, and the easiest way by far, is with apt.

After installing it, we want to have webhook run as a service, and make sure it starts on boot, always being available for us.

Create a systemd service file /etc/systemd/system/webhook.service:

[Unit]
Description=Webhook Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/webhook -hooks /etc/webhook/hooks.json -verbose -hotreload -urlprefix ""
Restart=on-failure

[Install]
WantedBy=multi-user.target

This assumes a few things:

• You place the list of webhooks you want to create in /etc/webhook/hooks.json (we'll get to that later).
• Hotreload, makes it reload the hooks.json file when it changes.
• Verbose gives you verbose logging, so you can see what's happening - just for the testing!
• Urlprefix is purely cosmetic. By default, webhook listens on 0.0.0.0:9000/webhook, but I prefer that it listens on the root path, because I want it to run on a separate sub-domain, so there is no need to separate the traffic with a path.

Go ahead and create the hooks.json-file and start the service:

touch /etc/webhook/hooks.json
systemctl enable webhook.service
systemctl start webhook.service

Configure Traefik to route traffic to the webhook service

I'm using Traefik as my reverse proxy, so I need to configure it to route traffic to the webhook service. Add the following to your Traefik configuration file:

The easiest way is to set Traefik up in load balancer mode for the webhook server. In the static config for Traefik, you need to add the following:

  routers:
    webhooks:
      entryPoints:
        - "websecure"
        - "web"
      rule: Host(`{{ env "WEBHOOKS_DOMAIN" }}`)
      tls:
        certResolver: zerosslCloudflare
      service: webhooks

  services:
    webhooks:
      loadBalancer:
        servers:
          - url: http://192.168.10.10:9000

Just to be clear, you may already have the routers and services sections in your Traefik configuration file, so you just need to add the webhooks router and service to the existing sections.

This configuration assumes you have a certificate resolver already set up for ZeroSSL using DNS validation towards Cloudflare, and that you have passed the WEBHOOKS_DOMAIN as an environment variable to Traefik, which is a variable containing the domain set up for the webhook service. You can replace the certResolver and rule values with whatever you need for your setup.

The IP for the webhook server is the IP of the machine running the webhook service. You will probably have to replace the IP with the IP of your web server.

When you have added the configuration, restart Traefik to apply the changes. You will know things are working if you get "Ok" when visiting:

https://<WEBHOOKS_DOMAIN>

We'll come back to the actual hooks configurations in hooks.json when we set up your workflow. We're still just preparing your environment.

Adding a CI/CD pipeline to a GitHub repository

Now we are ready to set up our CI/CD pipeline. We will start with creating your workflow.

Adding the workflow code to your codebase

To create a basic workflow, we should branch out from main and add the following code to the root of your repository in a file named .github/workflows/build.yml:

name: Build, tag and deploy

on:
  push:
    branches:
      - main

permissions:
  contents: write   # allows pushing Git tags
  packages: write   # allows pushing to GHCR

jobs:
  build-and-deploy:
    runs-on: self-hosted
    steps:
      - name: Check out repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Determine next version from last tag
        id: semver
        run: |
          # Fetch all tags (sometimes needed explicitly)
          git fetch --tags --prune

          # Get the latest tag, or 0.0.0 if none
          LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "0.0.0")
          echo "Latest tag: $LATEST_TAG"

          MAJOR=$(echo "$LATEST_TAG" | cut -d '.' -f 1)
          MINOR=$(echo "$LATEST_TAG" | cut -d '.' -f 2)
          PATCH=$(echo "$LATEST_TAG" | cut -d '.' -f 3)

          NEXT_MAJOR=$MAJOR
          NEXT_MINOR=$MINOR
          NEXT_PATCH=$((PATCH + 1))

          # Check commit message for [major] or [minor]
          if [[ "${{ github.event.head_commit.message }}" =~ \[major\] ]]; then
            NEXT_MAJOR=$((MAJOR + 1))
            NEXT_MINOR=0
            NEXT_PATCH=0
          elif [[ "${{ github.event.head_commit.message }}" =~ \[minor\] ]]; then
            NEXT_MINOR=$((MINOR + 1))
            NEXT_PATCH=0
          fi

          NEXT_VERSION="${NEXT_MAJOR}.${NEXT_MINOR}.${NEXT_PATCH}"
          echo "Computed next version: $NEXT_VERSION"

          # Export variables
          echo "VERSION=$NEXT_VERSION" >> $GITHUB_ENV
          echo "MAJOR=$NEXT_MAJOR" >> $GITHUB_ENV

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Build Docker image
        run: |
          docker build -t <image>:build -f Dockerfile .

      - name: Tag Docker image
        run: |
          echo "Tagging image with version $VERSION ..."
          docker tag <image>:build ghcr.io/<github_org>/<image>:${{ env.VERSION }}
          docker tag <image>:build ghcr.io/<github_org>/<image>:v${{ env.MAJOR }}

      - name: Login to GitHub Container Registry
        if: always()
        run: |
          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u $GITHUB_ACTOR --password-stdin

      - name: Push Docker image
        run: |
          docker push ghcr.io/<github_org>/<image>:${{ env.VERSION }}
          docker push ghcr.io/<github_org>/<image>:v${{ env.MAJOR }}

      - name: Push new Git tag
        run: |
          # Configure Git
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

          # Create and push the new tag
          git tag "${{ env.VERSION }}" -m "ci: release version ${{ env.VERSION }}"
          git push origin "${{ env.VERSION }}"

      - name: Trigger production update
        run: |
          curl -X POST \
            -H "Content-Type: application/json" \
            -H "Secret: ${{ secrets.WEBHOOK_SECRET }}" \
            -d '{"version": "'${{ env.VERSION }}'"}' \
            https://WEBHOOKS_DOMAIN/update-<repo-name>

On each push to your main branch, this workflow will trigger and build a Docker image, tag it with the new version number, push it to the GitHub Container Registry, and trigger a production update. The version number is incremented based on the commit message. If the commit message contains [major], the major version number is incremented. If it contains [minor], the minor version number is incremented. Otherwise, the patch version number is incremented.

• The building of the Docker image is done using a "self-hosted" runner that we'll soon set up in the repository on our newly created github runner server.
• The image will be posted to the GitHub Container Registry. As long as the repo you're doing this on is public, this is a free, "unlimited" service.
• In the end, a production update is triggered by sending a POST request to a webhook, sending a WEBHOOK_SECRET for authentication. You'll have to set this secret up in GitHub as an action secret variable - we'll do this later on.

Let's start with the webhook.

Set up the webhook in the production environment

We've already set up the webhook server, now we need to add a webhook definition to our hooks.json file, and create a script that can execute the hook.

Add a webhook definition to hooks.json

Add the following code to the hooks.json file:

[
  {
    "id": "update-<repo-name>",
    "execute-command": "/etc/webhooks/update-<repo-name>.sh",
    "command-working-directory": "/etc/webhooks/webhooks",
    "response-message": "Webhook received, verifying signature...",
    "trigger-rule":
    {
          "match":
          {
            "type": "type",
            "value": "<WEBHOOK_SECRET>", 
            "parameter":
            {
              "source": "header",
              "name": "Secret"
            }
          }
    },
    "pass-arguments-to-command": [
      {
        "source": "payload",
        "name": "version"
      }
    ]
  }
]

This will create a new webhook definition that will trigger the update-<repo-name>.sh script when a POST request is received at the https://<WEBHOOK_SERVER>/update-<repo-name> endpoint. The script will be executed in the /etc/webhooks/webhooks directory.

The webhook will only be triggered if the secret matches the WEBHOOK_SECRET we'll soon define as a GitHub actions secret. You need to take note of the secret you define in GitHub and place it in the hooks.json file for authenticating webhook calls.

At this point, we face a security concern, because the secret is stored in plain text. There is a risky way around it, and there is a way to mitigate the threat. The risky way around is to remove the trigger rule and check the secret in the execute-command script instead. This script can read from .env or fetch secrets from some central location. If you do the check wrong, you risk running jobs you never should have ran. So I prefer to mitigate the threat, and limit readability of the hooks.json-file to root and the webhook user.

Create a script that can execute the hook

Create a new file called update-<repo-name>.sh in the /etc/webhooks directory. Add the following code to the file:

#!/bin/bash

DEPLOYED_VERSION="$1"

cd /var/www
docker compose pull <image-name>
docker compose up -d <image-name>

SUBJECT="<image-name> updated to $DEPLOYED_VERSION"
MESSAGE=$(printf "<image-name> has been updated\nBy the GitHub CI/CD Pipeline and your WebHook.")
HOST="localhost"
PORT="25"
RECIPIENT="[email protected]"
FROM="[email protected]"

/usr/bin/swaks \
  --to "${RECIPIENT}" \
  --from "${FROM}" \
  --server "${HOST}" \
  --port "${PORT}" \
  --header "Subject: ${SUBJECT}" \
  --body "${MESSAGE}" \
  -S

This script assumes you run the application with docker compose in the /var/www directory. It will pull the latest image from the registry and restart the container. It will also send an email to the specified recipient when the script is executed.

A nice takeaway here is that if you define your docker compose service for your application with a major version number, like this:

services:
  my-app:
   image: <image-name>:v1

The semantic versioning will be respected and the image will be updated to the latest version of the major version you defined. If you take care with your versioning, this will prevent breaking changes from being deployed to production, as long as you keep compatible versions on the same major version.

For mail sending to work, you must have set up a local relay server. You can use swaks to send the mail (but you need to install swaks first).

Before we set up the self-hosted GitHub actions runner, let's make some preparations on the GitHub repository.

Configuring the GitHub repository

Go to the repository you want to add the CI/CD pipeline to.

Set up branch protection rules

This will prevent anyone from pushing directly to the main branch. Instead, they will have to create a pull request and get it approved before merging. This is a good practice to ensure that the code is reviewed before it is merged into the main branch, and to prevent anyone from pushing broken or insecure code to the main branch for running on your self-hosted runner.

1. Go to Settings > Branches.
2. Click on Add rule.
3. Set the branch name pattern to main. (You can add more patterns if you want to protect other branches as well.)
4. Check the following boxes: Require pull request reviews before merging, Require status checks to pass before merging, Require branches to be up to date before merging.
5. Click on Create.

Configure workflow triggers

This will ensure that only approved pull requests can trigger workflows. This is a good practice to prevent anyone from triggering workflows with unapproved code.

1. Go to Settings > Actions > General.
2. Check the box for Require approval for all external contributors.
3. Check the box for Read and Write permissions for "Workflow permissions" to allow your workflows to persist changes (superfluous if you have the permission block in your build.yaml, but good for clarity.)
4. Click on Save.

Configure the WEBHOOK_SECRET

1. Go to Settings
2. In the left sidebar, choose Secrets and variables → Actions.
3. Click New repository secret and name it WEBHOOK_SECRET. Put the secret value (that must match the server’s hooks.json config).

Set up and install a self-hosted runner

This process will install a self hosted runner for a single repository. Repeat the entire process for each repository you want to add a runner to.

1. Click on Settings → Actions → Runners.
2. Click on New self-hosted runner.
3. Choose your operating system (Linux in this case).
4. Follow the instructions provided by GitHub, but adapt it so you can add more runners later for different Projects/repositories.

The instructions assume you log in to your actions runner, that we called my-github-runners. Log in as root, and change user to the github-runner user.

su - github-runner
mkdir <repo-name> && cd <repo-name>
curl -o actions-runner-linux-x64-2.322.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.322.0/actions-runner-linux-x64-2.322.0.tar.gz

# Validate the hash (!) Note that the hash is given by GitHub, don't reuse the example hash
echo "b13b784808359f31bc79b08a191f5f83757852957dd8fe3dbfcc38202ccf5768  actions-runner-linux-x64-2.322.0.tar.gz" | shasum -a 256 -c

tar xzf ./actions-runner-linux-x64-2.322.0.tar.gz

# Create the runner and start the configuration process
#  - Add the runner to the default group
#  - Name the worker <repo-name>
#  - Drop adding labels
#  - Use the default output folder.
./config.sh --url https://github.com/<your-org-or-user>/<repo-name> --token A-TOKEN-GENERATED-BY-GITHUB

# Now, test run the runner
./run.sh

Your job should now be available under /settings/actions/runners for your repo in GitHub, as "idle" (refresh the page if you're impatient)

Now, you should absolutely make the runner start on boot and let it run in the background:

cd /path/to/your/runner-directory
sudo ./svc.sh install
sudo ./svc.sh start

Now everything is ready!

Test the workflow

Now, test the workflow.

Commit and push the new workflow file to your repository to the branch you created earlier. This will not yet trigger the workflow, as the push doesn't happen to the main branch.

Now, create a pull request from your branch to the main branch, and merge it. This will trigger the workflow.

Now you should see the workflow running in the Actions tab of your repository.

In the end, your application should be updated on your server and you receive an email notifying you of the success of the deployment.

Update - add a VERSION file to your repository on each build

Your application may need or want a VERSION file to keep track of itself. With the above method, you have a Git Tag, but the tag isn't available in the source code. The problem with keeping an updated VERSION file, is that you can'y really create it until the tag is created. And the tag is created after you commited and pushed your pull request (which is what triggers the build in the first place). And on top of this, you have branch protection rules disallowing direct commits to main without doing a PR first. So you have a catch 22 here.

To amend this, the simplest way is to use Deploy Keys, and excempt Deploy Keys from the branch protection rules.

These are the steps:

Generate keys

Go to a Linux terminal and type:

ssh-keygen -t ed25519 -C "MyRepo GitHub Deploy Key"

Now, go to your repo Settings → Deploy keys → Add deploy key

• Add a descriptive name for your reference
• Paste the contents of the .pub-file form the ssh-keygen step above.
• Allow the deploy key write permission to your repo.

Then go to your branch protection rule and "Add bypass" and select "Deploy keys". Make sure Deploy Keys appear in the excemption list with "Always allow".

Now, we need to pass our private key to our runner, so it can use it to authenticate the deployment with the key.

Go to Settings → Secrets and Variables → Actions and "New repository secret". Call the secret DEPLOY_KEY and paste the contents of the other file generated by the ssh-keygen command (this is the private key).

Remember, the private key gives complete write access to your repo! As a safeguard, use one deploy key per repo. If a private key goes missing, just remove the Deploy Key from your repo and repeat the process setting up a new key and replacing the secret with the new private key.

NOTE! After you store the private key in this field, you won't be able to retrieve it through the GUI, so store the private and public keys somewhere safe.

Now, we're ready to update our workflow:

jobs:
  build-and-deploy:
    runs-on: self-hosted
    steps:

      # New code here
      - name: Start SSH agent with deploy key
        uses: webfactory/[email protected]
        with:
          ssh-private-key: ${{ secrets.DEPLOY_KEY }}

      - name: Add GitHub to known_hosts
        run: |
          mkdir -p ~/.ssh
          ssh-keyscan github.com >> ~/.ssh/known_hosts

      - name: Check out repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Update remote URL to use SSH
        run: git remote set-url origin [email protected]:${{ github.repository }}.git

      - name: Determine next version from last tag
      # This step remains unchanged
      # But immediately after this step, and before tagging, commit the VERSION file

      - name: Update VERSION file using deploy key
        run: |
          echo "${{ env.VERSION }}" > VERSION
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@workflow-testing"
          git add VERSION
          git commit -m "ci: update VERSION file to ${{ env.VERSION }} [skip ci]"
          git push origin main

      # And the rest of your code...

This will set up an ssh-agent to broker the GitHub connection with the Deploy Key for you and use it throughout the workflow. Note that we commit the VERSION file with a [skip ci] tag inside the commit message. This stops GitHub for running workflows based on this commit (so no testing or building).

And there you go. A Version file for your application in the repo.