Guide to Self-Hosting GitLab & GitLab Runner on K3s with Custom TLS (Because Security is Serious Business… But Not That Serious)

Hey there, fellow code wranglers and server tamers! Buckle up, because we’re about to embark on a glorious journey – installing our very own self-hosted GitLab with a trusty GitLab Runner on top of the tiny yet mighty K3s, all while spicing things up with a custom TLS certificate configuration.

Po (holding a giant wrench): “There is no secret ingredient to self-hosting GitLab… except maybe a little kung fu hustle!”

Now, before we dive in, a word of warning: this might get a little technical. But fear not, for I, your friendly neighborhood DevOps engineer (with a serious case of meme addiction), will guide you through the process with enough humor and helpful visuals to make even the most complex configurations feel like a walk in the park… or a victory lap around a server room, whichever floats your boat.

Why Self-Host, You Ask?

Because, my friends, freedom! No more relying on flaky cloud services or fighting for resources. With self-hosting, you’re the captain of your own DevOps destiny. Plus, it’s a fantastic learning experience (and a great way to impress your colleagues at the next virtual watercooler chat… or, you know, send them slightly concerned glances about your newfound DevOps skills 😎).

The Ingredients (Because We Don’t Code on Empty Stomachs):

A bunch of VMs (preferably one that doesn’t double as your toaster)
K3s – the lightweight Kubernetes distribution that’s like the tiny, adorable martial arts master of the container orchestration world (think: Kubernetes meets Yoda)
Self-hosted GitLab – the crown jewel, where all your code magic happens ✨
GitLab Runner – your tireless build automation buddy, ready to execute your CI/CD pipelines with lightning speed ⚡
A custom TLS certificate – the security shield that keeps the bad guys out and also avoids painful clicks in your browser every time you access Gitlab (think of it as a digital bouncer with a very strict dress code) ️

Step 1: Installing K3s – It’s K3s-y!

This is where things get fun. We’ll follow the K3s installation guide here.

Step 2: Custom TLS – Because Security is Still Serious Business (But Way More Metal Now)

This is where things get a little more hands-on. Instead of Let’s Encrypt, we’ll be using your battle-axe sharp custom TLS certificate and its trusty squire, the private key (PEM file). Here’s the breakdown:

Prepare Your Weapons: Make sure you have both the certificate file (CRT) and the private key file (PEM) readily available. These are the digital guardians that will keep unauthorized users at bay.
Mount the Cavalry: We’ll need to mount these files as secrets within the GitLab deployment on K3s. This ensures they are securely accessible to the application but hidden from prying eyes.

# Create generic secret
sudo kubectl create secret generic gitlab-tls-secret \                                                         
  --namespace gl \
  --from-file=/root/devops/certs/devops.crt 

# Create TLS secret
sudo kubectl create secret tls example-tls-secret \                                                     
  --namespace gl \           
  --cert=/root/certs/devops.pem \    
  --key=/root/certs/devops.key

Step 3: Deploying GitLab – Let the Code Flow!

Here comes the big kahuna! We’ll leverage the power of Helm charts to deploy GitLab on our K3s cluster provided by GitLab. Think of Helm charts as pre-made recipes for deploying complex applications on Kubernetes – like having a personal DevOps sous chef at your beck and call. ‍

Here are the commands to spin up GitLab

helm upgrade --install gitlab gitlab/gitlab \                                                            
  --namespace gl \
  --set global.hosts.domain=gitlab.example.com \
  --set global.hosts.gitlab.path=gitlab \
  --set global.hosts.disableCertmanager=true \
  --set certmanager-issuer.email=dummy@example.com \
  --set global.hosts.secretName=example-tls-secret \
  --set gitlab-runner.runners.privileged=true \
  --set global.edition=ce \
  --set gitlab.webservice.ingress.tls.secretName=example-tls-secret \
  --set global.ingress.tls.secretName=example-tls-secret \
  --set global.smtp_enabled=true \
  --set global.smtp.address='smtp-outbound.example.com' \
  --set global.smtp.port=25 \
  --set global.smtp.domain='smtp-outbound.example.com' \
  --set global.smtp.tls=false \
  --set global.smtp.openssl_verify_mode='none' \
  --set global.email.display_name='DevOps Gitlab' \
  --set global.email.from='no_reply@example.com' \
  --set global.email.reply_to='no_reply@example.com' \
  --set global.tls.secretName=example-tls-secret

Step 4: Unleash the GitLab Runner – Build Like a Boss!

With GitLab up and running, it’s time to unleash the GitLab Runner. We’ll use the Helm chart magic again to deploy the Runner and configure it to talk to our shiny new GitLab instance.

# Create generic secret
sudo kubectl create secret generic gitlab-cert \                                                         
  --namespace gitlab-runner \
  --from-file=/root/certs/devops.pem

For GitLab Runner to function, your configuration file must specify the following:

gitlabUrl: The GitLab server full URL to register the runner against. For example, https://gitlab.example.com.
rbac: { create: true }: Create RBAC rules for the GitLab Runner to create pods to run jobs in. If you have an existing serviceAccount you prefer to use, you should also set rbac: { serviceAccountName: "SERVICE_ACCOUNT_NAME" }. For more information about the minimal permissions required for the serviceAccount, see Configure runner API permissions.
runnerToken:
- The authentication token you obtain when you create a runner in the GitLab UI.
- Set the token directly or store it in a secret.
Set the certsSecretName in order to pass custom certficates for GitLab Runner to use
Provide resource name for a Kubernetes Secret Object in the same namespace,
this is used to populate the /etc/gitlab-runner/certs directory
ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates
#
certsSecretName: gitlab-cert

Step 5: Let’s Test Drive!

Here’s the .gitlab-ci.yml standard template file

image: ubuntu:22.10
before_script:
  - echo "Before script section"
  - echo "For example you might run an update here or install a build dependency"
  - echo "Or perhaps you might print out some debugging details"

after_script:
  - echo "After script section"
  - echo "For example you might do some cleanup here"

build1:
  stage: build
  tags:
    - k3s
  script:
    - echo "Do your build here"

test1:
  stage: test
  tags:
    - k3s
  script:
    - echo "Do a test here"
    - echo "For example run a test suite"

test2:
  stage: test
  script:
    - echo "Do another parallel test here"
    - echo "For example run a lint test"

deploy1:
  stage: deploy
  script:
    - echo "Do your deploy here"
  environment: production

and the Output/Outcome is as under… As you can it was not a cakewalk… I’ve failed 91 times and succeeded on the 92nd attempt. “Keep calm and carry a wand.” — A.W. Jantha

Step 6: Victory Lap – You Did It!

Congratulations, brave adventurer! You’ve successfully self-hosted GitLab and GitLab Runner on K3s, complete with a custom TLS configuration. Now, go forth and conquer your CI/CD pipelines with newfound confidence (and maybe a celebratory dance break).

Bonus Round: Troubleshooting – Because Things Don’t Always Go According to Plan

Let’s be honest, things might not always work perfectly on the first try with a personal track record of 91 attempts!! But fear not, for I’ve included a troubleshooting section with helpful tips and resources to get you back on track. “Debugging is like being the detective in a crime movie where you are also the murder suspect.” – Moshe Feder (Because troubleshooting your self-hosted setup might feel like a never-ending mystery)

# Checking the logs of Pod is the best way to start
kubectl logs -n gitlab-runner gitlab-runner-8b54596c8-hmdgg
# Describe Pod
kubectl describe -n gitlab-runner pod gitlab-runner-8b54596c8-hmdgg
# kubectl edit cm -n gitlab-runner gitlab-runner
Edit the configuration accordingly...
# Login to the container and check 
kubectl exec -it -n gitlab-runner gitlab-runner-8b54596c8-hmdgg -- bash
gitlab-runner-8b54596c8-hmdgg:/$ cat /home/gitlab-runner/.gitlab-runner/config.toml 
concurrent = 100
check_interval = 10
log_level = "debug"
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-runner-8b54596c8-hmdgg"
  url = "https://gitlab.example.com"
  id = 13
  token = "glrt-SGpPs5SpuGbYwfU9dzpB"
  token_obtained_at = 2024-06-22T10:46:19Z
  token_expires_at = 0001-01-01T00:00:00Z
  tls-ca-file = "/home/gitlab-runner/.gitlab-runner/certs/devops.pem"
  executor = "kubernetes"
  builds_dir = "/cache"
  cache_dir = "/cache"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "ubuntu:22.04"
    namespace = "gitlab-runner"
    namespace_overwrite_allowed = ""
    namespace_per_job = false
    privileged = true
    skip-tls-verify = true
    allow_privilege_escalation = true
    node_selector_overwrite_allowed = ""
    node_tolerations_overwrite_allowed = ""
    pod_labels_overwrite_allowed = ""
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.init_permissions_container_security_context]
      [runners.kubernetes.init_permissions_container_security_context.capabilities]
    [runners.kubernetes.build_container_security_context]
      [runners.kubernetes.build_container_security_context.capabilities]
    [runners.kubernetes.helper_container_security_context]
      [runners.kubernetes.helper_container_security_context.capabilities]
    [runners.kubernetes.service_container_security_context]
      [runners.kubernetes.service_container_security_context.capabilities]
    [runners.kubernetes.volumes]

      [[runners.kubernetes.volumes.pvc]]
        name = "gitlab-runner"
        mount_path = "/cache"
    [runners.kubernetes.dns_config]

# Check the status of the Pod after the necessary changes
kgp -n gitlab-runner
NAME                            READY   STATUS    RESTARTS       AGE
gitlab-runner-8b54596c8-hmdgg   1/1     Running   10 (17h ago)   17h

Remember: This guide is meant to be informative and lighthearted. Always refer to the official documentation for detailed instructions. But hey, with a little humor and these handy tips, you’ll be self-hosting GitLab and Gitlab-Runner with TLS like a pro in no time!

If you enjoyed this post, I’d be very grateful if you’d help it spread by emailing it to a friend, or sharing it on Social Media Platforms. Thank you!

What am I missing here? Let me know in the comments and I’ll add it in! OR tweet it to me @babvin

Guide to Self-Hosting GitLab & GitLab Runner on K3s with Custom TLS (Because Security is Serious Business… But Not That Serious)

Why Self-Host, You Ask?

The Ingredients (Because We Don’t Code on Empty Stomachs):

Step 1: Installing K3s – It’s K3s-y!

Step 2: Custom TLS – Because Security is Still Serious Business (But Way More Metal Now)

Step 3: Deploying GitLab – Let the Code Flow!

Step 4: Unleash the GitLab Runner – Build Like a Boss!

Step 5: Let’s Test Drive!

Step 6: Victory Lap – You Did It!

Bonus Round: Troubleshooting – Because Things Don’t Always Go According to Plan

Published by Vinay

Leave a comment Cancel reply

Why Self-Host, You Ask?

The Ingredients (Because We Don’t Code on Empty Stomachs):

Step 1: Installing K3s – It’s K3s-y!

Step 2: Custom TLS – Because Security is Still Serious Business (But Way More Metal Now)

Step 3: Deploying GitLab – Let the Code Flow!

Step 4: Unleash the GitLab Runner – Build Like a Boss!

Step 5: Let’s Test Drive!

Step 6: Victory Lap – You Did It!

Bonus Round: Troubleshooting – Because Things Don’t Always Go According to Plan

Share this:

Related

Published by Vinay

Leave a comment Cancel reply