Automate K8S manifest files using Python and Jinja2

Don’t type, Use Template

Have you ever thought about automation of K8S manifest files (YAML) to simplify building DEV, TEST, PROD environments?

Even though modern day tools does the heavy lifting by automagically building desired infrastructure in minutes compared to days, weeks, or even months. However, the human factor of committing errors while writing these manifests/YAML files can still exists!

During my Storage Admin days, we used to write/copy-paste variables based on the requirements from the requestor or platform teams into an excel file. Using excel formulae and “drag” magic, commands were made available to execute on the enterprise class storage arrays. I was thinking how to apply the similar wisdom into SRE world…

Say Hello to Jinja2 templates

Jinja2 is a modern day templating language for Python developers. It was made after Django’s template. It is used to create HTML, XML or other markup formats that are returned to the user via an HTTP request. However Jinja2 templates can be used for various use cases. In this blog, jinja2 template with python used to build manifests/YAML files.

Installing Jinja2 is pretty easy using pip or easy_install

pip install jinja2
 
easy_install jinja2

Jinja2 Templates

Jinja2 template contain variables which are replaced by the values. These values are passed to the template file using template rendering method. These variables can be strings, dictionaries or even values inside loops like if, for etc..

Delimeters

{%....%} are for statements

{{....}} are expressions used to print to template output

{#....#} are for comments which are not included in the template output

#....## are used as line statements

An example of Jinja2 template manifest file

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ pv.name }}
  labels:
    type: {{ pv.type }}
spec:
  storageClassName: {{ pv.class }}
  capacity:
    storage: {{ pv.capacity }}
  accessModes:
    - {{ pv.mode }}
  hostPath:
    path: {{ pv.path }}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: bitwarden
  name: bitwarden
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 800Mi
---

In the above manifest/YAML file (pv_pvc.yml), first half is about Persistent Volume where the values are template values which begins with ‘pv.<variable name>’

Properties Values

name,type,class,capacity,mode,path,environment
bitwarden.sample,local,manual,800Mi,ReadWriteOnce,'/mnt/sample/bitwarden', sample
bitwarden.pre,local,manual,1000Mi,ReadWriteOnce,'/mnt/pre/bitwarden', pre
bitwarden.prd,local,nfs,2000Mi,ReadWriteOnce,'/mnt/prd/bitwarden', prd
bitwarden.dev,local,manual,800Mi,ReadWriteOnce,'/mnt/dev/bitwarden', dev

In the above properties.csv file, first line is a header, followed by sample value. Next 3 lines are meant for DEV, TEST and PROD.

Python Jinja2 Code with documentation

from os import pread
from jinja2 import Environment, FileSystemLoader
import csv

csvfile = open('properties.csv')

lists = []
for index,line in enumerate(csvfile):
    if not index == 0 and not index == 1:        
        data_line = line.rstrip().split(',')
        lists.append(data_line)

print(lists)
file_loader = FileSystemLoader("templates")
env = Environment(loader=file_loader)
template = env.get_template("pv_pvc.yml")
dev = ["bitwarden.dev","local","manual","800Mi","ReadWriteOnce","'/mnt/dev/bitwarden'", "dev"]
uat = ["bitwarden.pre","local","manual","1000Mi","ReadWriteOnce","'/mnt/pre/bitwarden'", "pre"]
prod = ["bitwarden.prd","local","nfs","2000Mi","ReadWriteOnce","'/mnt/prd/bitwarden'", "prd"]
pv = {}
# lists = [dev, uat, prod]
for index,element in enumerate(lists):
    # print(element[0])
    pv["name"] = element[0]    
    pv["type"] = element[1]
    pv["class"] = element[2]
    pv["capacity"] = element[3]
    pv["mode"] = element[4]
    pv["path"] = element[5]
    # template.render will replace the jinja template variables with list values
    output = template.render(pv=pv)
    # print(output)
    # Create manifests/YAML files from the about output variable
    try:
        with open(f"{element[-1]}.yml", "w+") as yml_file:
            yml_file.write(output)
        yml_file.close()
    except Exception as Error:
        print(f"Unable to open the file, {yml_file} due to the error {Error}")

Results – python process_yamls.py

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: bitwarden.dev
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 800Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: '/mnt/dev/bitwarden'
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: bitwarden
  name: bitwarden
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 800Mi
---

dev.yml (DEV)

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: bitwarden.pre
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 1000Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: '/mnt/pre/bitwarden'
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: bitwarden
  name: bitwarden
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 800Mi
---

pre.yml (TEST)

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: bitwarden.prd
  labels:
    type: local
spec:
  storageClassName: nfs
  capacity:
    storage: 2000Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: '/mnt/prd/bitwarden'
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: bitwarden
  name: bitwarden
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 800Mi
---

prd.yml (PROD)

Three files generated based on the values provided in the properties.csv file. In the example code above, I’ve hard-coded the values for simplicity and to easily understand the code. This can be improved and can be applied complex manifests/yamls like configmaps, deployments, DS, RS etc…

I’d like to thank Gangireddy who came up with this idea to automate manifests to automate SRE/DEVOPS engineers BAU stuff!

If you enjoyed this post, I’d be very grateful if you’d help it spread by emailing it to a friend, or sharing it on your social platforms. Thank you!

What am I missing here? Let me know in the comments and I’ll add it in!

One thought on “Automate K8S manifest files using Python and Jinja2

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s