Last modified on 22 Jul 2020.

Installation

sudo snap install microk8s --classic
# make an alias
alias k='microk8s.kubectl'
# add to sudo group
sudo usermod -a -G microk8s thi
sudo chown -f -R thi ~/.kube
# log out and log in again

We use k which stands for kubectl or microk8s.kubectl in this notebook!

k8s

Pods

Pod: (full): a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers.

# list of pods
microk8s.kubectl get pods
# enter a pod
microk8s.kubectl exec -it <pod_id> bash

Namespaces

  • Detail is here.
  • Multiple virtual clusters (namespaces) backed by the same physical cluster.
  • Motivation for using namespaces: here => one word: isolation!
  • Avoid creating namespace with prefix kube-.
# list of current namespaces
k get namespace
# create
k create namespace <name_space>
# follows DBS label: https://bit.ly/2Cxge0K
# delete
k delete namespaces <name_space>
# This deletes everything under the namespace!

Airflow

Quickstart

Check here.

KubernetesPodOperator

  • task_id="abc_xyz" (required): the name of task given in the airflow ui.
  • image="localhost:32000/airflow-abc:debug" (required): docker image to use.
  • namespace="default" (required, ref):
  • in_cluster=True:
  • trigger_rule="all_success": ref
  • image_pull_policy="Always": if you changes something on the docker image.

If using a dictionary and feed to env_vars (error “TypeError: string indices must be integers”),

ex_var = {'a': 1, 'b': 2}
with DAG(
    # ...
) as dag:
    abc = KubernetesPodOperator(
        # ...
        env_vars = {
            "EX_VAR": str(ex_var)
        }
        # ...
    )
# used in another file
import os
import ast

ex_var = ast.literal_eval(os.environ["EX_VAR"])

References:

Branching

Airflow Errors

🔅 ERROR - Exception when attempting to create Namespaced Pod.

  • Note that if you don’t use in_cluster=True, then you won’t run into this problem.[ref]

  • There may be something wrong with variables or their type of values. Check again carefully!

References

Notes with this notation aren't good enough. They are being updated. If you can see this, you are so smart. ;)