Airflow + Kubernetes 101

24-12-2020 / Edit on Github

👉 Note: Docker 101
👉 Note: Wordpress Docker
👉 Note: Docker & GPUs
👉 Note: Tensorflow extra

Installation #

sudo snap install microk8s --classic
# make an alias
alias k='microk8s.kubectl'
# add to sudo group
sudo usermod -a -G microk8s thi
sudo chown -f -R thi ~/.kube
# log out and log in again

We use k which stands for kubectl or microk8s.kubectl in this notebook!

k8s #

Pods #

Pod: (full): a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers.

# list of pods
microk8s.kubectl get pods
# enter a pod
microk8s.kubectl exec -it <pod_id> bash

Namespaces #

  • Detail is here.
  • Multiple virtual clusters (namespaces) backed by the same physical cluster.
  • Motivation for using namespaces: here => one word: isolation!
  • Avoid creating namespace with prefix kube-.
# list of current namespaces
k get namespace
# create
k create namespace <name_space>
# follows DBS label:
# delete
k delete namespaces <name_space>
# This deletes everything under the namespace!

Airflow #

Quickstart #

Check here.

KubernetesPodOperator #

  • task_id="abc_xyz" (required): the name of task given in the airflow ui.
  • image="localhost:32000/airflow-abc:debug" (required): docker image to use.
  • namespace="default" (required, ref):
  • in_cluster=True:
  • trigger_rule="all_success": ref
  • image_pull_policy="Always": if you changes something on the docker image.

If using a dictionary and feed to env_vars (error "TypeError: string indices must be integers"),

ex_var = {'a': 1, 'b': 2}
with DAG(
# ...
) as dag:
abc = KubernetesPodOperator(
# ...
env_vars = {
"EX_VAR": str(ex_var)
# ...
# used in another file
import os
import ast

ex_var = ast.literal_eval(os.environ["EX_VAR"])


Branching #

Airflow Errors #

🔅 ERROR - Exception when attempting to create Namespaced Pod.

  • Note that if you don't use in_cluster=True, then you won't run into this problem.[ref]
  • There may be something wrong with variables or their type of values. Check again carefully!

References #

Notes with this notation aren't good enough. They are being updated.