Live demo details

• Create a brand new Kubernetes cluster in AWS’ Elastic Kubernetes Service

• Creates Virtual Private Cloud

• Creates Auto Scaling Group for worker nodes

• Demonstrate some of the best practices for Kubernetes configuration management

• From zero to ready-to-go cluster all in Ansible

About Kubernetes

• Kubernetes is a platform for managing, scaling and deploying applications running on containers across distributed networks

• Kubernetes continually runs a reconciliation loop to ensure the cluster is in the desired state and corrects it if possible

Kubernetes: a common language

Kubernetes allows the specification of common characteristics of applications and services:

• an application’s deployable and its associated configuration (Pod)

• how many replicas should run, and where; how should updates be handled (Deployment/DaemonSet)

• what endpoints should be exposed to the applications (Service)

• what traffic should go to the Service (Ingress)

Kubernetes: a common language

• ConfigMap—one or more configuration items in key-value form. Useful for setting environment variables or specifying the entire contents of one or more files for a Pod

• Secret—similar to ConfigMap but better protected from casual view

Kubernetes resource definitions

• Resource definitions are in YAML form

  apiVersion: v1 kind: ConfigMap metadata:   name: my-config-map   namespace: my-namespace data:   hello: world

• Typically managed by the kubectl command line tool kubectl apply -f
resource.yml

Anti-pattern: using kubectl
in playbooks

• kubectl is awesome

• But all the usual caveats to running commands apply^†

• You have to do a template/kubectl/delete dance

Are there reasons to use kubectl?

• kubectl does validation of resource definitions against their specification

• kubectl can append hashes to ConfigMaps and Secrets to make them immutable

• ad-hoc tasks:

  kubectl get configmap -n some-namespace some-config-map ansible -m k8s_facts -a 'namespace=some-namespace kind=ConfigMap \   api_version=v1 name=some-config-map' localhost

Reuse

• Our goal is to use as much common code for Kubernetes management as possible

• A single Ansible role that takes a set of resource manifests and ensures that Kubernetes meets those expectations

• Ideally, one manifest template that works for most applications would be great, but harder

Ansible’s Kubernetes strengths

• Templating

• Roles

• Hierarchical inventory

• Secrets management

• Modules, lookup plugins, filter plugins

  Reuse

• Our goal is to use as much common code for Kubernetes management as possible

• A single Ansible role that takes a set of resource manifests and ensures that Kubernetes meets those expectations

• Ideally, one manifest template that works for most applications would be great, but harder

Ansible’s Kubernetes strengths

• Templating

• Roles

• Hierarchical inventory

• Secrets management

• Modules, lookup plugins, filter plugins

Templating

• Templates are super-powerful

• Reuse resource definition files for all environments

• Use a common language where possible – e.g. {{ kube_resource_name }} {{ kube_ingress_fqdn }} across all applications

• Avoid control structures where possible

  replicas = {% 5 if env == 'prod' else 1 %}

  vs

  replicas = {{ kube_deployment_replicas }}

Templating
dicts

Sometimes whole sections of manifests differ between environment

metadata:   annotations:     {{ kube_ingress_annotations | to_nice_yaml(indent=2) | indent(4) }}

Roles

• Roles are excellent for sharing a common set of tasks across multiple playbooks

• One role should be suitable for almost all Kubernetes operations

• We have moved from per-application roles to one generic role for all but a couple of applications

Hierarchical inventory

• Some properties are the same across many applications within an environment

• Some properties are the same across all environments for an application

• Some properties can be composed from other properties

• Some properties may need specific overrides for certain application/environment combinations

• All of these needs are met by inventory groups

Avoiding hosts: localhost

• Typically people will use hosts:
localhost to talk to Kubernetes

• This reduces the power of inventory and reuse

Using the runner pattern

• Runner pattern uses hosts declarations like hosts: "{{ env }}-{{
app }}-runner" with e.g. -e
env=test -e app=web

• inventory hierarchies allow runners to gather their inventory from groups such as test, web and test-web

• Set ansible_connection: local and ansible_python_interpreter: "{{
ansible_playbook_python }}" in the runner group_vars file

Flat vs hierarchical inventory

Generating inventory

• Group combinations explode as applications and environments increase

• It’s easy to get this wrong with standard hosts files

• generator inventory plugin generates such group combinations from a list of layers

Standard hosts file

[test:children] test-web test-api [api:children] test-api [web:children] test-web [test-web] test-web-runner [test-api] test-api-runner [runner] test-web-runner test-api-runner

Generating inventory

• Group combinations explode as applications and environments increase

• It’s easy to get this wrong with standard hosts files

• generator inventory plugin generates such group combinations from a list of layers

Standard hosts file

[test:children] test-web test-api [api:children] test-api [web:children] test-web [test-web] test-web-runner [test-api]

Generator plugin hosts file

# inventory.config file in YAML format plugin: generator strict: False hosts:     name: "{{ environment }}-{{ application }}-runner"     parents:       - name: "{{ environment }}-{{ application }}"         parents:           - name: "{{ application }}"             vars:               application: "{{ application }}"           - name: "{{ environment }}"             vars:               environment: "{{ environment }}"       - name: runner layers:     environment:         - test     application:         - web         - api

Secrets

• We use ansible-vault for all of our secrets

• Kubernetes expects secrets to be base64 encoded

• Use no_log with the k8s module when uploading secrets

• Avoid vaulting whole variables files

• Use ansible-vault encrypt_string to encrypt each secret inline

• Don’t forget to use echo -n $secret
| ansible-vault encrypt_string to avoid encrypting the newline!

Secrets in environment variables

• Use a Secret resource to store secret environment variables

• Use envFrom if you then want to include all the secrets from that resource

Secrets in environment variables

key1: !vault |   $ANSIBLE_VAULT;1.1;AES256   61666162663666643939353165393833383331313664616234343739653937336337626263663538   3335336263303963623332666639666364356166393462370a396465393637363938656562393936   61663834376235613564303237313131396335303336636466326430353530613836356564343832   6638393533663931640a663438313461616436393365346566313037613034323738646234363534   3734 my_secret_env:   KEY1: "{{ key1 | b64encode }}"

A Secret manifest

apiVersion: v1 kind: Secret metadata:   name: my-secret-env   namespace: my-namespace data:   {{ my_secret_env | to_nice_yaml(indent=2) | indent(2) }}

Using the Secret

--- kind: Deployment spec:   template:     spec:       containers:       - envFrom:           - secretRef:               name: my-secret-env

Secrets in environment variables key1: !vault | $ANSIBLE_VAULT;1.1;AES256 61666162663666643939353165393833383331313664616234343739653937336337626263663538 3335336263303963623332666639666364356166393462370a396465393637363938656562393936 61663834376235613564303237313131396335303336636466326430353530613836356564343832 6638393533663931640a663438313461616436393365346566313037613034323738646234363534 3734 my_secret_env: KEY1: “{{ key1 | b64encode }}”

A Secret manifest

apiVersion: v1 kind: Secret metadata:   name: my-secret-env   namespace: my-namespace data:   {{ my_secret_env | to_nice_yaml(indent=2) | indent(2) }}

Using the Secret

--- kind: Deployment spec:   template:     spec:       containers:       - envFrom:           - secretRef:               name: my-secret-env

Modules

• k8s—main module for managing Kubernetes resources

• k8s_facts—useful for run-time querying of resources

• aws_eks_cluster—manages AWS EKS clusters

• azure_rm_aks—manages Azure Kubernetes Service clusters

• gcp_container_cluster and gcp_container_nodepool—manage GKE clusters and node pools

k8s module

• uses the same manifest definitions as kubectl

• can take inline resource definitions, or src from file

• inline definitions work well with template lookup definition:
"{{ lookup('template', 'path/to/resource.j2')
| from_yaml }}"

• invoke once with a manifest containing a list of resources, or invoke in a loop over a list of resources

• copes with Custom Resource Definitions (2.7)

Plugins

• yaml stdout callback plugin is great for having output match input

• k8s lookup plugin returns information about Kubernetes resources

• from_yaml and from_yaml_all (2.7) read from templates into module data

• b64encode encodes secrets in base64

• k8s_config_hash and k8s_config_resource_name for immutable ConfigMaps (likely 2.8)

Demo

Scenario:

• We’re practising Continuous Delivery with Rolling Deployments and feature flags

• Upgrade application

• Enable feature flag

• Realise feature is buggy

• Disable feature flag

Why Immutable ConfigMaps?

• Updating a ConfigMap used in a Deployment will not update the Pods in that Deployment

• Rolling back to a previous configuration will not cause the Pods to pick up the ConfigMap or Secrets changes

• kubectl rollout undo for emergency purposes will only roll back containers, not configuration

  Immutable ConfigMaps

• Name ConfigMaps based on a hash of its data

• Reference this ConfigMap name in a Deployment

• Changing a existing ConfigMap will change its name, triggering Pod updates

• Rolling back a Deployment will then roll back to the old config

• Use append_hash to generate immutable ConfigMaps

• Use k8s_config_resource_name filter plugin

Demo part two

Planned k8s improvements

• append_hash will enable immutable ConfigMaps and Secrets (likely 2.8)

• validate will return helpful warning and/or error messages if a resource manifest does not match the Kubernetes resource specification (likely 2.8)

• wait will allow you to wait until the Kubernetes resources are actually in the desired state (hopefully 2.8)