This is an old revision of the document!

Ansible Playbook - Install Kubernetes

In our previous step we cleaned up the FAH installation on our Stack of NUCs.

Now we are going to install Kubernetes with:

one NUC master node this we can call NUC 3
remaining NUCs “workers”

Purpose:

Demonstrate a running a complex workload of web applications on Kubernetes

References

Step 1 - Set Up for Kubernetes

From NUC 1, log in to the Ansible control node, NUC 2.
Create the k8s folder on the Ansible control node, NUC 2
- mkdir /home/ansible/my-project/k8s

Create inventory file

Choose one IP address to be the Kubernetes master, add it to [master]
The rest of the IP address should be added to [workers]

inventory

[master]
[workers]
 
[all:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_user='ansible'
ansible_become=true
ansible_become_method=sudo

Set up ansible.cfg file to tell Ansible to use the inventory file
- ansible.cfg
  [defaults] inventory = inventory

Set up the host files for name resolution

Playbook will update the '/etc/hosts' files on all the nodes to allow them to resolve each other by name, but without DNS
Also adds this information to the Ansible control node (see 'localhost' in the script)

updatehostsfile.yml

---
- name: Update etc/hosts file
  hosts: all, localhost
  gather_facts: true
  tasks:
    - name: Populate all /etc/hosts files
      tags: etchostsupdate
      become: true
      become_user: root
      lineinfile:
        path: "/etc/hosts"
        regexp: '.*{{ item }}$'
        line: "{{ hostvars[item]['ansible_default_ipv4'].address }}\t{{ hostvars[item]['ansible_hostname'] }}\t{{ hostvars[item]['ansible_hostname'] }}"
        state: present
      with_items: '{{ groups.all }}'

ansible-playbook updatehostsfile.yml

Step 2 - Set Up Kubernetes Using Ansible

Check https://github.com/torgeirl/kubernetes-playbooks for updates to the playbooks below

Install some prerequisites on ALL the Kubernetes nodes

kube-dependencies.yml

---
- hosts: all
  become: true
  tasks:
    - fail:
        msg: "OS should be Ubuntu 22.04, not {{ ansible_distribution }} {{ ansible_distribution_version }}"
      when: ansible_distribution != 'Ubuntu' or ansible_distribution_version != '22.04'

    - name: Update APT packages
      apt:
        update_cache: true

    - name: Reboot and wait for reboot to complete
      reboot:

    - name: Disable SWAP (Kubeadm requirement)
      shell: |
        swapoff -a

    - name: Disable SWAP in fstab (Kubeadm requirement)
      replace:
        path: /etc/fstab
        regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
        replace: '# \1'

    - name: Create an empty file for the Containerd module
      copy:
        content: ""
        dest: /etc/modules-load.d/containerd.conf
        force: false

    - name: Configure modules for Containerd
      blockinfile:
        path: /etc/modules-load.d/containerd.conf
        block: |
             overlay
             br_netfilter

    - name: Create an empty file for Kubernetes sysctl params
      copy:
        content: ""
        dest: /etc/sysctl.d/99-kubernetes-cri.conf
        force: false

    - name: Configure sysctl params for Kubernetes
      lineinfile:
        path: /etc/sysctl.d/99-kubernetes-cri.conf
        line: "{{ item }}"
      with_items:
        - 'net.bridge.bridge-nf-call-iptables  = 1'
        - 'net.ipv4.ip_forward                 = 1'
        - 'net.bridge.bridge-nf-call-ip6tables = 1'

    - name: Apply sysctl params without reboot
      command: sysctl --system

    - name: Install APT Transport HTTPS
      apt:
        name: apt-transport-https
        state: present

    - name: Add Docker apt-key
      apt_key:
        url: https://download.docker.com/linux/ubuntu/gpg
        state: present

    - name: Add Docker's APT repository
      apt_repository:
        repo: "deb [arch=amd64] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
        filename: "docker-{{ ansible_distribution_release }}"

    - name: Add Kubernetes apt-key
      apt_key:
        url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
        state: present

    - name: Add Kubernetes' APT repository
      apt_repository:
        repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
        state: present
        filename: 'kubernetes'

    - name: Install Containerd
      apt:
        name: containerd.io
        state: present

    - name: Create Containerd directory
      file:
        path: /etc/containerd
        state: directory

    - name: Add Containerd configuration
      shell: /usr/bin/containerd config default > /etc/containerd/config.toml

    - name: Configuring the systemd cgroup driver for Containerd
      lineinfile:
        path: /etc/containerd/config.toml
        regexp: '            SystemdCgroup = false'
        line: '            SystemdCgroup = true'

    - name: Enable the Containerd service and start it
      systemd:
        name: containerd
        state: restarted
        enabled: true
        daemon-reload: true

    - name: Install Kubelet
      apt:
        name: kubelet=1.26.*
        state: present
        update_cache: true

    - name: Install Kubeadm
      apt:
        name: kubeadm=1.26.*
        state: present

    - name: Enable the Kubelet service, and enable it persistently
      service:
        name: kubelet
        enabled: true

    - name: Load br_netfilter kernel module
      modprobe:
        name: br_netfilter
        state: present

    - name: Set bridge-nf-call-iptables
      sysctl:
        name: net.bridge.bridge-nf-call-iptables
        value: 1

    - name: Set ip_forward
      sysctl:
        name: net.ipv4.ip_forward
        value: 1

    - name: Check Kubelet args in Kubelet config
      shell: grep "^Environment=\"KUBELET_EXTRA_ARGS=" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf || true
      register: check_args
 
    - name: Add runtime args in Kubelet config
      lineinfile:
        dest: "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
        line: "Environment=\"KUBELET_EXTRA_ARGS= --runtime-cgroups=/system.slice/containerd.service --container-runtime-endpoint=unix:///run/containerd/containerd.sock\""
        insertafter: '\[Service\]'
      when: check_args.stdout == ""
 
    - name: Reboot and wait for reboot to complete
      reboot:
 
- hosts: master
  become: true
  tasks:
    - name: Install Kubectl
      apt:
        name: kubectl=1.26.*
        state: present
        force: true  # allow downgrades

ansible-playbook kube-dependencies.yml

Configure kubernetes cluster on master node

master.yml

---
- hosts: master
  become: true
  tasks:
    - name: Create an empty file for Kubeadm configuring
      copy:
        content: ""
        dest: /etc/kubernetes/kubeadm-config.yaml
        force: false

    - name: Configuring the container runtime including its cgroup driver
      blockinfile:
        path: /etc/kubernetes/kubeadm-config.yaml
        block: |
             kind: ClusterConfiguration
             apiVersion: kubeadm.k8s.io/v1beta3
             networking:
               podSubnet: "10.244.0.0/16"
             ---
             kind: KubeletConfiguration
             apiVersion: kubelet.config.k8s.io/v1beta1
             runtimeRequestTimeout: "15m"
             cgroupDriver: "systemd"
             systemReserved:
               cpu: 100m
               memory: 350M
             kubeReserved:
               cpu: 100m
               memory: 50M
             enforceNodeAllocatable:
             - pods

    - name: Initialize the cluster (this could take some time)
      shell: kubeadm init --config /etc/kubernetes/kubeadm-config.yaml >> cluster_initialized.log
      args:
        chdir: /home/ansible
        creates: cluster_initialized.log

    - name: Create .kube directory
      become: true
      become_user: ansible
      file:
        path: $HOME/.kube
        state: directory
        mode: 0755

    - name: Copy admin.conf to user's kube config
      copy:
        src: /etc/kubernetes/admin.conf
        dest: /home/ansible/.kube/config
        remote_src: true
        owner: ansible

    - name: Install Pod network
      become: true
      become_user: ansible
      shell: kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml >> pod_network_setup.log
      args:
        chdir: $HOME
        creates: pod_network_setup.log

ansible-playbook master.yml

SSH to the master and verify the master nodes gets status Ready
- ssh MASTER_IP kubectl get nodes

Initialize workers

Modify the file to replace MASTERIP with the IP address of your master node in two (2) places

workers.yml

---
- hosts: master
  become: true
  # gather_facts: false
  tasks:
    - name: Get join command
      shell: kubeadm token create --print-join-command
      register: join_command_raw

    - name: Set join command
      set_fact:
        join_command: "{{ join_command_raw.stdout_lines[0] }}"

- hosts: workers
  become: true
  tasks:
    - name: TCP port 6443 on master is reachable from worker
      wait_for: "host={{ 'MASTERIP' }} port=6443 timeout=1"

    - name: Join cluster
      shell: "{{ hostvars['MASTERIP'].join_command }} >> node_joined.log"
      args:
        chdir: /home/ansible
        creates: node_joined.log

ansible-playbook workers.yml

SSH to the master and verify all the nodes return status Ready
- ssh MASTER_IP kubectl get nodes

Install kubectl on NUC 2 for automation with Kubernetes

create file /home/ansible/my-project/k8s/kubectlcontrolnode.yml

kubectlcontrolnode.yml

---
- hosts: localhost
  become: true
  gather_facts: false
  tasks:
    - name: Update APT packages
      apt:
        update_cache: true

    - name: Add Kubernetes apt-key
      apt_key:
        url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
        state: present

    - name: Add Kubernetes' APT repository
      apt_repository:
        repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
        state: present
        filename: 'kubernetes'

    - name: Install Kubectl
      apt:
        name: kubectl=1.26.*
        state: present
        force: true  # allow downgrades

ansible-playbook kubectlcontrolnode.yml
Running kubctl version will fail at this point because you don't have credentials
Copy credentials
- scp -r IPMASTER:/home/ansible/.kube ~/
Confirm it's working by running kubectl version

Install kubectl on NUC 1 for remote testing

sudo apt update
sudo apt install -y ca-certificates curl
sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubectl

Running kubctl version will fail at this point because you don't have credentials
Copy credentials
- scp -r tux@IPMASTER:/home/ansible/.kube ~/
Confirm it's working by running
- kubectl version
- kubectl get nodes

Next Step

You have successful installed Kubernetes on your Stack of NUCs! Next we will demonstrate running a web app on Kubernetes.

Optional

Do some tests with Kubernetes.

Manually Create Pod

kubectl run speedtester --image=docker.io/doritoes/speedtester:latest

kubectl get pods
- kubectl get pods -w will “watch” the process of the container being created and changing status to Running; press control-c to exit
kubectl describe pod speedtester
kubectl delete pods/speedtester
- this demonstrates another way to specify the pod, same as kubectl delete pod speedtester

Create Pod using Pod Manifest

Create yaml file

speedtest-pod.yml

apiVersion: v1
kind: Pod
metadata:
  name: speedtester
spec:
  containers:
    - image: docker.io/doritoes/speedtester:latest
      name: speedtester
      ports:
        - containerPort: 8080
          name: http
          protocol: TCP

Build the pod
- kubectl apply -f speedtest-pod.yml
Examine the pod
- kubectl get pods
- kubectl describe pods speedtester
Test connecting to the speedtest application on the pod
- from NUC 1
  - kubectl port-forward speedtester 8080:80
  - browse to http://localhost:8080
  - NOTE in testing, the port forwarding tended to break when it the upload test started (“error creating error stream for port 8080 → 80: Timeout occurred”)
    - the worker node NUC doesn't seem to be loaded at all
      - kubectl exec –stdin –tty speedtester – /bin/bash
      - htop
    - one workaround is to keep re-launching the port forward command, but that doesn't solve the upload issue
    - later on, we will demonstrate it is stable when using a better forwarding mechanism
    - while true; do kubectl port-forward speedtester 8080:80; done

Manage the pod

kubectl logs speedtester
kubectl exec speedtester -- date
kubectl exec speedtester -- uname -a

Delete the pod

kubectl delete -f speedtest-pod.yaml
kubectl get pods

k9s

You can experiment running k9s on your Kubernetes master node

https://github.com/derailed/k9s/releases

mkdir k9s
cd k9s
wget https://github.com/derailed/k9s/releases/download/v0.27.3/k9s_Linux_amd64.tar.gz
tar xzvf k9s_Linux_amd64.tar.gz
./k9s

Kubernetes Dashboard

https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

This is a more advanced exercise to try. I won't be covering this in detail here.

Step 1 - Install
- Install the dashboard from NUC 1 or NUC 2
  - kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

Step 2 - Create a service account

kubernetes-dashboard.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

kubectl apply -f kubernetes-dashboard.yml

Step 3 - RoleBinding for this service account
- kubectl create serviceaccount dashboard -n kubernetes-dashboard or the following
- ```
kubectl create clusterrolebinding dashboard-admin -n kubernetes-dashboard  --clusterrole=cluster-admin  --serviceaccount=default:dashboar
```
- kubectl apply -f rolebinding.yml
Step 4 - Get a Bearer Token
- kubectl -n kubernetes-dashboard create token admin-user
- or kubectl get secret $(kubectl get serviceaccount dashboard -o jsonpath=“{.secrets[0].name}”) -o jsonpath=“{.data.token}” | base64 –decode
Step 5 - Access
- From NUC 1 run
  - kubectl proxy
  - UI can only be accessed from the machine where the command is executed
  - See kubectl proxy –help for more options.
- From a web browser on NUC 1 open:
  - http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
- Authenticate with “Token” and paste in the token from Step 3 above
Step - 6 Deploy containerized applications
- https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

UncleNUC Wiki

Table of Contents