This is an old revision of the document!

Step 3 - Set up Kubernetes

In our previous step we deployed our fleet of VMs.

Now we are going to install Kubernetes on the VMs:

The first will be the Kubernetes (k8s) master node
The second will be the node running the SQL service
The remaining VMs will be the “worker” nodes

Purpose:

Demonstrate a running a complex workload of web applications on Kubernetes

References

Update the inventory File

Modify the inventory file to assign nodes to groups. Here is an example assigning my server IPs to each group.

inventory

[master]
192.168.99.201
[sql]
192.168.99.202
[workers]
192.168.99.203
192.168.99.204

Configure ansible.cfg to use the inventory file

ansible.cfg

[defaults]
inventory         = inventory

Update the /etc/hosts files with the hostnames. This allows all nodes to resolve each other by name, but without DNS

updatehostsfile.yml

---
- name: Update etc/hosts file
  hosts: all, localhost
  gather_facts: true
  tasks:
    - name: Populate all /etc/hosts files
      tags: etchostsupdate
      become: true
      become_user: root
      lineinfile:
        path: "/etc/hosts"
        regexp: '.*{{ item }}$'
        line: "{{ hostvars[item]['ansible_default_ipv4'].address }}\t{{ hostvars[item]['ansible_hostname'] }}\t{{ hostvars[item]['ansible_hostname'] }}"
        state: present
      with_items: '{{ groups.all }}'

Run

ansible-playbook updatehostsfile.yml --ask-become

Install Kubernetes using Ansible

Check https://github.com/torgeirl/kubernetes-playbooks for updates to the playbooks below

Install some prerequisites on ALL the Kubernetes nodes

kube-dependencies.yml

---
- hosts: all
  become: true
  tasks:
    - fail:
        msg: "OS should be Ubuntu 22.04, not {{ ansible_distribution }} {{ ansible_distribution_version }}"
      when: ansible_distribution != 'Ubuntu' or ansible_distribution_version != '22.04'

    - name: Update APT packages
      apt:
        update_cache: true

    - name: Reboot and wait for reboot to complete
      reboot:

    - name: Disable SWAP (Kubeadm requirement)
      shell: |
        swapoff -a

    - name: Disable SWAP in fstab (Kubeadm requirement)
      replace:
        path: /etc/fstab
        regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
        replace: '# \1'

    - name: Create an empty file for the Containerd module
      copy:
        content: ""
        dest: /etc/modules-load.d/containerd.conf
        force: false

    - name: Configure modules for Containerd
      blockinfile:
        path: /etc/modules-load.d/containerd.conf
        block: |
             overlay
             br_netfilter

    - name: Create an empty file for Kubernetes sysctl params
      copy:
        content: ""
        dest: /etc/sysctl.d/99-kubernetes-cri.conf
        force: false

    - name: Configure sysctl params for Kubernetes
      lineinfile:
        path: /etc/sysctl.d/99-kubernetes-cri.conf
        line: "{{ item }}"
      with_items:
        - 'net.bridge.bridge-nf-call-iptables  = 1'
        - 'net.ipv4.ip_forward                 = 1'
        - 'net.bridge.bridge-nf-call-ip6tables = 1'

    - name: Apply sysctl params without reboot
      command: sysctl --system

    - name: Install APT Transport HTTPS
      apt:
        name: apt-transport-https
        state: present

    - name: Add Docker apt-key
      apt_key:
        url: https://download.docker.com/linux/ubuntu/gpg
        state: present

    - name: Add Docker's APT repository
      apt_repository:
        repo: "deb [arch=amd64] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
        filename: "docker-{{ ansible_distribution_release }}"

    - name: Add Kubernetes apt-key
      apt_key:
        url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
        state: present

    - name: Add Kubernetes' APT repository
      apt_repository:
        repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
        state: present
        filename: 'kubernetes'

    - name: Install Containerd
      apt:
        name: containerd.io
        state: present

    - name: Create Containerd directory
      file:
        path: /etc/containerd
        state: directory

    - name: Add Containerd configuration
      shell: /usr/bin/containerd config default > /etc/containerd/config.toml

    - name: Configuring the systemd cgroup driver for Containerd
      lineinfile:
        path: /etc/containerd/config.toml
        regexp: '            SystemdCgroup = false'
        line: '            SystemdCgroup = true'

    - name: Enable the Containerd service and start it
      systemd:
        name: containerd
        state: restarted
        enabled: true
        daemon-reload: true

    - name: Install Kubelet
      apt:
        name: kubelet=1.26.*
        state: present
        update_cache: true

    - name: Install Kubeadm
      apt:
        name: kubeadm=1.26.*
        state: present

    - name: Enable the Kubelet service, and enable it persistently
      service:
        name: kubelet
        enabled: true

    - name: Load br_netfilter kernel module
      modprobe:
        name: br_netfilter
        state: present

    - name: Set bridge-nf-call-iptables
      sysctl:
        name: net.bridge.bridge-nf-call-iptables
        value: 1

    - name: Set ip_forward
      sysctl:
        name: net.ipv4.ip_forward
        value: 1

    - name: Check Kubelet args in Kubelet config
      shell: grep "^Environment=\"KUBELET_EXTRA_ARGS=" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf || true
      register: check_args
 
    - name: Add runtime args in Kubelet config
      lineinfile:
        dest: "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
        line: "Environment=\"KUBELET_EXTRA_ARGS= --runtime-cgroups=/system.slice/containerd.service --container-runtime-endpoint=unix:///run/containerd/containerd.sock\""
        insertafter: '\[Service\]'
      when: check_args.stdout == ""
 
    - name: Reboot and wait for reboot to complete
      reboot:
 
- hosts: master
  become: true
  tasks:
    - name: Install Kubectl
      apt:
        name: kubectl=1.26.*
        state: present
        force: true  # allow downgrades

Run ansible-playbook kube-dependencies.yml

Configure kubernetes cluster on master node

master.yml

---
- hosts: master
  become: true
  tasks:
    - name: Create an empty file for Kubeadm configuring
      copy:
        content: ""
        dest: /etc/kubernetes/kubeadm-config.yaml
        force: false

    - name: Configuring the container runtime including its cgroup driver
      blockinfile:
        path: /etc/kubernetes/kubeadm-config.yaml
        block: |
             kind: ClusterConfiguration
             apiVersion: kubeadm.k8s.io/v1beta3
             networking:
               podSubnet: "10.244.0.0/16"
             ---
             kind: KubeletConfiguration
             apiVersion: kubelet.config.k8s.io/v1beta1
             runtimeRequestTimeout: "15m"
             cgroupDriver: "systemd"
             systemReserved:
               cpu: 100m
               memory: 350M
             kubeReserved:
               cpu: 100m
               memory: 50M
             enforceNodeAllocatable:
             - pods

    - name: Initialize the cluster (this could take some time)
      shell: kubeadm init --config /etc/kubernetes/kubeadm-config.yaml >> cluster_initialized.log
      args:
        chdir: /home/ansible
        creates: cluster_initialized.log

    - name: Create .kube directory
      become: true
      become_user: ansible
      file:
        path: $HOME/.kube
        state: directory
        mode: 0755

    - name: Copy admin.conf to user's kube config
      copy:
        src: /etc/kubernetes/admin.conf
        dest: /home/ansible/.kube/config
        remote_src: true
        owner: ansible

    - name: Install Pod network
      become: true
      become_user: ansible
      shell: kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml >> pod_network_setup.log
      args:
        chdir: $HOME
        creates: pod_network_setup.log

run ansible-playbook master.yml

SSH to the master and verify the master nodes gets status Ready
- ssh MASTER_IP kubectl get nodes
Initialize the SQL worker?

Initialize workers

Modify the file to replace MASTERIP with the IP address of your master node in two (2) places

workers.yml

---
- hosts: master
  become: true
  # gather_facts: false
  tasks:
    - name: Get join command
      shell: kubeadm token create --print-join-command
      register: join_command_raw

    - name: Set join command
      set_fact:
        join_command: "{{ join_command_raw.stdout_lines[0] }}"

- hosts: workers
  become: true
  tasks:
    - name: TCP port 6443 on master is reachable from worker
      wait_for: "host={{ 'MASTERIP' }} port=6443 timeout=1"

    - name: Join cluster
      shell: "{{ hostvars['MASTERIP'].join_command }} >> node_joined.log"
      args:
        chdir: /home/ansible
        creates: node_joined.log

ansible-playbook workers.yml

SSH to the master and verify all the nodes return status Ready
- ssh MASTER_IP kubectl get nodes

Install kubectl on NUC 2 for automation with Kubernetes

create file /home/ansible/my-project/k8s/kubectlcontrolnode.yml

kubectlcontrolnode.yml

---
- hosts: localhost
  become: true
  gather_facts: false
  tasks:
    - name: Update APT packages
      apt:
        update_cache: true

    - name: Add Kubernetes apt-key
      apt_key:
        url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
        state: present

    - name: Add Kubernetes' APT repository
      apt_repository:
        repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
        state: present
        filename: 'kubernetes'

    - name: Install Kubectl
      apt:
        name: kubectl=1.26.*
        state: present
        force: true  # allow downgrades

ansible-playbook kubectlcontrolnode.yml
Running kubctl version will fail at this point because you don't have credentials
Copy credentials
- scp -r IPMASTER:/home/ansible/.kube ~/
Confirm it's working by running kubectl version

Install kubectl on NUC 1 for remote testing

sudo apt update
sudo apt install -y ca-certificates curl
sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubectl

Running kubctl version will fail at this point because you don't have credentials
Copy credentials
- scp -r tux@IPMASTER:/home/ansible/.kube ~/
Confirm it's working by running
- kubectl version
- kubectl get nodes

Next Step

Continue to Step 4 - SQL Server

Or back to Step 2 - Deploy the VMs or Start.

UncleNUC Wiki

Table of Contents

Step 3 - Set up Kubernetes

Update the inventory File

Install Kubernetes using Ansible

Next Step