Multi node K3s Installation

Install K3s

Note:

Make sure you have /usr/local/bin configured in your PATH: export PATH=$PATH:/usr/local/bin). All the commands must be executed as root user.
The commands have been tested on Ubuntu Server 20.04 LTS, SUSE Linux Enterprise Server 15 SP4 and RHEL 8.6.
For RHEL, K3s needs the following package to be installed: k3s-selinux (repo rancher-k3s-common-stable) and its dependencies container-selinux (repo rhel-8-appstream-rhui-rpms) and policycoreutils-python-utils (repo rhel-8-baseos-rhui-rpms). Also, firewalldnm-cloud-setup.service and nm-cloud-setup.timer must be disabled and the server restarted before the installation, click here for more information.

The steps below you guide you through the air-gap installation of K3s, a lightweight Kubernetes distribution created by Rancher Labs:

Create at least 4 VMs with the same specs.
Extract the downloaded file: tar -xf gv-platform-$VERSION.tar to all the VMs
Create a local DNS entry private-docker-registry.local across all the nodes resolving to the master1 node:
```
cat >> /etc/hosts  << EOF
<Master1_node_VM_IP>  private-docker-registry.local
EOF
```

Prepare the K3s for air-gap installation files:

$ mkdir -p /var/lib/rancher/k3s/agent/images/
$ gunzip -c assets/k3s-airgap-images-amd64.tar.gz > /var/lib/rancher/k3s/agent/images/airgap-images.tar
$ cp assets/k3s /usr/local/bin && chmod +x /usr/local/bin/k3s
$ tar -xzf assets/helm-v3.8.2-linux-amd64.tar.gz && cp linux-amd64/helm /usr/local/bin

Update the registries.yaml file across all the nodes.

$ mkdir -p /etc/rancher/k3s
$ cp assets/registries.yaml  /etc/rancher/k3s/

Install K3s in the 1st master node:
To get started launch a server node using the cluster-init flag:
```
cat scripts/k3s.sh | INSTALL_K3S_SKIP_DOWNLOAD=true K3S_KUBECONFIG_MODE="644" sh -s - server --node-name=master1 --cluster-init
```
Check for your first master node status, it should have the Ready state:
```
kubectl get nodes
```
Use the following command to copy the TOKEN from this node that will be used to join the other nodes to the cluster:
```
cat /var/lib/rancher/k3s/server/node-token
```
Also, copy the IP address of the 1st master node which will be used by the other nodes to join the cluster.
Install K3s in the 2nd master node:

Run the following command and assign the contents of the file: /var/lib/rancher/k3s/server/node-token from the 1st master node to the K3S_TOKEN variable.

Set --node-name to “master2”

Set --server to the IP address of the 1st master node

cat scripts/k3s.sh | K3S_TOKEN=$K3S_TOKEN INSTALL_K3S_SKIP_DOWNLOAD=true K3S_KUBECONFIG_MODE="644" sh -s - server --node-name=master2 --server https://<ip or hostname of any master node>:6443

Check the node status:

kubectl get nodes

Install K3s in the 3rd master node:
Run the following command and assign the contents of the file: /var/lib/rancher/k3s/server/node-token from the 1st master node to the K3S_TOKEN variable.

Set --node-name to “master3”
Set --server to the IP address of the 1st master node.
```
cat scripts/k3s.sh | K3S_TOKEN=$K3S_TOKEN INSTALL_K3S_SKIP_DOWNLOAD=true K3S_KUBECONFIG_MODE="644" sh -s - server --node-name=master3 --server https://<ip or hostname of any master node>:6443
```
Check the node status:
Install K3s in the 1st worker node:
Use the same approach to install K3s and to connect the worker node to the cluster group. The installation parameter would be different in this case. Run the following command: Set --node-name to “worker1” (where n is the nth number of the worker node)
```
cat scripts/k3s.sh | $K3S_TOKEN INSTALL_K3S_SKIP_DOWNLOAD=true K3S_TOKEN=$K3S_TOKEN K3S_KUBECONFIG_MODE="644" sh -s - agent --node-name=worker1 --server https://<ip or hostname of any master node>:6443
```
Check the node status:
```
kubectl get nodes
```