Install and Configure Kubernetes¶

MemVerge.ai supports multiple kubernetes flavors. Use the installation guides to provision Kubernetes in the AWS environment.

K3s: Follow the K3s Quick-Start Guide.

Installing K3s¶

This guide follows https://docs.k3s.io/quick-start.

K3s provides an installation script that is a convenient way to install it as a service on systemd or openrc based systems. This script is available at https://get.k3s.io.

Install K3s on the Management Node¶

To install K3s on the management node using this method, run:

curl -sfL https://get.k3s.io | sh -

If you do not want to run any jobs on the management host, run curl -sfL https://get.k3s.io | sh -s server instead.

Example output:

$ curl -sfL https://get.k3s.io | sh -

[INFO]  Finding release for channel stable
[INFO]  Using v1.31.5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.31.5+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.31.5+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

Run systemctl status k3s to verify the service is running. For example:

$ systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2025-02-20 23:56:47 UTC; 39s ago
       Docs: https://k3s.io
    Process: 1227 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 1229 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 1231 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 1232 (k3s-server)
      Tasks: 93
     Memory: 1.3G
        CPU: 33.197s
     CGroup: /system.slice/k3s.service
             ├─1232 "/usr/local/bin/k3s server"
             ├─1257 "containerd " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "">
             ├─1985 /var/lib/rancher/k3s/data/da3ffc1d30a49a23449847b31d95bf4c96c8551396573c18886c9d0c4a63c710/bin/containerd-shim-runc-v2 -namespace k8s.io -id 2f54140>
             ├─2018 /var/lib/rancher/k3s/data/da3ffc1d30a49a23449847b31d95bf4c96c8551396573c18886c9d0c4a63c710/bin/containerd-shim-runc-v2 -namespace k8s.io -id 92456dd>
             ├─2157 /var/lib/rancher/k3s/data/da3ffc1d30a49a23449847b31d95bf4c96c8551396573c18886c9d0c4a63c710/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5e2b356>
             ├─3028 /var/lib/rancher/k3s/data/da3ffc1d30a49a23449847b31d95bf4c96c8551396573c18886c9d0c4a63c710/bin/containerd-shim-runc-v2 -namespace k8s.io -id a88ccf7>
             └─3118 /var/lib/rancher/k3s/data/da3ffc1d30a49a23449847b31d95bf4c96c8551396573c18886c9d0c4a63c710/bin/containerd-shim-runc-v2 -namespace k8s.io -id ebd8daa>

Feb 20 23:57:22 mvai-mgmt k3s[1232]: I0220 23:57:22.908532    1232 resource_quota_monitor.go:227] "QuotaMonitor created object count evaluator" resource="middlewares.tr>
Feb 20 23:57:22 mvai-mgmt k3s[1232]: I0220 23:57:22.908548    1232 resource_quota_monitor.go:227] "QuotaMonitor created object count evaluator" resource="tlsoptions.tra>
Feb 20 23:57:22 mvai-mgmt k3s[1232]: I0220 23:57:22.908563    1232 resource_quota_monitor.go:227] "QuotaMonitor created object count evaluator" resource="ingressrouteud>
Feb 20 23:57:22 mvai-mgmt k3s[1232]: I0220 23:57:22.908578    1232 resource_quota_monitor.go:227] "QuotaMonitor created object count evaluator" resource="middlewaretcps>
Feb 20 23:57:22 mvai-mgmt k3s[1232]: I0220 23:57:22.908597    1232 resource_quota_monitor.go:227] "QuotaMonitor created object count evaluator" resource="middlewaretcps>
Feb 20 23:57:22 mvai-mgmt k3s[1232]: I0220 23:57:22.908616    1232 resource_quota_monitor.go:227] "QuotaMonitor created object count evaluator" resource="ingressroutetc>
Feb 20 23:57:22 mvai-mgmt k3s[1232]: I0220 23:57:22.908792    1232 shared_informer.go:313] Waiting for caches to sync for resource quota
Feb 20 23:57:23 mvai-mgmt k3s[1232]: I0220 23:57:23.009009    1232 shared_informer.go:320] Caches are synced for resource quota
Feb 20 23:57:23 mvai-mgmt k3s[1232]: I0220 23:57:23.323885    1232 shared_informer.go:313] Waiting for caches to sync for garbage collector
Feb 20 23:57:23 mvai-mgmt k3s[1232]: I0220 23:57:23.323937    1232 shared_informer.go:320] Caches are synced for garbage collector

Install K3s and Join Worker Nodes to the Cluster¶

To install additional agent nodes and add them to the cluster, run the installation script with the K3S_URL and K3S_TOKEN environment variables. Setting the K3S_URL parameter causes the installer to configure K3s as an agent, instead of a management server. The K3s agent will register with the K3s server listening at the supplied URL. The value to use for K3S_TOKEN is stored at /var/lib/rancher/k3s/server/node-token on your management node.

On the management host, run sudo cat /var/lib/rancher/k3s/server/node-token to get the K3S_TOKEN.

On the worker node, use the following to install K3s and join it to the cluster:

curl -sfL https://get.k3s.io | K3S_URL=https://mvai-mgmt:6443 K3S_TOKEN=<k3s_token> sh -

Example command and output:

ubuntu@mvai-nvgpu01:~$ curl -sfL https://get.k3s.io | K3S_URL=https://mvai-mgmt:6443 K3S_TOKEN=K10f125cd21ccff1f40d93fc85ee2080da622a7153b8b9d95f28ff2d8653817ee5e::server:719056123428a646178043d311a799d0 sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.31.5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.31.5+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.31.5+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

Confirm the service is running using systemctl status k3s-agent

Example output:

ubuntu@mvai-nvgpu01:~$ systemctl status k3s-agent
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2025-02-21 00:05:07 UTC; 1min 8s ago
       Docs: https://k3s.io
    Process: 1260 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 1262 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 1264 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 1266 (k3s-agent)
      Tasks: 43
     Memory: 347.1M
        CPU: 3.850s
     CGroup: /system.slice/k3s-agent.service
             ├─1266 "/usr/local/bin/k3s agent"
             ├─1295 "containerd " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "">
             └─1703 /var/lib/rancher/k3s/data/da3ffc1d30a49a23449847b31d95bf4c96c8551396573c18886c9d0c4a63c710/bin/containerd-shim-runc-v2 -namespace k8s.io -id 79e20ba>

Feb 21 00:05:08 mvai-nvgpu01 k3s[1266]: I0221 00:05:08.225029    1266 iptables.go:125] Setting up masking rules
Feb 21 00:05:08 mvai-nvgpu01 k3s[1266]: I0221 00:05:08.228352    1266 iptables.go:226] Changing default FORWARD chain policy to ACCEPT
Feb 21 00:05:08 mvai-nvgpu01 k3s[1266]: time="2025-02-21T00:05:08Z" level=info msg="Wrote flannel subnet file to /run/flannel/subnet.env"
Feb 21 00:05:08 mvai-nvgpu01 k3s[1266]: time="2025-02-21T00:05:08Z" level=info msg="Running flannel backend."
Feb 21 00:05:08 mvai-nvgpu01 k3s[1266]: I0221 00:05:08.230707    1266 vxlan_network.go:65] watching for new subnet leases
Feb 21 00:05:08 mvai-nvgpu01 k3s[1266]: I0221 00:05:08.230741    1266 subnet.go:152] Batch elem [0] is { lease.Event{Type:0, Lease:lease.Lease{EnableIPv4:true, EnableIP>
Feb 21 00:05:08 mvai-nvgpu01 k3s[1266]: I0221 00:05:08.230867    1266 vxlan_network.go:100] Received Subnet Event with VxLan: BackendType: vxlan, PublicIP: 172.31.25.21>
Feb 21 00:05:08 mvai-nvgpu01 k3s[1266]: I0221 00:05:08.243220    1266 iptables.go:372] bootstrap done
Feb 21 00:05:08 mvai-nvgpu01 k3s[1266]: I0221 00:05:08.251534    1266 iptables.go:372] bootstrap done
Feb 21 00:05:11 mvai-nvgpu01 k3s[1266]: I0221 00:05:11.046769    1266 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/svclb-traefik>

Configuring kubectl Access for Non-Root Users¶

After installing K3s, you may encounter permission issues when running kubectl commands as a non-root user. This section guides you through setting up kubectl access for your user account without requiring sudo privileges.

Set up the KUBECONFIG environment variable

Add the following line to your .bashrc file to set the KUBECONFIG environment variable:
```
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
source ~/.bashrc
```
This step ensures that kubectl knows where to find your configuration file.
Create a local copy of the K3s configuration

Run these commands to create a local copy of the K3s configuration file:
```
mkdir -p ~/.kube
sudo k3s kubectl config view --raw > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"
```
This creates a .kube directory in your home folder, copies the K3s configuration to it, and sets appropriate permissions.

Verify the setup

Test your configuration by running:

kubectl get nodes

You should now be able to see the list of nodes in your cluster without any permission errors. For example:

ubuntu@mvai-mgmt:~$ kubectl get nodes
NAME           STATUS   ROLES                  AGE   VERSION
mvai-mgmt      Ready    control-plane,master   53m   v1.31.5+k3s1
mvai-nvgpu01   Ready    <none>                 45m   v1.31.5+k3s1

Show the running pods in all namespaces

kubectl get pods -A

Example:

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-ccb96694c-wpq25                   1/1     Running     0          32m
kube-system   helm-install-traefik-crd-77lxh            0/1     Completed   0          32m
kube-system   helm-install-traefik-l598j                0/1     Completed   1          32m
kube-system   local-path-provisioner-5b5f758bcf-svnrn   1/1     Running     0          32m
kube-system   metrics-server-7bf7d58749-t5j45           1/1     Running     0          32m
kube-system   svclb-traefik-45be11ee-kbtp9              2/2     Running     0          32m
kube-system   svclb-traefik-45be11ee-r4xhz              2/2     Running     0          22m
kube-system   traefik-5cbdcf97f4-xj2ln                  1/1     Running     0          32m

Troubleshooting¶

If you encounter issues after following these steps, verify that the K3s service is running:

sudo systemctl status k3s

If the service is not active, start it with:

sudo systemctl start k3s

Security Note¶

This configuration creates a user-specific kubeconfig file, avoiding the need to modify system-wide permissions. Always ensure that your home directory has appropriate permissions to protect the kubeconfig file.