100 Ways to NOT Install OpenShift Virtualization
100 Ways to NOT Install
OpenShift Virtualization
Why?
What?
How?
Dear Lord. This is my own fault. I went into this multi-hypervisor research/science project with as little RTFM as possible. I wanted to see how easy/hard each install was to the average IT person. I think I found _all_ the ways to fail at installing OpenShift Virtualization – bare metal, vSphere (kind of defeats the purpose), single-node, laptop local. It really is impressive all the methods and platforms that OpenShift will run on. You can find the entire list here.
Let’s walk through all my failures and my near successes… 😉
================================================
#1 Bare Metal – Failed on Boot
To create _most_ cluster types, just click the “Create cluster” button
There are currently 5 Cloud options, 9 on-premises Datacenter options, and 3 Local (laptop) options. Let’s choose “Bare Metal (x86_64)” Datacenter option to match our NUC.
I gave my cluster a name, entered the local domain name, chose Single Node OpenShift (SNO), and clicked Next
On the next screen, I chose OpenShift Virtualization (which then checked LVM) and clicked Next
On the next screen, I clicked “Add Host” to get access to the custom boot ISO. The “Minimum hardware requirements” link pops up this window
(Um, my NUC has 4 cores and 8 threads, so will most likely fail install)
OK. I can now download the custom ISO for boot and install.
I will place this ISO in my trusty Ventoy USB key and boot the NUC.
Oops. System panic. Drop to shell. I could not copy off the log to investigate. Most likely, not enough CPU cores. The NUC never reached the RedHat console over the Internet so no messages there. Failure #1.
================================================
#2 Bare Metal-Ish – Failed on Virtualization Operator Install
Host showing up in the cloud console but has warning. Let’s investigate.
I’ll take the defaults and click Next again
We get a summary of what is about to be done (I collapsed the sections to fit on one screen). Next, click Install cluster
During the install process, you can click View cluster event if you want to see what is going on behind the scenes.
An initial burst of network activity as the OpenShift pieces are downloaded, then CPU activity as they are installed.
After the first automated reboot, a metric ton of containers get installed.
Let’s see if the console will respond. Probably not as it has all its CPU cores pegged.
Maybe not yet…
nmap is showing that ports are open including the console port of 6443. Probably just too busy to respond
I saved mine in ~/.kube/nuc-kubeconfig
After 2 hours of installation, and a brief period where the VM was not crushing all 16 vCPUs, I was able to login to the console with the username kubeadmin and the password shown in the cloud console
In terms of the OpenShift Virtualization menu items, the OpenShift Virtualization Operator is still being installed. Once installed, I will be prompted to refresh the OpenShift console page and the Virtualization menu items will be added.
Most of the Operator installation messages are of this type probably due to CPU contention: “
ComponentUnhealthy: installing: waiting for deployment virt-operator to become ready: deployment “virt-operator” not available: Deployment does not have minimum availability.”
================================================
#3 Multi-Node vSphere Deployment – Failed on VM Creation
================================================
#4 Single-Node Install as ESXi VM on 8 Core NUC – Failed on Not Enough Cores
Let is settle. CPU and RAM activity are now at zero, RH console says: Failed
Minimum CPU cores for selected role: Require at least 15 CPU cores for master role, found only 8.
================================================
#5 Laptop Install as ESXi VM on 8 Core/24 GB Fedora VM – Failed Install of Virtualization Operator
NAME READY STATUS RESTARTS
aaq-operator-6985fcd6d6-6wxc2 0/1 Pending 0
cdi-operator-75d967b479-vg9rf 0/1 Pending 0
cluster-network-addons-operator-84488b7548-vhxhz 2/2 Running 0
hco-operator-c9964b489-pk7d6 1/1 Running 0
hco-webhook-b647d765d-p27ml 1/1 Running 0
hostpath-provisioner-operator-67d7b9768-ksslc 0/1 Pending 0
hyperconverged-cluster-cli-download-59f9f5867f-dt4w8 1/1 Running 0
mtq-operator-c499999fc-bpqmn 0/1 Pending 0
ssp-operator-555bb4c8bc-mvdbn 1/1 Running 0
virt-operator-6d558dd77f-s4xcl 0/1 Pending 0
virt-operator-6d558dd77f-vlxvc 0/1 Pending 0
Finally, the install of the Operator gave up
Operator failed
install failed: deployment virt-operator not ready before timeout: deployment “virt-operator” exceeded its progress deadline. Failure #5
Let’s try giving the VM more RAM
================================================
#6 Laptop Install as ESXi VM on 8 Core/32 GB Fedora VM – Failed Install of Virtualization Operator
$ crc config get memory
Configuration property ‘memory’ is not set. Default value ‘10752’ is used
$ crc config get cpus
Configuration property ‘cpus’ is not set. Default value ‘4’ is used
ERRO Cluster is not ready: cluster operators are still not stable after 10m0.784655588s Failure #6
INFO Adding crc-admin and crc-developer contexts to kubeconfig…
Started the OpenShift cluster.
The server is accessible via web console at:
https://console-openshift-console.apps-crc.testing
Log in as administrator:
Username: kubeadmin
Password: jiFr4-ckMXJ-yGGoY-vmMqJ
Log in as user:
Username: developer
Password: developer
Use the ‘oc’ command line interface:
$ eval $(crc oc-env)
$ oc login -u developer https://api.crc.testing:6443
$ crc config set memory 24576
$ crc config set cpus 8
$ crc start
[SNIP]
Started the OpenShift cluster.
The GUI did not request a restart to add the VM menu items. I logged out and back in but still no VM menu items
Let’s stop and start the cluster and see what happens
================================================
#7 Laptop Install on Windows VM – Failed Install of Virtualization Operator
Cluster is progressing. Slowly. I noticed that CPU is pegged. I think I should increase the number of vCPUs from 8 to 16 and restart.
Almost there “2 operators are progressing: console, ingress” 🤞
Nope
Let’s see how the “oc login” command fares…
Son of a gun. I left this alone for about an hour to delight a customer and when I came back the GUI was available
The OpenShift Virtualization Operator is not installed yet. I’ll install the Operator as I have had so much success with this so far 🙄
================================================
#8 Public Cloud Install – Failed Creating Virtual Machine
There are many cloud options, I will choose the RedHat Trial option.
https://console.redhat.com/openshift/create/osdtrial?trial=osd
OK. Let’s figure this out.
Known issue: https://access.redhat.com/solutions/6960534
“After creating the user “osdCcsAdmin”, provide the IAM user credentials ( access key ID and secret access key ) to Red Hat through OpenShift Cluster Manager.”
OK, we’re off!
My credit card is already smoking 🙂
Holy Mother! I have a cluster on AWS. Let’s see if the GUI works…
I have no idea what the kubeadmin password is…
OK. I was too dumb to set up GitHub auth or Google Oauth, but I added simple htpasswd auth with a username and password and logged into the cluster
There are 13 OpenShift Operators installed automatically but not OpenShift Virtualization
Kicked off an install of the Virtualization Operator
Installed quickly without drama
Menus did not update to reflect virtualization. Going to try installing this:
Finally got the request to refresh the GUI for the added Virtualization menus
Virtualization menus are now there
Let’s try uploading a Bootable Volume
Miraculously, that worked. Now let’s create a VM.
I’ll use the Fedora template and use the ISO I just uploaded to “Boot from CD”
I’ve seen “ErrorUnschedulable” be a red herring before. Let’s see if that fixes itself after a minute.
Hmm… The Diagnostics tab for this VM lists this error:
“0/7 nodes are available: 2 Insufficient devices.kubevirt.io/kvm, 2 node(s) had untolerated taint {node-role.kubernetes.io/infra: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/7 nodes are available: 2 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..”
Also, the clone of my Boot CD never cloned. That is a problem.
Just for fun, lets create another VM from an Instance Type but with my Boot CD rather than a cloud init disk
Apparently, not a good idea
OK, let’s create a VM from and instance with all defaults. Failed.
I noticed that I did not have a Host Path Provisioner deployed for the OpenShift Virtualization Operator. Maybe that is important. I created one.
Let’s create a new VM
That didn’t fix it. All VMs fail due to ErrorUnschedulable which means a new pod could not be created.
Failure #∞
================================================
That’s it. I’m out. This was a great learning experience. Kubernetes is awesome for highly-available and scalable containers _not_ virtual machines. 🙂
https://console.redhat.com/openshift/create/osdtrial?trial=osd
My credit card is already smoking 🙂
I have no idea what the kubeadmin password is…
OK. I was too dumb to set up GitHub auth or Google Oauth, but I added simple htpasswd auth with a username and password and logged into the cluster
Installed quickly without drama
Menus did not update to reflect virtualization. Going to try installing this:
Finally got the request to refresh the GUI for the added Virtualization menus
Virtualization menus are now there
Let’s try uploading a Bootable Volume
Miraculously, that worked. Now let’s create a VM.
I’ve seen “ErrorUnschedulable” be a red herring before. Let’s see if that fixes itself after a minute.
Just for fun, lets create another VM from an Instance Type but with my Boot CD rather than a cloud init disk
Apparently, not a good idea
OK, let’s create a VM from and instance with all defaults. Failed.
Let’s create a new VM
Failure #∞
================================================
That’s it. I’m out. This was a great learning experience. Kubernetes is awesome for highly-available and scalable containers _not_ virtual machines. 🙂
Thank You