It takes a while for the installation to settle, if ever. At this point (45 minutes in), 15 CPUs cores are being kept very busy on my host by the OpenShift VM.
Let’s see if the console will respond. Probably not as it has all its CPU cores pegged.
Maybe not yet…
nmap is showing that ports are open including the console port of 6443. Probably just too busy to respond
PORT STATE SERVICE
22/tcp open ssh
53/tcp open domain
80/tcp open http
111/tcp open rpcbind
443/tcp open https
9001/tcp open tor-orport
9099/tcp open unknown
9100/tcp open jetdirect
9103/tcp open jetdirect
6443/tcp open sun-sr-https
Installation is stuck at “Updated finalizing stage of the cluster to ‘Waiting for olm operators csv initialization'”. Sometimes a reboot helps, so I will reboot the VM after an hour of installing.
Let’s try hitting the cluster with kubectl while we wait for the console. From the Cloud Console, click Download kubeconfig
I saved mine in ~/.kube/nuc-kubeconfig
$ export KUBECONFIG=~/.kube/nuc-kubeconfig
$ kubectl get nodes
(Took a while, but responded)
NAME STATUS ROLES AGE VERSION
nuc Ready control-plane,master,worker 91m v1.28.10+a2c84a5
$ kubectl get pods -A
(Still settling…)
After 2 hours of installation, and a brief period where the VM was not crushing all 16 vCPUs, I was able to login to the console with the username kubeadmin and the password shown in the cloud console
In terms of the OpenShift Virtualization menu items, the OpenShift Virtualization Operator is still being installed. Once installed, I will be prompted to refresh the OpenShift console page and the Virtualization menu items will be added.
Most of the Operator installation messages are of this type probably due to CPU contention: “
ComponentUnhealthy: installing: waiting for deployment virt-operator to become ready: deployment “virt-operator” not available: Deployment does not have minimum availability.”You can stream the Operator install events from this page
Gave up – Failure #2
================================================
#3 Multi-Node vSphere Deployment – Failed on VM Creation
(Sort of defeats the purpose, but is one supported install option for OpenShift)
Unfortunate blogging event: Well, I just got back from traveling and switched from my travel Linux laptop to my work Mac laptop. Blogger decided to revert my blog to the version from 3 days ago on my Mac laptop. and I lost three days of testing and blogging. I will try to re-wite from memory without the screen shots and test I typed into Blogger. – Lesson learned: Don’t write in Blogger; write offline and paste into Blogger when done.
./openshift-install create cluster –dir . –log-level=info
? SSH Public Key /Users/faucherd/.ssh/id_rsa.pub
? Platform vsphere
? vCenter 192.168.1.190
? Username [email protected]
? Password [? for help] *********
INFO Connecting to vCenter 192.168.1.190
INFO Defaulting to only available datacenter: Datacenter
INFO Defaulting to only available cluster: /Datacenter/host/Cluster
INFO Defaulting to only available datastore: /Datacenter/datastore/datastore1
INFO Defaulting to only available network: VM Network
? Virtual IP Address for API 192.168.1.30
? Virtual IP Address for Ingress 192.168.1.31
? Base Domain fios-router.home
? Cluster Name openshift-vm
? Pull Secret [? for help] **************************************************************************************************************************************************************************************************************
INFO Creating infrastructure resources…
INFO Waiting up to 20m0s (until 8:08PM EDT) for the Kubernetes API at https://api.openshift-vm.fios-router.home:6443…
INFO API v1.28.10+a2c84a5 up
INFO Waiting up to 1h0m0s (until 8:53PM EDT) for bootstrapping to complete…
INFO Destroying the bootstrap resources…
INFO Waiting up to 40m0s (until 8:41PM EDT) for the cluster at https://api.openshift-vm.fios-router.home:6443 to initialize…
INFO Waiting up to 30m0s (until 8:50PM EDT) to ensure each cluster operator has finished progressing…
INFO All cluster operators have completed progressing
INFO Checking to see if there is a route at openshift-console/console…
INFO Install complete!
INFO To access the cluster as the system:admin user when using ‘oc’, run ‘export KUBECONFIG=/Users/faucherd/Downloads/openshift-install-mac-4.15.17/auth/kubeconfig’
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.openshift-vm.fios-router.home
INFO Login to the console with user: “kubeadmin”, and password: “WudX4-JDYei-MvHXU-HnmdZ”
INFO Time elapsed: 38m3s
export KUBECONFIG=/Users/faucherd/Downloads/openshift-install-mac-4.15.17/auth/kubeconfig
kubectl get nodes
NAME STATUS ROLES AGE VERSION
openshift-vm-qcbtl-master-0 Ready control-plane,master 149m v1.28
openshift-vm-qcbtl-master-1 Ready control-plane,master 149m v1.28
openshift-vm-qcbtl-master-2 Ready control-plane,master 149m v1.28
openshift-vm-qcbtl-worker-0-8mxlp Ready worker 133m v1.28
openshift-vm-qcbtl-worker-0-gq7zr Ready worker 133m v1.28
openshift-vm-qcbtl-worker-0-tbmrn Ready worker 133m v1.28
Was able to get the web GUI with this DNS
Check Installed Operators for Virtualization: No
Installed the OpenShift Virtualization operator from Operator Hub
Installs in namespace openshift-cnv
kubectl get pods -n openshift-cnv
NAME READY STATUS RESTARTS
aaq-operator-6c8d77b497-tqqf2 1/1 Running 0
cdi-operator-754fc75476-7zczc 1/1 Running 0
cluster-network-addons-operator-6fb577d5b5-pqkxg 2/2 Running 0
hco-operator-6c4ccffd46-vq74s 1/1 Running 0
hco-webhook-7b464898c5-cpfz4 1/1 Running 0
hostpath-provisioner-operator-b46544895-8t7mt 1/1 Running 0
hyperconverged-cluster-cli-download-5bff678dd4-hq2hs 1/1 Running 0
mtq-operator-666b8bdfd6-62flz 1/1 Running 0
ssp-operator-fb44777b-9p27r 1/1 Running 0
virt-operator-65f5d66c96-2kn74 1/1 Running 0
virt-operator-65f5d66c96-xx7v6 1/1 Running 0
No Virtualization in Workloads menu yet
Virtualization Operator: Create Hyperconverged (Take Defaults)
Creates a bunch more pods in openshift-cnv namespace
NUC is a bit oversubscribed – both CPU & RAM
Told me to refresh the console. When I did, “Virtualization” was added to the menu
Virtualization Operator: Create Hostpath Provisioner (Take Defaults)
Creates a bunch more pods in openshift-cnv namespace
Create VM
Upload bootable volume fedora-cloud-base-qcow2
“Allocating resources, please wait for upload to start.”
An error occurred
Data Volume failed to initiate upload.
Changed api. From 1.30 to 1.31 – still failed to upload
Try the CLI
virtctl image-upload dv fedora-volume-01 –size=10240Mi –image-path=/Users/faucherd/Downloads/Fedora-Cloud-Base-UEFI-UKI.x86_64-40-1.14.qcow2 –access-mode=ReadWriteOnce –force-bind
PVC default/fedora-volume-01 not found
DataVolume default/fedora-volume-01 created
Waiting for PVC fedora-volume-01 upload pod to be ready…
Pod now ready
Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.openshift-vm.fios-router.home
385.06 MiB / 385.06 MiB [==================================] 100.00% 42s
Uploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress
Processing completed successfully
Uploading /Users/faucherd/Downloads/Fedora-Cloud-Base-UEFI-UKI.x86_64-40-1.14.qcow2 completed successfully
Upload an ISO with the CLI
virtctl image-upload dv xubuntu-iso-01 –size=3072Mi –image-path=/Users/faucherd/Downloads/Xubuntu-24.04-minimal-amd64.iso –access-mode=ReadWriteOnce –force-bind
Created a new VM in the GUI with these volumes
VMs Stuck at “Provisioning” – Failure #3
================================================
#4 Single-Node Install as ESXi VM on 8 Core NUC – Failed on Not Enough Cores
Download the minimal ISO from RedHat Console: wget -O discovery_image_openshift-vm-sno.iso ‘https://api.openshift.com/api/assisted-images/bytoken/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MTg2NDQ2MjMsInN1YiI6ImZlMjNlNmI2LTE1ZmUtNDQxZS05YWJjLWFiZjU4NWEyZDdlNSJ9.voD8T6asWC0I-WPKdgXMTWvh5DY7Q6CpLU04telkgeA/4.15/x86_64/minimal.iso’
Boot 8 core NUC from Minimal ISO on Ventoy USB thinb drive
Let is settle. CPU and RAM activity are now at zero, RH console says: Failed
Warning alert:Insufficient
Minimum CPU cores for selected role: Require at least 15 CPU cores for master role, found only 8.
Failure #4
================================================
#5 Laptop Install as ESXi VM on 8 Core/24 GB Fedora VM – Failed Install of Virtualization Operator
Install a Fedora VM with 8 cores and 24 GB RAM
$ crc setup
$ crc start
[SNIP]
Started the OpenShift cluster.
The server is accessible via web console at:
https://console-openshift-console.apps-crc.testing
Log in as administrator:
Username: kubeadmin
Password: jiFr4-ckMXJ-yGGoY-vmMqJ
Log in as user:
Username: developer
Password: developer
Use the ‘oc’ command line interface:
$ eval $(crc oc-env)
$ oc login -u developer https://api.crc.testing:6443
Open console URL local to VM
kubeadmin login works fine in the local GUI.
OpenShift Virtualization Operator has not been installed. Try installing.
Keep an eye on the Operator install with
$ eval $(crc oc-env)
$ oc get pods -n openshift-cnv
NAME READY STATUS RESTARTS
aaq-operator-6985fcd6d6-6wxc2 0/1 Pending 0
cdi-operator-75d967b479-vg9rf 0/1 Pending 0
cluster-network-addons-operator-84488b7548-vhxhz 2/2 Running 0
hco-operator-c9964b489-pk7d6 1/1 Running 0
hco-webhook-b647d765d-p27ml 1/1 Running 0
hostpath-provisioner-operator-67d7b9768-ksslc 0/1 Pending 0
hyperconverged-cluster-cli-download-59f9f5867f-dt4w8 1/1 Running 0
mtq-operator-c499999fc-bpqmn 0/1 Pending 0
ssp-operator-555bb4c8bc-mvdbn 1/1 Running 0
virt-operator-6d558dd77f-s4xcl 0/1 Pending 0
virt-operator-6d558dd77f-vlxvc 0/1 Pending 0
When I look at the Operator event stream in the GUI, it looks like the Virtualizatipon Operator is trying and trying to install the deployment “virt-operator”. I’ll let this run over night and see what happens. Only 1.5 CPU cores are being used but almost all 24 GB or RAM in the Fedora VM are being consumed.
“installing: waiting for deployment virt-operator to become ready: deployment “virt-operator” not available: Deployment does not have minimum availability.”
Finally, the install of the Operator gave up
Operator failed
install failed: deployment virt-operator not ready before timeout: deployment “virt-operator” exceeded its progress deadline. Failure #5
Let’s try giving the VM more RAM
================================================
#6 Laptop Install as ESXi VM on 8 Core/32 GB Fedora VM – Failed Install of Virtualization Operator
$ crc config get memory
Configuration property ‘memory’ is not set. Default value ‘10752’ is used
$ crc config get cpus
Configuration property ‘cpus’ is not set. Default value ‘4’ is used
$ crc setup
$ crc start
[SNIP]
ERRO Cluster is not ready: cluster operators are still not stable after 10m0.784655588s Failure #6
INFO Adding crc-admin and crc-developer contexts to kubeconfig…
Started the OpenShift cluster.
The server is accessible via web console at:
https://console-openshift-console.apps-crc.testing
Log in as administrator:
Username: kubeadmin
Password: jiFr4-ckMXJ-yGGoY-vmMqJ
Log in as user:
Username: developer
Password: developer
Use the ‘oc’ command line interface:
$ eval $(crc oc-env)
$ oc login -u developer https://api.crc.testing:6443
Let’s define a higher limit for CPU and RAM
$ crc stop
$ crc config set memory 24576
$ crc config set cpus 8
$ crc start
[SNIP]
Started the OpenShift cluster.
Let’s try adding the Virtualization Operator again
Oh my. “Succeeded”
The GUI did not request a restart to add the VM menu items. I logged out and back in but still no VM menu items
Let’s stop and start the cluster and see what happens
$ crc stop
$ crc start
ERRO Cluster is not ready: cluster operators are still not stable after 10m0.784655588s Failure #7
Opened up the GUI. VM menus had not appeared yet. Checked Operators and the Virtualization Operator was trying to install again. “Deployment does not have minimum availability”. Giving up.
================================================
#7 Laptop Install on Windows VM – Failed Install of Virtualization Operator
Let’s try installing a Local install on a Windows 11 VM. Couldn’t hurt 🙂. Created a Windows 11 VM in Workstation Pro from the Tiny Win 11 ISO and gave it 8 vCPUs and 16 GB RAM.
Microsoft Windows [Version 10.0.22621.2283]
(c) Microsoft Corporation. All rights reserved.
C:UsersDennis Faucher>crc setup
CRC is constantly improving and we would like to know more about usage (more details at https://developers.redhat.com/article/tool-data-collection)
Your preference can be changed manually if desired using ‘crc config set consent-telemetry <yes/no>’
Would you like to contribute anonymous usage statistics? [y/N]: y
Thanks for helping us! You can disable telemetry with the command ‘crc config set consent-telemetry no’.
INFO Using bundle path C:UsersDennis Faucher.crccachecrc_hyperv_4.15.14_amd64.crcbundle
INFO Checking minimum RAM requirements
INFO Check if Podman binary exists in: C:UsersDennis Faucher.crcbinoc
INFO Checking if running in a shell with administrator rights
INFO Checking Windows release
INFO Checking Windows edition
INFO Checking if Hyper-V is installed and operational
INFO Checking if Hyper-V service is enabled
INFO Checking if crc-users group exists
INFO Checking if current user is in crc-users and Hyper-V admins group
INFO Checking if vsock is correctly configured
INFO Checking if CRC bundle is extracted in ‘$HOME/.crc’
INFO Checking if C:UsersDennis Faucher.crccachecrc_hyperv_4.15.14_amd64.crcbundle exists
INFO Getting bundle for the CRC executable
INFO Downloading bundle: C:UsersDennis Faucher.crccachecrc_hyperv_4.15.14_amd64.crcbundle…
4.69 GiB / 4.69 GiB [———-] 100.00% 111.42 KiB/s
INFO Uncompressing C:UsersDennis Faucher.crccachecrc_hyperv_4.15.14_amd64.crcbundle
crc.vhdx: 22.45 GiB / 22.45 GiB [———-] 100.00%
oc.exe: 120.19 MiB / 120.19 MiB [———-] 100.00%
INFO Checking if the win32 background launcher is installed
INFO Checking if the daemon task is installed
INFO Installing the daemon task
INFO Checking if the daemon task is running
INFO Running the daemon task
INFO Checking admin helper service is running
INFO Checking SSH port availability
Your system is correctly setup for using CRC. Use ‘crc start’ to start the instance
C:UsersDennis Faucher> crc start
INFO Using bundle path C:UsersDennis Faucher.crccachecrc_hyperv_4.15.14_amd64.crcbundle
INFO Checking minimum RAM requirements
INFO Check if Podman binary exists in: C:UsersDennis Faucher.crcbinoc
INFO Checking if running in a shell with administrator rights
INFO Checking Windows release
INFO Checking Windows edition
INFO Checking if Hyper-V is installed and operational
INFO Checking if Hyper-V service is enabled
INFO Checking if crc-users group exists
INFO Checking if current user is in crc-users and Hyper-V admins group
INFO Checking if vsock is correctly configured
INFO Checking if the win32 background launcher is installed
INFO Checking if the daemon task is installed
INFO Checking if the daemon task is running
INFO Checking admin helper service is running
INFO Checking SSH port availability
INFO Loading bundle: crc_hyperv_4.15.14_amd64…
CRC requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://console.redhat.com/openshift/create/local.
? Please enter the pull secret **************************************************************************************************************INFO Creating CRC VM for OpenShift 4.15.14…
INFO Generating new SSH key pair…
INFO Generating new password for the kubeadmin user
INFO Starting CRC VM for openshift 4.15.14…
Error starting machine: Error in driver during machine start: exit status 1
Turned Windows Defender Firewall Off and tried again
Same error. Might be lack of resources
======================
crc start –log-level debug
DEBU stderr: Hyper-VStart-VM : ‘crc’ failed to start.
Failed to start the virtual machine ‘crc’ because one of the Hyper-V components is not running.
‘crc’ failed to start. (Virtual machine ID 6782EE85-7D6E-430A-BB79-30B3893C5FF7)
The Virtual Machine Management Service failed to start the virtual machine ‘crc’ because one of the Hyper-V components is not running
(Virtual machine ID 6782EE85-7D6E-430A-BB79-30B3893C5FF7).
At line:1 char:43
+ $ProgressPreference = ‘SilentlyContinue’; Hyper-VStart-VM crc
+ ~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-VM], VirtualizationException
+ FullyQualifiedErrorId : OperationFailed,Microsoft.HyperV.PowerShell.Commands.StartVM
I opened up Windows Services and started every Hyper-V service that could be started
Second crc start –log-level debug
Same error
Just noticed that I did not give the Windows VM access to Intel virtualization. Oops. PEBCAK. Let’s fix that. and restart
crc start –log-level debug
Cluster started but crc start having trouble accessing
Cluster is progressing. Slowly. I noticed that CPU is pegged. I think I should increase the number of vCPUs from 8 to 16 and restart.
crc start is much happier. Making better progress. CPU & RAM are fine. Disk is _very_ busy.
Almost there “2 operators are progressing: console, ingress” 🤞
INFO 2 operators are progressing: console, ingress
ERRO Cluster is not ready: cluster operators are still not stable after 11m33.3766227s
INFO Adding crc-admin and crc-developer contexts to kubeconfig…
DEBU Using address: api.crc.testing:6443
DEBU Dialing to 127.0.0.1:6443
ERRO Cannot update kubeconfig: read tcp 127.0.0.1:51321->127.0.0.1:6443: wsarecv: An existing connection was forcibly closed by the remote host.
Started the OpenShift cluster.
The server is accessible via web console at:
https://console-openshift-console.apps-crc.testing
Log in as administrator:
Username: kubeadmin
Password: 7ZiMh-6iPa7-EiaUI-D89Er
Log in as user:
Username: developer
Password: developer
Use the ‘oc’ command line interface:
PS> & crc oc-env | Invoke-Expression
PS> oc login -u developer https://api.crc.testing:6443
