General concern If you want to move VMs to an another Storage Domain, you need to copy the template from it as well!
Remove a disk:
1# IF RHV does not use anymore disk those should appear empty in lsblk: 2lsblk -a 3sdf 8:80 0 4T 0 disk 4└─36001405893b456536be4d67a7f6716e3 253:38 0 4T 0 mpath 5sdg 8:96 0 4T 0 disk 6└─36001405893b456536be4d67a7f6716e3 253:38 0 4T 0 mpath 7sdh 8:112 0 4T 0 disk 8└─36001405893b456536be4d67a7f6716e3 253:38 0 4T 0 mpath 9sdi 8:128 0 0 disk 10└─360014052ab23b1cee074fe38059d7c94 253:39 0 100G 0 mpath 11sdj 8:144 0 0 disk 12└─360014052ab23b1cee074fe38059d7c94 253:39 0 100G 0 mpath 13sdk 8:160 0 0 disk 14└─360014052ab23b1cee074fe38059d7c94 253:39 0 100G 0 mpath 15 16# find all disks from LUN ID 17LUN_ID="360014054ce7e566a01d44c1a4758b092" 18list_disk=$(dmsetup deps -o devname ${LUN_ID}| cut -f 2 |cut -c 3- |tr -d "()" | tr " " "\n") 19echo ${list_disk} 20 21# Remove from multipath 22multipath -f "${LUN_ID}" 23 24# remove disk 25for i in ${list_disk}; do echo ${i}; blockdev --flushbufs /dev/${i}; echo 1 > /sys/block/${i}/device/delete; done 26 27# You can which disk link with which LUN on CEPH side 28ls -l /dev/disk/by-* NFS for OLVM/oVirt Since oVirt need a shared stockage, we can create a local NFS to bypass this point if no Storage bay.
install Pfsense VM Download from Netgate website (account requested)
Make network config
Important note: no need to prepare NetworkManager config, KVM will handle creation of the bridge. Also note that dns enable is set to disables the use of libvirts DHCP server (pfsense is taking over).
1cat > pfsense.xml << EOF 2<network> 3 <name>pfsense-router</name> 4 <uuid></uuid> 5 <forward mode='nat'> 6 </forward> 7 <bridge name='virbr1' stp='on' delay='0'/> 8 <dns enable='no'/> 9 <ip address='192.168.123.1' netmask='255.255.255.0'> 10 </ip> 11</network> 12EOF 13 14sudo virsh net-define pfsense.xml 15sudo virsh net-start pfsense-router 16sudo virsh net-autostart pfsense-router 17 18# Give qemu ACL 19echo "allow all" | sudo tee /etc/qemu-kvm/${USER}.conf 20echo "include /etc/qemu-kvm/${USER}.conf" | sudo tee --append /etc/qemu/bridge.conf 21sudo chown root:${USER} /etc/qemu-kvm/${USER}.conf 22sudo chmod 640 /etc/qemu-kvm/${USER}.conf 23 24# Check network 25nmcli con show --active 26sudo virsh net-list --all 27sudo virsh net-edit pfsense-router 28sudo virsh net-info pfsense-router 29sudo virsh net-dhcp-leases pfsense-router Create and Run Pfsense VM 1# Create pfsense vm 2virt-install \ 3--name pfsense --ram 2048 --vcpus 2 \ 4--disk $HOME/pfsense/disk0.qcow2,size=12,format=qcow2 \ 5--cdrom $HOME/pfsense/netgate-installer-amd64.iso \ 6--network bridge=virbr0,model=e1000 \ 7--network bridge=virbr1,model=e1000 \ 8--graphics vnc,listen=0.0.0.0 --noautoconsole \ 9--osinfo freebsd14.0 \ 10--autostart \ 11--debug 12 13virsh start pfsense Create OKD vm 1virt-install \ 2--name okd --ram 2048 --vcpus 2 \ 3--disk $HOME/okd-latest/disk0.qcow2,size=50,format=qcow2 \ 4--autostart \ 5--cdrom $HOME/okd-latest/rhcos-live.iso \ 6--network bridge=virbr0,model=e1000 \ 7--network bridge=virbr1,model=e1000 \ 8--graphics vnc,listen=0.0.0.0 --noautoconsole \ 9--osinfo detect=on,require=off \ 10--debug 1sudo virt-install -n master01 \ 2 --description "Master01 OKD Cluster" \ 3 --ram=8192 \ 4 --cdrom "$HOME/okd-latest/rhcos-live.iso" \ 5 --vcpus=2 \ 6 --disk pool=default,bus=virtio,size=10 \ 7 --graphics none \ 8 --osinfo detect=on,require=off \ 9 --serial pty \ 10 --console pty \ 11 --network network=openshift4,mac=52:54:00:36:14:e5 1sudo cp {{OKUB_INSTALL_PATH}}/rhcos-live.iso /var/lib/libvirt/images/rhcos-live-{{PRODUCT}}-{{RELEASE_VERSION}}.iso 2export COREOS_INSTALLER="podman run --privileged --pull always --rm -v /dev:/dev -v /var/lib/libvirt/images:/data -w /data quay.io/coreos/coreos-installer:release" 3sudo ${COREOS_INSTALLER} iso kargs modify -a "ip={{IP_MASTERS}}::{{GATEWAY}}:{{NETMASK}}:okub-sno:{{INTERFACE}}:none:{{DNS_SERVER}}" "rhcos-live-{{PRODUCT}}-{{RELEASE_VERSION}}.iso" 4sudo virt-install --name="openshift-sno" \ 5 --vcpus=4 \ 6 --ram=8192 \ 7 --disk path=/var/lib/libvirt/images/sno-{{PRODUCT}}-{{RELEASE_VERSION}}.qcow2,bus=sata,size=120 \ 8 --network network=sno,model=virtio \ 9 --boot menu=on \ 10 --graphics vnc --console pty,target_type=serial --noautoconsole \ 11 --cpu host-passthrough \ 12 --osinfo detect=on,require=off \ 13 --cdrom /var/lib/libvirt/images/rhcos-live-{{PRODUCT}}-{{RELEASE_VERSION}}.iso Checks Pfsense VM 1# Checks 2virsh list 3virsh domifaddr pfsense 4virsh domiflist pfsense 5 6# Connect to console 7virt-viewer --domain-name pfsense Delete Pfsense VM 1virsh destroy pfsense 2virsh undefine pfsense --remove-all-storage 3 4# disk can be deleted only manually 5rm -f ~/pfsense/disk0.qcow2 6 7# delete network 8sudo virsh net-destroy pfsense-router 9sudo virsh net-undefine pfsense-router 10sudo nmcli con del virbr1 11sudo nmcli con del eno1 Create a worker 1# Generate a MAC address 2date +%s | md5sum | head -c 6 | sed -e 's/\([0-9A-Fa-f]\{2\}\)/\1:/g' -e 's/\(.*\):$/\1/' | sed -e 's/^/52:54:00:/';echo 3 4sudo virt-install -n worker03.ocp4.example.com \ 5 --description "Worker03 Machine for Openshift 4 Cluster" \ 6 --ram=8192 \ 7 --vcpus=4 \ 8 --os-type=Linux \ 9 --os-variant=rhel8.0 \ 10 --noreboot \ 11 --disk pool=default,bus=virtio,size=50 \ 12 --graphics none \ 13 --serial pty \ 14 --console pty \ 15 --pxe \ 16 --network bridge=openshift4,mac=52:54:00:95:d4:ed
Prerequisites install KVM on RHEL 1# pre-checks hardware for intel CPU 2egrep -c '(vmx|svm)' /proc/cpuinfo 3lscpu | grep Virtualization 4lsmod | grep kvm 5 6# on RHEL9 Workstation 7sudo dnf install virt-install virt-viewer -y 8sudo dnf install -y libvirt 9sudo dnf install virt-manager -y 10sudo dnf install -y virt-top libguestfs-tools guestfs-tools 11sudo gpasswd -a $USER libvirt 12 13# Helper 14sudo dnf -y install bridge-utils 15 16# Start libvirt 17sudo systemctl start libvirtd 18sudo systemctl enable libvirtd 19sudo systemctl status libvirtd install KVM on Ubuntu 1sudo apt update && sudo apt upgrade -y 2sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients libvirt-daemon virtinst -y 3sudo usermod -aG libvirt $(whoami) 4sudo usermod -aG kvm $(whoami) 5 6# Helper 7sudo apt install bridge-utils cpu-checker -y 8 9# Start libvirt 10sudo systemctl start libvirtd 11sudo systemctl enable libvirtd 12sudo systemctl status libvirtd Bonus point: 1sudo apt install cockpit cockpit-machines -y 2sudo systemctl enable --now cockpit.socket 3systemctl status cockpit.socket Then manage your VMs from cockpit: https://localhost:9090 which could be an good alternative to virt-manager.
Basic Checks 1virsh nodeinfo Config a Bridge network Important note that network are created with root user but VM with current user.
Non permanent bridge: 1sudo ip link add virbr1 type bridge 2sudo ip link set eno1 up 3sudo ip link set eno1 master virbr1 4sudo ip address add dev virbr1 192.168.2.1/24 Permanent bridge 1sudo nmcli con add ifname virbr1 type bridge con-name virbr1 2sudo nmcli con add type bridge-slave ifname eno1 master virbr1 3sudo nmcli con modify virbr1 bridge.stp no 4sudo nmcli con down eno1 5sudo nmcli con up virbr1 6sudo ip address add dev virbr1 192.168.123.1/24 KVM - Bridge Network 1cat > hostbridge.xml << EOF 2<network> 3 <name>hostbridge</name> 4 <forward mode='bridge'/> 5 <bridge name='virbr1'/> 6</network> 7EOF 8 9sudo virsh net-define hostbridge.xml 10sudo virsh net-start hostbridge 11sudo virsh net-autostart hostbridge Give qemu ACL 1echo "allow all" | sudo tee /etc/qemu-kvm/${USER}.conf 2echo "include /etc/qemu-kvm/${USER}.conf" | sudo tee --append /etc/qemu/bridge.conf 3sudo chown root:${USER} /etc/qemu-kvm/${USER}.conf 4sudo chmod 640 /etc/qemu-kvm/${USER}.conf Check network 1sudo nmcli con show --active 2sudo virsh net-list --all 3sudo virsh net-edit hostbridge 4sudo virsh net-info hostbridge 5sudo virsh net-dhcp-leases hostbridge Check with a small script 1echo -e "\n##### KVM networks #####\n" 2kvm_system_networks_all=$(sudo virsh net-list --all) 3echo -e "Available KVM networks in qemu:///system :\n$kvm_system_networks_all" 4for net in $(sudo virsh net-list --name); do 5 bridge_name=$(sudo virsh net-info --network ${net} | grep Bridge | cut -d":" -f2 | sed 's/^[[:space:]]*//') 6 for br in ${bridge_name}; do 7 br_info=$(ip -br -c address show dev ${br} || echo "No IP address assigned to bridge ${br}") 8 done 9 echo -e "\n\033[1;34m${net}\033[0m have the Bridge: $br_info" 10done 11echo -e "\n" thanks to bridge-utils package installed ealier: 1brctl show Create a VM with this bridge 1virt-install \ 2--name pfsense --ram 2048 --vcpus 2 \ 3--disk $HOME/pfsense/disk0.qcow2,size=12,format=qcow2 \ 4--autostart \ 5--cdrom $HOME/pfsense/netgate-installer-amd64.iso \ 6--network bridge=virbr0,model=e1000 \ 7--network network=hostbridge,model=e1000 \ 8--graphics vnc,listen=0.0.0.0 --noautoconsole \ 9--osinfo freebsd14.0 \ 10--debug Delete network 1sudo virsh net-destroy hostbridge 2sudo virsh net-undefine hostbridge 3sudo nmcli con del virbr1 4sudo nmcli con del eno1 Sources Blog redhat
Hosted-engine Administration Connect to VM hosted-engine with root and password setup during the install: 1# Generate a backup 2engine-backup --scope=all --mode=backup --file=/root/backup --log=/root/backuplog 3 4# Restore from a backup on Fresh install 5engine-backup --mode=restore --file=file_name --log=log_file_name --provision-db --restore-permissions 6engine-setup 7 8# Restore a backup on existing install 9engine-cleanup 10engine-backup --mode=restore --file=file_name --log=log_file_name --restore-permissions 11engine-setup host Administration Connect in ssh to the Host: 1# Pass a host in maintenance mode manually 2hosted-engine --vm-status 3hosted-engine --set-maintenance --mode=global 4hosted-engine --vm-status 5 6# Remove maintenance mode 7hosted-engine --set-maintenance --mode=none 8hosted-engine --vm-status 9 10# upgrade hosted-engine 11hosted-engine --set-maintenance --mode=none 12hosted-engine --vm-status 13engine-upgrade-check 14dnf update ovirt\*setup\* # update the setup package 15engine-setup # launch it to update the engine /!\ Connect individually to KVM Virtmanager does not work OVirt use libvirt but not like KVM do…
Prerequisistes Check Compatibilty hardware: Oracle Linux Hardware Certification List (HCL)
A minimum of two (2) KVM hosts and no more than seven (7).
A fully-qualified domain name for your engine and host with forward and reverse lookup records set in the DNS.
/var/tmp 10 GB space at least
Prepared a shared-storage (nfs or iscsi) of at least 74 GB to be used as a data storage domain dedicated to the engine virtual machine. ISCSI need to be discovered before oVirt install.
