In this article, I want to show you how to recover a broken virtual machine, specifically how to increase the file system in case we cannot connect to the instance by ssh. In this article, we will learn how to make the necessary changes and mount a disk from a broken instance to the “rescuer” instance and make the necessary changes. Since version 7.2, Proxmox VE supports safe and easy reassignments of a VM disk via GUI, allowing us to make all the required changes without a ssh connection to the host. Please note: Although I have tested all of these instructions, any changes you make are at your own risk. Make sure that you have a backup before intervening in a “broken” virtual machine. Step 1: First of all, decide which instance you will use as a rescuer If you don’t have a free instance, you need to create one. After that, stop the broken VM and reassign its disk to the new owner. You can do that by going to “hardware”, choosing the needed disk, and selecting the Reassign Owner option: In the popup menu, choose the required VM instance, set the necessary disk parameters (usually default ones work perfectly), and press Reassign Disk. Step 2: Extend the disk size in the Proxmox UI To do that, select the required disk in the Hardware menu, and in the Disk Action, choose the Resize option: Step 3: Login to the rescue instance Inside the temporary instance we need to find the attached disk and extend its filesystem. It is not enough to extend a physical device; we need to extend its filesystem (and underlying LVM config if we have any) to make all the space available for the OS. The simplest method to find out the new disk is to check the last logs from the dmesg command output. You will see something like this: [  847.872511] sd 2:0:0:1: Power-on or device reset occurred
[  847.872524] sd 2:0:0:1: Attached scsi generic sg2 type 0
[  847.872800] sd 2:0:0:1: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[  847.873192] sd 2:0:0:1: [sdb] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
[  847.873238] sd 2:0:0:1: [sdb] Write Protect is off
[  847.873240] sd 2:0:0:1: [sdb] Mode Sense: 63 00 00 08
[  847.873342] sd 2:0:0:1: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  847.875058]  sdb: sdb1 sdb2
[  847.884885] sd 2:0:0:1: [sdb] Attached SCSI disk That means our new disk was added to the system as /dev/sdb. Step 4: Let's check what partitions we have on that disk and resize them To do that, make sure that you a superuser (all commands run from the superuser), and we will use the fdisk utility for that: fdisk -l /dev/sdb

Disk /dev/sdb: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0007e67f

Device     Boot   Start      End  Sectors Size Id Type
/dev/sdb1  *       2048  2099199  2097152   1G 83 Linux
/dev/sdb2       2099200 20971519 18872320   9G 8e Linux LVM Here we see that we need to extend the /dev/sdb2 device. To do that, we need to remove it and create a new one in its place but with an extended size: fdisk /dev/sdb - command prints the disk info, same as running fdisk -l /dev/sdb p p - command delete the last partition (in this case, /dev/sdb2) d d - command creates a new partition; makes that an extended partition n e n e - changes the type of partition t t Enter as the partition type, specify here type what that you have previously. 8e - writes the changes to disk and exits fdisk w w After that, the disk will look like this: Disk /dev/sdb: 21 GiB, 22548578304 bytes, 44040192 sectors
Disk model: QEMU HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0007e67f

Device     Boot   Start      End  Sectors Size Id Type
/dev/sdb1  *       2048  2099199  2097152   1G 83 Linux
/dev/sdb2       2099200 44040191 41940992  20G 8e Linux LVM Step 5: Now we need to extend our LVM physical volume (PV) Without any options, the size of the volume will be extended to all the available space: pvresize  /dev/sdb2 pvdisplay will show us, that PV Size updated up to 20GiB: --- Physical volume ---
  PV Name               /dev/sdb2
  VG Name               centos
  PV Size               <20.00 GiB / not usable 2.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              5119
  Free PE               2816
  Allocated PE          2303
  PV UUID               L1x3fi-tjdd-A0N0-6Iqg-NSNd-kLID-S6o5Hr After extending the physical volume, we need to extend the logical volume (LV). Let's check what we have at this time: lvdisplay 

  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  LV UUID                IpaS3d-3h27-oSWR-sOjr-QLqo-2zwv-zGJEqY
  LV Write Access        read/write
  LV Creation host, time localhost, 2023-03-01 22:16:28 +0000
  LV Status              available
  # open                 1
  LV Size                <8.00 GiB
  Current LE             2047
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2 We can see that the LV size is still 8 GiB. Let's extend it to all the available space: lvextend -l +100%FREE /dev/centos/root Step 6: We've completed LVM updating. Now, let's update the filesystem In my case, it's XFS. If you're using a different filesystem, please use the specific commands for that filesystem. Let’s mount it temporarily and extend it via xfs_growfs: mkdir  /tmp/rescue/
mount  /dev/centos/root /tmp/rescue/
xfs_growfs /dev/centos/root

 df -h
/dev/mapper/centos-root             19G  1.3G   18G   7% /tmp/rescue

umount /tmp/rescue/ Step 7: Now we have completed all the necessary actions for growing the partition. Let's return the disk to its place Detach the volume from the rescue VM. Only after that we will have the ability to reassign it to the new owner: In the rescued VM, if the disk was bootable, we need to check that the disk is enabled for boot and chosen for the first place: Conclusion In this article, we walked through the comprehensive process of recovering a broken virtual machine with low disk space in Proxmox VE. The steps included mounting the broken instance's disk to a rescuer instance, increasing the disk size through the Proxmox UI, and extending the file system using commands on the rescuer instance. Remember to exercise caution, back up your data, and adapt the commands according to your specific filesystem.

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Resize filesystem or Change Any File on Your Proxmox VM if Locked Out!

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

6 GitHub Repos for DevSecOps in 2024

The Noonification: The Sun Snarers (11/1/2022)

100 Days of Code: Death of Summer on the Island of NixOS

10+ Things I Love About Linux

10 Basic Tips on Working Fast in UNIX or Linux Terminal

The Noonification: Breaking Axioms in Program Execution (10/29/2023)

6 GitHub Repos for DevSecOps in 2024

The Noonification: The Sun Snarers (11/1/2022)

100 Days of Code: Death of Summer on the Island of NixOS

10+ Things I Love About Linux

10 Basic Tips on Working Fast in UNIX or Linux Terminal

The Noonification: Breaking Axioms in Program Execution (10/29/2023)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps