Add a larger disk and remove smaller disk from LVM, online!

Goal: To add new larger disk to LVM, move data and remove old smaller one

When using Oracle VM in my work, a disk can be enlarged in Oracle VM manager for a virtual machine. But because of an internal loop-mount on the XEN hypervisor level, the disk-file which is presented to the virtual machine as a disk, is not able to ‘pass on’ it’s resize information to the OS. This requires an unmount and mount of the disk-file on hypervisor level, or in simple words: a reboot of the virtual machine… yikes!

For this I have been starting to use LVM. To bad, because plain disk-files with just a file system on them makes them easy to handle in case of trouble…

Now when you *only* want to enlarge an LVM Logical Volume and the File System on it, go ahead, this can be done easily. Look at step 7 and 8 of this blog for the Logical Volume extend command for 100% space expansion.

But, if you want to *replace* an old disk with a new disk and you start to do the steps just like when you only want to add a new disk, you would think removing the old one afterwards should be possible because there is enough space… unfortunately not… LVM does not allow for this and prutsing can result in data-loss…

The trick is NOT to add the space from the new Physical Volume to the Logical Volume, but add the Physical Volume to the Volume Group only! Then the old Physical Volume ‘content’ is MOVED to the new Physical Volume. The Volume Group can still have space left from the new Physical Volume for either an other Physical Volume ‘content’ move or to extend the Logical Volume(s) to use all available space. The ‘content’ is the Logical Volume(s) information.

Steps (descriptive)

• Start: in the beginning…
• Step 0: add the new disk to the OS and partition it
  • adding here is the creation of a new disk on hyper-visor level
  • new OSes can detect the addition of new disks, try ‘fdisk -l’
  • partitioning is not really necessary, but a good practice
• Step 1: add the new disk/partition to LVM as a new Physical Volume
• Step 2: add the new Physical Volume to the Volume Group
• Step 3: move the old Physical Volume content to the new Physical Volume
  • the ‘content’ contains the ‘Logical Volume(s)’ on it
• Step 4: remove the old Physical Volume from the Volume Group
• Step 5: remove the old Physical Volume from LVM
• Step 6: if there is enough space on the new Physical Volume, repeat steps 3, 4 and 5 for another old Physical Volume
  • if you replaced a 100gb drive with a 250gb one, there is 150gb space left for an other 100gb drive
• Step 7: if there is space left in the Volume Group, extend the Logical Volume
  • if you replaced a 100gb drive with a 150gb one, there is 50gb space left
• Step 8: resize the file system on the Logical Volume
  • depending on the file system chosen this can also be online (ext3 and ext4 works fine)
• End: done!

Steps (the commands)

Step 0: add the new disk to the OS and partition it

Do your thing with ‘fdisk’ or ‘parted’…

Step 1: add the new disk/partition to LVM as a new Physical Volume

$ pvcreate /dev/sdd1

Step 2: add the new Physical Volume to the Volume Group

$ vgextend datavg /dev/sdd1

Step 3: move the old Physical Volume content to the new Physical Volume

$ pvmove /dev/sdc1 /dev/sdd1

Step 4: remove the old Physical Volume from the Volume Group

$ vgreduce datavg /dev/sdc1

Step 5: remove the old Physical Volume from LVM

$ pvremove /dev/sdc1

Step 6: if there is enough space on the new Physical Volume, repeat steps 3, 4 and 5 for another old Physical Volume

...move, reduce, remove, repeat...

Step 7: if there is space left in the Volume Group, extend the Logical Volume

$ lvdisplay
 --- Logical volume ---
 LV Path /dev/datavg/datalv
 LV Name datalv
 VG Name datavg
 
$ lvextend -l +100%FREE /dev/datavg/datalv

Step 8: resize the file system on the Logical Volume

$ resize2fs /dev/datavg/datalv
  or
$ resize2fs /dev/mapper/vgdata-lvdata

Steps (commands and images)

In these examples I have created one Logical Volume in the Volume Group, but there can be more Logical Volumes in a Volume Group. Nothing is done with Logical Volume groups until step 7.

Start: in the beginning…

Step 0: add the new disk to the OS and partition it

Step 1: add the new disk/partition to LVM as a new Physical Volume

$ pvcreate /dev/sdd1

Step 2: add the new Physical Volume to the Volume Group

$ vgextend datavg /dev/sdd1

Step 3: move the old Physical Volume ‘content’ to the new Physical Volume

Now the Logical Volume information ‘content’ of /dev/sdc1 is transferred to /dev/sdd1. When the old Physical Volume was smaller, the Logical Volume is not as big as the complete Volume Group (as illustrated).

$ pvmove /dev/sdc1 /dev/sdd1

Step 4: remove the old Physical Volume from the Volume Group

$ vgreduce datavg /dev/sdc1

Step 5: remove the old Physical Volume from LVM

$ pvremove /dev/sdc1

Step 6: if there is enough space on the new Physical Volume, repeat steps 3, 4 and 5 for another old Physical Volume

...move, reduce, remove, repeat...

Step 7: if there is space left in the Volume Group, extend the Logical Volume

$ lvdisplay
 --- Logical volume ---
 LV Path /dev/datavg/datalv
 LV Name datalv
 VG Name datavg
 
$ lvextend -l +100%FREE /dev/datavg/datalv

Step 8: resize the file system on the Logical Volume

$ resize2fs /dev/datavg/datalv
  or
$ resize2fs /dev/mapper/vgdata-lvdata

End: done!

The Physical Devices are now ‘detached’ from LVM and they can be removed from the virtual machine.

Happy logical-voluming!