LVM expansion and recovery of deleted LV or VG

Posted on Edited on In , , Symbols count in article: 5.9k Reading time ≈ 5 mins.

Working forward in the snow @by haven200

Two new hard disks were added to NAS, and one LV was deleted due to an operation error when expanding the capacity of LVs. I took some time to find it back. So this article is a memo.

The difference between LVM2 and RAID

Simply RAID is a method to store your data safely whereas LVM is a software which can extend the performance of a RAID.

RAID

RAID is a combination of multiple disks to form a single module which can store data on one or more disk to improve redundancy and also to reduce the risk of data loss. RAID can be achieved either using physical device or a software.

LVM2

LVM is a method of logically partitioning your memory device over multiple disks in order to have a single or multiple partition over multiple disks. LVM is purely a software which manages multiple disks. LVM can be made to work over RAID for achieving improved DATA speeds.

Recover LV/VG which deleted

vgcfgrestore is a tool of restore VG for LVM.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
:~$ vgcfgrestore --help
  vgcfgrestore - Restore volume group configuration

  Restore VG metadata from last backup.
  vgcfgrestore VG
        [ COMMON_OPTIONS ]

  Restore VG metadata from specified file.
  vgcfgrestore -f|--file String VG
        [ COMMON_OPTIONS ]

  List all VG metadata backups.
  vgcfgrestore -l|--list VG
        [ COMMON_OPTIONS ]

  List one VG metadata backup file.
  vgcfgrestore -l|--list -f|--file String
        [ COMMON_OPTIONS ]
        [ VG ]

  Common options for command:
        [ -M|--metadatatype lvm2 ]
        [    --force ]

Backgroud

The lvm information is as follows:

1
2
3
4
5
6
7
8
9
10
:~$ lvs
  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  movies   media  -wi-a-----  <2.64t
  haven200 shares -wi-ao---- 400.00g
  public   shares -wi------- 200.00g

:~$ vgs
  VG     #PV #LV #SN Attr   VSize     VFree
  media    2   1   0 wz--n-    <2.64t    77g
  shares   1   2   0 wz--n- <1024.00g <424.00g

The recovery steps of lvremove and vgremove are the same, so let’s take LV as an example

When operating on LVM, lvremove was executed by mistake and LV[public] was deleted, as follows:

1
2
3
4
5
6
7
8
9
10
11
:~$ lvs
  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  movies   media  -wi-a-----  <2.64t
  haven200 shares -wi-ao---- 400.00g
  public   shares -wi------- 200.00g

:~$ lvremove /dev/shares/haven200 shares
:~$ lvs
  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  movies   media  -wi-a-----  <2.64t
  haven200 shares -wi-ao---- 400.00g

Recover LV

View the operation history of VG[shares] where LV[public] is belong to

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
:~$ vgcfgrestore --list shares
 File:         /etc/lvm/archive/shares_00000-367217791.vg/shares_00000-367217791.vg
  VG name:      shares
  Description:  Created *before* executing 'lvcreate -L 400G -n haven200 shares'
  Backup Time:  Thu Jan  5 08:40:25 2023

  File:         /etc/lvm/archive/shares_00001-1927234942.vg/shares_00001-1927234942.vg
  VG name:      shares
  Description:  Created *before* executing 'lvcreate -L 200G -n public shares'
  Backup Time:  Thu Jan  5 08:40:52 2023

  File:         /etc/lvm/archive/shares_00003-1430114073.vg/shares_00003-1430114073.vg
  VG name:      shares
  Description:  Created *before* executing 'lvremove /dev/shares/public'
  Backup Time:  Sun Jan 15 15:32:54 2023

  File:         /etc/lvm/backup/shares/shares
  VG name:      shares
  Description:  Created *after* executing 'lvremove /dev/shares/public'
  Backup Time:  Sun Jan 15 15:32:54 2023

Look at this paragraph in the information above

1
2
3
4
File:         /etc/lvm/archive/shares_00003-1430114073.vg/shares_00003-1430114073.vg
VG name:      shares
Description:  Created *before* executing 'lvremove /dev/shares/public'
Backup Time:  Sun Jan 15 15:32:54 2023

In the Description line, there is a command to delete LV[public], also there is a keyword before to indicate that the file shares_ 00003-1430114073.vg is the state of LV[public] before deleting, so we need to recover LVM to this step.

1
2
3
4
5
6
7
8
9
10
11
:~$ vgcfgrestore -f /etc/lvm/archive/shares_00003-1430114073.vg shares
  Volume group shares has active volume: haven200.
  WARNING: Found 1 active volume(s) in volume group "shares".
  Restoring VG with active LVs, may cause mismatch with its metadata.
Do you really want to proceed with restore of volume group "shares", while 1 volume(s) are active? [y/n]: y
  Restored volume group shares.

:~$ lvscan
  ACTIVE            '/dev/media/movies' [<2.64 TiB] inherit
  ACTIVE            '/dev/shares/haven200' [400.00 GiB] inherit
  inactive          '/dev/shares/public' [200.00 GiB] inherit

Now, the deleted LV[public] has been recovered, but its status is inactive, so we need to activate it manually.

1
2
3
4
5
6
7
8
9
10
11
12
:~$ lvchange -ay /dev/shares/public
:~$ lvscan
  ACTIVE            '/dev/media/movies' [<2.64 TiB] inherit
  ACTIVE            '/dev/shares/haven200' [400.00 GiB] inherit
  ACTIVE            '/dev/shares/public' [200.00 GiB] inherit

:~$ mount /dev/shares/public /mnt/public
:~$ ll /mnt/public
drwxr-xr-x  2 nas      nas      4.0K Jun 18  2019 pure-ftpd/
drwxr-xr-x  2 root     root     4.0K Jun 19  2019 rsyncd/
drwxr-xr-x  2 nas      nas      4.0K Feb 22  2020 server/
drwxr-xr-x  2 nas      nas      4.0K Jun 18  2019 transmission-daemon/

Here, the deleted LV [public] is recovered and can be mounted and used normally.

Expand LV/VG

  • Expand VG：Each PV can only join one VG, so when there is not enough VG space, you can only add PV for this VG.
  • Expand LV: as follows
    • VG has free space: simply expand this LV.
    • VG has no free space: Reduce the space of other LVs on this VG and add this space to the LV that needs to be expanded.

LV[movies] ran out of space, so a new hard drive was added. The following is the expansion record.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# format new disk
:~$ gdisk /dev/sdk
...
...
Command (m for help): n
Partition number (1-?): 1                                           
First cylinder (1-xxxx, default 1):                                     
Last cylinder, +cylinders or +size{K,M,G} (1-xxxxx, default xxxxx): 
Command (m for help): t                                             
Selected partition 1
Hex code (type L to list codes): lvm                        
                                                                        
Changed system type of partition 1 to 43 (Linux LVM)
...
...
# create pv
:~$ pvcreate /dev/sdk1
# add pv to vg
:~$ vgextend media /dev/sdk1
:~$ vgs
  VG     #PV #LV #SN Attr   VSize     VFree
  media    2   1   0 wz--n-    <4.64t    3.77g
  shares   1   2   0 wz--n- <1024.00g <424.00g

VG[media] has been expanded.
Next, we will start to expand the capacity of LV[movies].

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
:~$ lvs
  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  movies   media  -wi-a-----  <2.64t
  haven200 shares -wi-ao---- 400.00g
  public   shares -wi------- 200.00g

# umount LV fist.
:~$ umount /dev/media/movies
# check the LV
:~$ e2fsck -f /dev/media/movies
....
....
# add another space to lv
:~$ lvresize -L +2T /dev/media/movies
....
....
:~$ e2fsck -f /dev/media/movies
....
....
#  enlarge file system located on device
:~$ resize2fs /dev/media/movies
....
....
:~$ lvs
  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  movies   media  -wi-a-----  <4.64t
  haven200 shares -wi-ao---- 400.00g
  public   shares -wi------- 200.00g

References: