Using LVM to Serve Old Laptop as a Cloud Storage Engine Service at home with all External Hard Drive

Using LVM to Serve Old Laptop as a Cloud Storage Engine Service at home with all External Hard Drives.

I have a very old experimental Toshiba laptop with a 640GB (596 GB usable) HDD and an optical drive. And lately, my brother was curious in converting the laptop storage into a Storage Engine for all of us at home. This will allow us to share files easily among us.

We installed OS in a 120 GB SSD and moved this HDD to a HDD case. We have another external Seagate HDD with 1 TB. And we wanted to have a single storage partition where we could save files without any thoughts of storing at partition 1 or partition 2.

We were able to achieve a whooping 1.6TB partition using LVM. We can even extend it dynamically to more size with more Hard Disks attached. This post is about how we achieve that. By the end of the post, it will make sure that you will able to achieve similar results. :) At the end, I will point to some open source softwares which we used to host it as a storage service at home.

Please note that system administration understanding is extremely important. If you are not an experienced System Engineer, please do not try this.

Illustrative figure, on what we will be achieving.

Before I jump into the implementation, would like to put in and share some basic disks concept.

LVM — Logical Volume Management.

It is a storage device management technology which gives us the power to pool and abstract the physical layout of the component individual devices for easier and flexible administration.We can use to accumulate multiple storage system into a single groups and get a combined space. It is much more flexible than the traditional systems. Volumes can be resized dynamically.

Physical Volumes
These are regular storage devices. They provide space to store the logical volumes. They are used by the LVM as raw building material as a higher level of abstraction.Logical Volumes
It corresponds to partitions holding a file system. They can span across multiple disks, and do not have to be physically contiguous. They are the primary components that users and applications interact with.Volume Groups
It is a named collection of physical and logical volumes. They abstract the characteristics of the underlying devices and functions as a unified logical device with combined storage capacity.So, now we understand the basics. Lets implement this.

We need to have LVM Physical Device. So, assuming that the disks partitions are deleted and raw disks can be seen when you fire, fdisk -l . We will follow the below steps to create scalable volumes.

Create LVM Physical Volume

➜  ~ sudo lvmdiskscan
  /dev/loop0 [      88.48 MiB]
  /dev/sda   [     596.17 GiB]
  /dev/loop1 [     <97.42 MiB]
  /dev/sdb2  [    <111.79 GiB]
  /dev/sdc   [     931.51 GiB]
  2 disks
  3 partitions
  0 LVM physical volume whole disks
  0 LVM physical volumes

The output tells that there are 2 disks available. We will create their physical volumes.

➜  ~ sudo pvcreate /dev/sda /dev/sdc
  Wiping dos signature on /dev/sda.
  Wiping dos signature on /dev/sdc.
  Physical volume "/dev/sda" successfully created.
  Physical volume "/dev/sdc" successfully created.
➜  ~ sudo pvs    # To list the LVM physical Volumes
  PV         VG Fmt  Attr PSize   PFree
  /dev/sda      lvm2 ---  596.17g 596.17g
  /dev/sdc      lvm2 ---  931.51g 931.51g

Create a LVM Volume Group

Once the physical volumes are created, they can grouped using vgcreateas shown below:

➜  ~ sudo vgcreate VgHulkStorage /dev/sda /dev/sdc
  Volume group "VgHulkStorage" successfully created
➜  ~ sudo vgs    # To list the Volume Groups created
  VG            #PV #LV #SN Attr   VSize VFree
  VgHulkStorage   2   0   0 wz--n- 1.49t 1.49t

Create a LVM Logical Volume Partition

Finally the logical volume can be created as below:

➜  ~ sudo lvcreate -L 1.49t -n LvHulkStorage VgHulkStorage<br>  Rounding up size to full physical extent 1.49 TiB<br>  Logical volume "LvHulkStorage" created.

-L is to specify the size of the Logical Volume. Multiple Logical Volumes with smaller size can be created.-n is to specify the name of the logical volumelast parameter is for Volume Group Name from where logical volume will be createdYou can query sudo lvsto see the logical volume partitions

Its time create a file system and mount it.

We are now ready to create a file system in the logical volume and mount it for use.

➜  ~ sudo mkfs.ext4 /dev/VgHulkStorage/LvHulkStorage
mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 399969280 4k blocks and 99999744 inodes
Filesystem UUID: 13022223-d447-4a7c-9fb6-cd870f53f207
Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
 102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done
➜  ~ sudo mount /dev/VgHulkStorage/LvHulkStorage /mnt/hulkstorage

Extending an existing volume with a new external hard disk space

In future, if more disks are to be added, then the logical volume can be extended in similar way. A quick set of commands are here:

➜  ~ sudo pvcreate /dev/sdd
  Wiping dos signature on /dev/sdd.
  Physical volume "/dev/sdd" successfully created.
➜  ~ sudo vgextend VgHulkStorage /dev/sdd
  Volume group "VgHulkStorage" successfully extended
➜  ~ sudo lvextend -L +467.61g /dev/VgHulkStorage/LvHulkStorage
  Rounding size to boundary between physical extents: 467.61 GiB.
  Size of logical volume VgHulkStorage/LvHulkStorage changed from 1.49 TiB (390595 extents) to <1.95 TiB (510304 extents).
  Logical volume VgHulkStorage/LvHulkStorage successfully resized.
➜  ~ sudo resize2fs /dev/VgHulkStorage/LvHulkStorage
resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/VgHulkStorage/LvHulkStorage is mounted on /mnt/hulkstorage; on-line resizing required
old_desc_blocks = 191, new_desc_blocks = 250
The filesystem on /dev/VgHulkStorage/LvHulkStorage is now 522551296 (4k) blocks long.
➜  ~ sudo vgs
  VG            #PV #LV #SN Attr   VSize  VFree
  VgHulkStorage   3   1   0 wz--n- <1.95t 4.00m

Pretty Simple. Huh?

So, how does it really look like?

➜  ~ df -h --output=target,size
Mounted on        Size
/dev              2.9G
/run              595M
/                 110G
/dev/shm          3.0G
/run/lock         5.0M
/sys/fs/cgroup    3.0G
/mnt/hulkstorage  2.0T
/run/user/1000    595M

My hulkstorageis whooping 2TB and I do not have to really bother about pushing files to any specific harddisk anymore. These things can be done from from a folder inside

Host this partition over network so that you can mount that in you laptops/development zones.a self hosted storage engine can be served to share files across all people at home. :)

Storage Engine

Hosting this local storage for home is pretty simple, actually. I started Droppyinside with docker. Running it is fairly simple.

➜  ~ docker run -d --restart always --name storage_engine -p 8989:8989 -e UID=1000 -e GID=1000 -v /mnt/hulkstorage/StorageEngine:/files silverwind/droppy

After the server starts, the local server is accessible at port 8989.

References:LVM — UbuntuAn Introductions to LVM Concepts

This keeps motivating to contribute more. Feel free to follow me, here.