Using LVM to Serve Old Laptop as a Cloud Storage Engine Service at home with all External Hard Drive

Using LVM to Serve Old Laptop as a Cloud Storage Engine Service at home with all External Hard Drives. I have a very old experimental Toshiba laptop with a 640GB (596 GB usable) HDD and an optical drive. And lately, my brother was curious in converting the laptop storage into a Storage Engine for all of us at home. This will allow us to share files easily among us. We installed OS in a 120 GB SSD and moved this HDD to a HDD case. We have another external Seagate HDD with 1 TB. And we wanted to have a single storage partition where we could save files without any thoughts of storing at partition 1 or partition 2. We were able to achieve a whooping 1.6TB partition using LVM. We can even extend it dynamically to more size with more Hard Disks attached. This post is about how we achieve that. By the end of the post, it will make sure that you will able to achieve similar results. :) At the end, I will point to some open source softwares which we used to host it as a storage service at home. Please note that system administration understanding is extremely important. If you are not an experienced System Engineer, please do not try this. Illustrative figure, on what we will be achieving. Before I jump into the implementation, would like to put in and share some basic disks concept. LVM — Logical Volume Management. It is a storage device management technology which gives us the power to pool and abstract the physical layout of the component individual devices for easier and flexible administration.We can use to accumulate multiple storage system into a single groups and get a combined space. It is much more flexible than the traditional systems. Volumes can be resized dynamically. These are regular storage devices. They provide space to store the logical volumes. They are used by the LVM as raw building material as a higher level of abstraction. It corresponds to partitions holding a file system. They can span across multiple disks, and do not have to be physically contiguous. They are the primary components that users and applications interact with. It is a named collection of physical and logical volumes. They abstract the characteristics of the underlying devices and functions as a unified logical device with combined storage capacity.So, now we understand the basics. Lets implement this. Physical Volumes Logical Volumes Volume Groups We need to have LVM Physical Device. So, assuming that the disks partitions are deleted and raw disks can be seen when you fire, fdisk -l . We will follow the below steps to create scalable volumes. Create LVM Physical Volume ➜ ~ sudo lvmdiskscan /dev/loop0 [ 88.48 MiB] /dev/sda [ 596.17 GiB] /dev/loop1 [ <97.42 MiB] /dev/sdb2 [ <111.79 GiB] /dev/sdc [ 931.51 GiB] 2 disks 3 partitions 0 LVM physical volume whole disks 0 LVM physical volumes The output tells that there are 2 disks available. We will create their physical volumes. ➜ ~ sudo pvcreate /dev/sda /dev/sdc Wiping dos signature on /dev/sda. Wiping dos signature on /dev/sdc. Physical volume successfully created. Physical volume successfully created. ➜ ~ sudo pvs PV VG Fmt Attr PSize PFree /dev/sda lvm2 --- 596.17g 596.17g /dev/sdc lvm2 --- 931.51g 931.51g "/dev/sda" "/dev/sdc" # To list the LVM physical Volumes Create a LVM Volume Group Once the physical volumes are created, they can grouped using vgcreateas shown below: ➜ ~ sudo vgcreate VgHulkStorage /dev/sda /dev/sdc Volume group successfully created ➜ ~ sudo vgs VG VgHulkStorage 2 0 0 wz--n- 1.49t 1.49t "VgHulkStorage" # To list the Volume Groups created #PV #LV #SN Attr VSize VFree Create a LVM Logical Volume Partition Finally the logical volume can be created as below: ➜ ~ sudo lvcreate -L t -n LvHulkStorage VgHulkStorage Rounding size full physical extent TiB Logical volume created. 1.49 up to 1.49 "LvHulkStorage" -L is to specify the size of the Logical Volume. Multiple Logical Volumes with smaller size can be created.-n is to specify the name of the logical volumelast parameter is for Volume Group Name from where logical volume will be createdYou can query sudo lvsto see the logical volume partitions Its time create a file system and mount it. We are now ready to create a file system in the logical volume and mount it for use. ➜ ~ sudo mkfs.ext4 /dev/VgHulkStorage/LvHulkStorage mke2fs 1.44.1 (24-Mar-2018) Creating filesystem with 399969280 4k blocks and 99999744 inodes Filesystem UUID: 13022223-d447-4a7c-9fb6-cd870f53f207 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000, 214990848 Allocating group tables: Writing inode tables: Creating journal (262144 blocks): Writing superblocks and filesystem accounting information: ➜ ~ sudo mount /dev/VgHulkStorage/LvHulkStorage /mnt/hulkstorage done done done done Extending an existing volume with a new external hard disk space In future, if more disks are to be added, then the logical volume can be extended in similar way. A quick set of commands are here: ➜ ~ sudo pvcreate /dev/sdd Wiping dos signature on /dev/sdd. Physical volume successfully created. ➜ ~ sudo vgextend VgHulkStorage /dev/sdd Volume group successfully extended ➜ ~ sudo lvextend -L +467.61g /dev/VgHulkStorage/LvHulkStorage Rounding size to boundary between physical extents: 467.61 GiB. Size of logical volume VgHulkStorage/LvHulkStorage changed from 1.49 TiB (390595 extents) to <1.95 TiB (510304 extents). Logical volume VgHulkStorage/LvHulkStorage successfully resized. ➜ ~ sudo resize2fs /dev/VgHulkStorage/LvHulkStorage resize2fs 1.44.1 (24-Mar-2018) Filesystem at /dev/VgHulkStorage/LvHulkStorage is mounted on /mnt/hulkstorage; on-line resizing required old_desc_blocks = 191, new_desc_blocks = 250 The filesystem on /dev/VgHulkStorage/LvHulkStorage is now 522551296 (4k) blocks long. ➜ ~ sudo vgs VG VgHulkStorage 3 1 0 wz--n- <1.95t 4.00m "/dev/sdd" "VgHulkStorage" #PV #LV #SN Attr VSize VFree Pretty Simple. Huh? So, how does it really look like? ➜ ~ df -h --output=target,size Mounted on Size /dev 2.9G /run 595M / 110G /dev/shm 3.0G /run/lock 5.0M /sys/fs/cgroup 3.0G /mnt/hulkstorage 2.0T /run/user/1000 595M My hulkstorageis whooping 2TB and I do not have to really bother about pushing files to any specific harddisk anymore. These things can be done from from a folder inside Host this partition over network so that you can mount that in you laptops/development zones.a self hosted storage engine can be served to share files across all people at home. :) Storage Engine Hosting this local storage for home is pretty simple, actually. I started inside with docker. Running it is fairly simple. Droppy ➜ ~ docker -d --restart always --name storage_engine -p 8989:8989 -e =1000 -e =1000 -v /mnt/hulkstorage/StorageEngine:/files silverwind/droppy run UID GID After the server starts, the local server is accessible at port 8989. References: LVM — Ubuntu An Introductions to LVM Concepts This keeps motivating to contribute more. Feel free to follow me, . here