I was working on a genome assembly project when I was caught off guard by an issue: the assembly process was terminated due to insufficient space on the hard disk. I was assembling the linked reads using Universal Sequencing Technology's TELLYSIS pipeline. After consulting with the company, I was informed that I needed at least 1024 GB of free space before starting the assembly.
I am using an IBM server installed with Ubuntu distro, version 22.04. This server is a decade old but has recently undergone an upgrade, increasing its memory capacity to an impressive 248 GB.
This ten-year-old machine is equipped with eight SAS hard disks, each with a capacity of 300 GB. This theoretically scales up to 8x300 = 2400 GB or 2.4 TB of storage capacity. However, there are slots for 12 physical disks, of which four slots cannot accommodate additional disks due to a lack of adequate SAS connectivity on the motherboard. A quick and technically less demanding solution I considered was to replace one of the existing 300 GB SAS disks with a higher-capacity one.
Upon further investigation and communication with local procurement services, I discovered that the SAS disks compatible with my machine are quite rare in the market, and the available ones are rather expensive.
As mentioned earlier, there are eight SAS disks, each with a 300 GB capacity, so the total storage capacity should amount to 2400 GB. However, the system's usable storage capacity is only 1800 GB. Later, it became clear that two sets of four hard disks had been configured in a RAID 5 setup.
When four hard disks are RAID 5 configured, a pooled storage capacity equivalent to three hard disks is available for use. In the event of disk failure, the fourth disk acts as a backup. One set of four hard disks is configured as Drive 0, and the next set of four disks is similarly configured as Drive 1. This means that when you write a file to a disk, it is not immediately clear which one of the four disks it is on, or in whose clusters the file was registered.
Drive 0 has two partitions: /dev/sda1 and /dev/sda2. Similarly, Drive 1 has two partitions: /dev/sdb1 and /dev/sdb2. Both partitions are created with an ext4 filesystem. While /dev/sda2 is a /boot/ partition, /dev/sdb2 is a general partition.
The capacity of /dev/sda2 is 835 GB, and that of /dev/sdb2 is 870 GB.
In each of these partitions, there is approximately 750 GB of free space, which is less than the free space required for running the genome assembly. To meet the requirement, I need to install an additional hard disk with a storage capacity of more than 1024 GB. Since these partitions belong to two distinct drives, resizing or merging the partitions is not as straightforward as it is in the Windows OS.
This is when I came across LVM (Logical Volume Management) technology. Using LVM, multiple physical disks can be merged to create a logical volume partition with the pooled storage capacity of the disks involved.
LVM stands for Large Volume Management. With this technology, physical volumes can be created using desired partitions, a volume group can be formed from these physical volumes, and finally, a logical volume of the desired size can be created from the volume group. The storage capacity of the logical volume partition is accounted for by multiple physical disks.
However, there is a risk involved. If any of the physical disks that are part of the volume group fail, the entire volume group fails, rendering the data irrecoverable. So, caution should be exercised, and data backup is essential.
However I have a backup, and I want to proceed with creating an LVM partition on the existing physical disks in my system.
Here's a simplified protocol for the process:
1. Boot your system from an Ubuntu bootable USB drive.
2. Delete the partitions you want to use for creating Physical Volumes (in this case, /dev/sda2 and /dev/sdb2).
3. Create new primary partitions and convert them into "LVM type."
4. Create a Physical Volume using these partitions.
5. Create a volume group involving /dev/sda2 and /dev/sdb2.
6. Create a logical volume partition of the combined desired size based on the storage capacity of /dev/sda2 and /dev/sdb2 within the volume group.
7. Create an ext4 filesystem for the newly created logical volume.
8. Mount the new logical volume partition.
I have yet to implement these steps to create the LVM partition, and I have concerns about the success rate given the age of the SAS disks in the system.
What do you think? How about I install Linux on a Virtual Machine using VirtualBox? Can I reduce the risk of data loss in case the logical volume somehow gets damaged? Share your thoughts!
Comments