This paper introduces research results of I/O performance for
modern filesystems configured on multi-disk storage configuration in GNU/Linux
operating system. Filesystem performance tests include basic file operations
statistics depending on local disks management software RAID or logical
volume manager LVM. The results allow to choose the appropriate filesystem
to data storage space configuration in GNU/Linux, that provides bests
performance

Smoliński, Mateusz

Lodz University of Technology Repository

JOURNAL OF APPLIEDCOMPUTER SCIENCEVol. 22 No. 2 (2014), pp. 65-80Filesystems Performance in GNU/LinuxMulti-Disk Data StorageMateusz Smolin´ski11Lodz University of TechnologyFaculty of Technical Physics, Information Technology and AppliedMathematics/Institute of Information Technologyul. Wólczan´ska 215, 90-924 Lodz, Polandmateusz.smolinski@p.lodz.plAbstract. This paper introduces research results of I/O performance formodern filesystems configured on multi-disk storage configuration in GNU/Linuxoperating system. Filesystem performance tests include basic file operationsstatistics depending on local disks management software RAID or logicalvolume manager LVM. The results allow to choose the appropriate filesys-tem to data storage space configuration in GNU/Linux, that provides bestsperformance.Keywords: storage space, filesystem performance, multi-disk configuration.1. IntroductionOperating systems offer efficient data storage and processing. For security andperformance reasons data storage space is often divided to zones, each one in-cludes block device configuration and filesystem. Many storage zones offers alsoflexibility, because each zone can be configured according to stored data character-istic. I/O performance for single zone grow, when in its configuration many disksare utilized with striping mechanism. It offers storage space of many disks as onelogical block device (volume). Single I/O operation on striped logical volume can66 Filesystems Performance in GNU/Linux Multi-Disk Data Storagebe performed in parallel by many disks. Stripping has two main parameters: num-ber of disks and disk allocation unit (chunk).GNU/Linux kernel in version 2.6 and newer offers various striping implemen-tation like Device Mapper or software software RAID (Redundant Array of Inde-pendent Disk) [1,2,3]. Device Mapper implementation is used by LVM (LogicalVolume Manager), which offers flexible and persistent logical volume configura-tion. Configuration of logical volume base on earlier volume group configuration,which uses physical volumes. Stripping allocation policy for logical volume re-quires many physical volumes assigned to volume group and each physical volumeshould be localized on separate disk. Except striping LVM software offers otherallocations policy for each managed logical volume. In any LVM configurationlogical volume size is logical extent multiplicity. In executed storage performancetests was used default logical extend size 4MB. Software RAID implementation inLinux kernel offers many disk array levels, each has specific characteristic to datadistribution between managed disks. Stripping is used in software RAID level 0(RAID0).Data storage zone has only one filesystem of fixed type. Filesystems type dif-fers in logical and physical layer. Logical layer is responsible for file organizationlike namespace, files attributes. Filesystem physical layer corresponds to data andmetadata distribution on block devices. Metadata are additional data in filesystem,that provide organization of data saved in stored files.Kernel of GNU/Linux supports many filesystem types. For research purposesthree modern and popular filesystems types were chosen: XFS, EXT4 and BTRFS[1, 2, 3]. Configuration of each filesystem include default settings except size ofallocation unit, which was fixed to 4KB. For each storage performance testing sce-nario its filesystem is mounted in read write asynchronous mode in GNU/Linuxoperating system.2. Storage test environmentAll storage space configuration scenarios were configured in the same envi-ronment using identical program tool bonnie++. Storage performance test wasobtained in GNU/Linux operating system from Fedora 18 distribution for x86_64computer architecture (kernel version 3.6.10). Computer hardware used for stor-age testing include: i7-2600 CPU, 16GB memory, six identical SATA-2 WesternM. Smolin´ski 67Table 1. The parameters of tested multi-disk storage configuration scenarioFilesystem type Disk manager Disks No. Chunk sizeBTRFS,EXT4,XFS LVM,RAID0 2,3,4,5 16KB,64KB,256KB,1MBDigital disks model WD500AAKX with 465GB capacity and 16MB cache. Op-erating system files were installed on the first of the six disks, other were used instorage configuration scenarios.Software used for storage configuration: btrfs-progs in version 0.20, mdadmin version 3.26, xfs-progs in version 3.1.8, e2fsprogs in version 1.42.5, bonnie++in version 1.96. Single program run includes one bonnie++ process creation forsequential data read and write in block to 32GB regular file and performing op-eration on 102400 thousands empty regular files distributed in 1000 directories.File operations statistic for single scenario include: number of sequentially andrandomly created files, number of files which attributes are read (stat) and numberof deleted files. In any storage configuration scenario executed operations utilizeboth regular files data and filesystem metadata.Storage performance tests takes into account basic file operations for zone’sstorage configuration scenario, which includes multi-disk striping configurationand filesystem type that manage storage space of logical volume. Any storage con-figuration scenario has also fixed number of disks and chunk size.3. Filesystems charasteristicPerformance analysis of results obtained in tested scenarios provides charac-teristic of each filesystem in multi-disk storage configurations.3.1. BTRFSDesign of BTRFS bases on b-tree structure with write on copy (COW) rule up-date method, which assures its consistency without journaling. All BTRFS meta-data are organized as b-tree structures and storage space allocation is dynamic us-ing chunks during filesystem operation. Additionally in B-tree structures can alsostore some saved in file data fragments. BTRFS uses 64bit addressing, so it canstore files up to 16EB [2].68 Filesystems Performance in GNU/Linux Multi-Disk Data StorageB-tree filesystem has internal block device manager, that can manage storagespace of many disks (even various HDD and SSD). BTRFS offers also logical sub-volumes, that can store files separately or using clones and snapshoting. It alsooffers on-line resizing and defragmentation, which are performed in background.In every subvolume BTRFS differentiates chunks with data and metadata anduses for them predefined allocation policy using striping or mirroring. Checksum-ming is very important reliability feature in BTRFS, which stores checksums forany data and metadata chunks. Any I/O operation in filesystem need calculate thechecksum and compare its value with stored checksum pattern. Checksum mis-match causes log entry and BTRFS automatically tries to find valid data blockstored in other mirrored chunk. If no mirrored chunk is present, BTRFS return aerror that could be served by task that orders I/O operation.For storage performance testing purposes many scenarios with BTRFS wereprepared, in which BTRFS has not directly managed disks. Its internal block de-vice manager has only one logical volume, in each tested BTRFS only one subvol-ume is created. As all tested filesystem BTRFS has identical 4KB block size andalso it uses the same size for b-tree node and leaf.3.2. EXT4Fourth version of popular journaled EXT filesystem introduces many improve-ments with backward compatibility. As earlier versions EXT4 use resource groupsthat limit file fragmentation. Used in EXT4 48bit addressing supports filesystemstorage up to 1EB with file sizes up to 16TB. It doubles the limit in EXT3 formaximum files in single directory. Implemented in EXT4 extent feature providescontinuous block allocation, additionally it supports also persistent pre-allocationblock device space for file. New extended filesystem versions has improvement inreliability, because EXT4 implements journal checksumming [4].For storage performance testing EXT4 was created on single logical volumeand its default configuration was used with specified 4KB block size. In everystorage configuration scenario with EXT4, its journal is stored on the same logicalvolume. In filesystem creation process no explicit optimization for striping wereused. All metadata structures are prepared statically when filesystem is created.EXT4 has limit of stored files, that is predefined in creation time.M. Smolin´ski 693.3. XFSXFS is popular journaled filesystem ported from IRIX to GNU/Linux. It wasdeveloped as filesystem with 64bit addressing. It can support logical volumes upto 16EB and stored file size limited by 8EB. XFS is transactional and uses log-ical journal. Other XFS features include delayed and sequential block allocationwith various size extent. It limits the file fragmentation using separate allocationresource groups. Part of its internal metadata base also on b-tree structures. XFScan be online resized and defragmented [5].Storage performance testing scenarios has XFS configured with 4KB block onsingle logical volume. As in any other storage configuration scenario no explicitoptimization was used for striping between disks.4. Filesystem operations performanceFilesystem structures type have impact on performance of basic file operation.I/O performance for single disk managed by BTRFS, EXT4, XFS has various dataread and write speed. Figure 1 presents data read speed form file localised in eachtested filesystem when it directly manages single disk, I/O performance differencesbetween XFS and other tested filesystems are up to 14 percent. The performanceresults measured for each tested filesystem localized on single disk are referencepoint for performance comparision with any other multi-disk filesystem storageconfiguration.4.1. Filesystem storage with striped logical volumeAll tested filesystems localized in logical volume, which data are striped onmany disks have better I/O performance like in single disk storage configurationscenario (fig. 1). However increasing number of disks that belong to striped logicalvolume managed by LVM with chunk 64KB does not always increase results infile operation statistics. For example XFS characteristic unlike other tested filesys-tems shows best performance of files creation in 2 disks scenario (fig. 2.)In storage scenario with LVM striping on two disks XFS has lowest sequentiolfiles attributes reading statistics (fig. 3). In any other tested storage configurationbase on LVM striping it has the best performance results in reading files attributes.BTRFS and EXT4 statistics in sequential or random files attribute reading are in-dependent from striped disk number.70 Filesystems Performance in GNU/Linux Multi-Disk Data StorageFigure 1. I/O performance for tested filesystems, striping on disks managed byLVM with chunk 64KB: (a) data read speed, (b) data write speed.Figure 2. Files creation statistics for tested filesystems, striping on disks managedby LVM with chunk 64KB: (a) sequential files operations, (b) random files opera-tions.M. Smolin´ski 71Figure 3. Files attributes reading statistics for tested filesystems, striping on disksmanaged by LVM with chunk 64KB: (a) sequential files operations, (b) randomfiles operations.Figure 4. Files deletion statistics for tested filesystems, striping on disks managedby LVM with chunk 64KB: (a) sequential files operations, (b) random files opera-tions.72 Filesystems Performance in GNU/Linux Multi-Disk Data StoragePerformance of file deletion also significantly varied between tested filesys-tems, especially for random file deletion between BTRFS and EXT4. This is resultof using b-tree metadata structures [5]. Additionally XFS statistics show that it haseight times higher performance in random file deletion in storage scenario withstriping on two or many disks managed by LVM than in scenario with single disk(fig. 4).Also chunk size used in managed by LVM disks striping has impact on filesoperations performance in tested filesystems placed in logical volume, especiallyfor data read from file localized in EXT4 or XFS filesystem. File operation statis-tistics of each tested filesystem in dependence on chunk size in five disks stripingmanaged by LVM were presented on figures 5 - 8.Figure 5. I/O performance for tested filesystems according to chunk size in LVMstriping on 5 disks: (a) data read speed, (b) data write speed.Research results show that BTRFS has the smallest differences relative to thechunk size in basic file operation performance, even thought it did not always havethe best efficiency as XFS or EXT4 (fig. 6, 8). The XFS has the biggest differencesrelative to the chunk size in file creation and deletion. In tested storage scenariothe best sequential and random file operation performance in XFS for multi-diskconfiguration was achieved in LVM striping on 5 disks with 256KB chunk size.File operation statistics for EXT4 like BTRFS is not high dependable on chunkM. Smolin´ski 73Figure 6. Files creation statistics for tested filesystems according to chunk size inLVM striping on 5 disks: (a) sequential files operations, (b) random files opera-tions.Figure 7. Files attributes reading statistics for tested filesystems according to chunksize in LVM striping on 5 disks: (a) sequential files operations, (b) random filesoperations.74 Filesystems Performance in GNU/Linux Multi-Disk Data StorageFigure 8. Files deletion statistics for tested filesystems according to chunk size inLVM striping on 5 disks: (a) sequential files operations, (b) random files opera-tions.size in LVM disk striping. The EXT4 filesystem has the best performance in se-quential and random file creation and sequential file deletion in multi-disk stripingmanaged by LVM (fig. 6, 8).4.2. Filesystem storage with software disk array stripingFilesystem performance testing tool shows that I/O performance in multi-diskstorage configuration managed by software disk array with level 0 has similar asLVM I/O performance if used chunk size is identical (fig. 1, 9).If striping on disks is managed by software RAID then basic file operationperformance characteristic for BTRFS and EXT4 is similar to adequate storageconfiguration scenario when striping on disks is managed by LVM. However XFSlocalised on software RAID level 0 shows less performance in sequential file cre-ation and sequential file deletion operations. In RAID level 0 storage space con-figuration with two disks it is up to 169 percent less files created (fig. 2, 10) and99 percent less deleted files then in two disk striping managed by logical volumemanager. (fig. 4, 10).M. Smolin´ski 75Figure 9. I/O performance for tested filesystems, striping on disks managed bysoftware RAID with level 0 and chunk 64KB: (a) data read speed, (b) data writespeed.Figure 10. Files creation statistics for tested filesystems, striping on disks managedby software RAID with level 0 and chunk 64KB:(a) sequential files operations, (b)random files operations.76 Filesystems Performance in GNU/Linux Multi-Disk Data StorageFigure 11. Files attributes reading statistics for tested filesystems, striping on disksmanaged by software RAID with level 0 and chunk 64KB: (a) sequential filesoperations, (b) random files operations.Figure 12. Files deletion statistics for tested filesystems, striping on disks managedby software RAID with level 0 and chunk 64KB: (a) sequential files operations,(b) random files operations.M. Smolin´ski 77Software disk array striping configuration also has chunk size parameter. Anal-ysis of measured performance results for each tested filesystem localized in stripedstorage space managed by software array show that files operations statistic forBTRFS and EXT4 according to chunk size is similar to striping managed by LVM(fig. 13 - 16). However XFS has lowest basic file operation statistics when stripingis managed by software disk array. Differences in sequential or random file cre-ation and deletion were up to 75 percent worse than in identical multi-disk storageconfiguration managed by LVM (fig. 6, 8, 14, 16).Figure 13. I/O performance for tested filesystems according to chunk size in soft-ware RAID level 0 with 5 disks: (a) data read speed, (b) data write speed.Taking into account research results statement only BTRFS from tested filesys-tems provides high efficiency in all base files operations performed either in se-quential and random manner. File operations performance in BTRFS are also leastdependent on chunk size in filesystem multi-disk storage speace configurationwhether striping is managed by software disk array or logical volume managerimplemented in GNU/Linux operating system.78 Filesystems Performance in GNU/Linux Multi-Disk Data StorageFigure 14. Files creation statistics for tested filesystems according to chunk sizein software RAID level 0 with 5 disks: (a) sequential files operations, (b) randomfiles operations.Figure 15. Files attributes reading statistics for tested filesystems according tochunk size in software RAID level 0 with 5 disks: (a) sequential files operations,(b) random files operations.M. Smolin´ski 79Figure 16. Files deletion statistics for tested filesystems according to chunk sizein software RAID level 0 with 5 disks: (a) sequential files operations, (b) randomfiles operations.5. ConclusionsConfiguration of storage zones in modern GNU/Linux operating system iscrucial for I/O performance also in multi-disk configurations. Present versionsof Linux kernel include various implementation of filesystem type and stripingmechanism. In storage zone configuration fitting logical volume with striping onmultiple disks and filesystem type has impact on basic file operation performance.Analysis of research results shows that increasing number of disk in storagespace configuration provides better I/O performance but not always guaranteesimprovement of file operation performance (i.e. increasing disk number managedby LVM or software disk array where XFS is placed).Performance of file operations in XFS unlike other tested filesystems strongdepends on chunk size in striping managed by LVM or software disk array. Addi-tionally performance of files creation and deletion in XFS significantly decreaseswhen striping is managed by software disk array.The EXT4 filesystem unlike XFS and BTRFS does not include b-tree struc-tures, therefore EXT4 in each tested storage configuration scenario random filedeletion has the worst performance results. In other storage configurations EXT480 Filesystems Performance in GNU/Linux Multi-Disk Data Storagefilesystem is efficient, its performance characteristic (includes both file data readand write and other basic file operations) increases when more disks are used andis not depend to chunk size in striping managed by LVM or software disk array.Modern BTRFS is usually not efficient as EXT4 in basic file operations but it hassimilar file operation characteristic. In multi-disk storage BTRFS provides leastdiverse of performance results in all statements for tested storage configurationscenarios.References[1] Sobell, M. G., Fedora and RedHat Enterprise Linux, Prentice Hall, 6th ed.,2012, (in English).[2] Bacik, J., Btrfs, The Swiss Army Knife of Storage, USENIX, Vol. 37, 2012.[3] Rodeh, O. and Bacik, J., The Linux B-tree filesystem, Tech. rep., IBM, 2012.[4] Avantika, M., MingMing, C., and Suparna, B., The new ext4 filesystem: currentstatus and future plans, In: Linux Symposium, 2007, (in English).[5] C., H., XFS the big storage file system for Linux, USENIX, Vol. 34, No. 2,2009, (in English).

Filesystems performance in GNU/Linux multi-disc data storage

Abstract

Similar works

Full text

Available Versions

Lodz University of Technology Repository