ext2 vs ext3 vs reiserfs vs reiserfs4
These comparisons were performed on a Pentium IV system, Linux 2.6 (Fedora Core 2) with a single 75G U320 10K SCSI drive (Maxtor)
Write-intensive operations used this
perl script
Read-intensive operations used this
perl script
These scripts require two directories simulating Maildirs. I used a 77,133 entry directory for write_src and a 1,177 entry directory
for read_src (entries for both dirs contained actual email).
Write-intensive Results
ext3 (-J size=256 -T news)
[service@imap2 exp2]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000
maildir setup in 103 s
8018 writes and 1982 reads in 946 s
real 17m31.489s
user 6m42.673s
sys 4m56.383s
Indexed directory (ext3) (-J size=256 -T news -O dir_index)
Remarkable results on unpacking 77K files in a single directory (28s vs. 6m25s for non-indexed ext3 above)
But slower overall
[service@imap2 exp1]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000
maildir setup in 177 s
8039 writes and 1961 reads in 1441 s
real 26m59.727s
user 6m30.147s
sys 13m19.201s
Reiserfs
[service@imap2 exp1]# time tar xzf /home/maildir_bench_data.tgz
real 2m5.876s
user 0m5.826s
sys 0m11.272s
[service@imap2 exp1]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000
maildir setup in 202 s
7962 writes and 2038 reads in 1286 s
real 24m51.175s
user 6m10.742s
sys 7m14.441s
Reiserfs4
able to unpack in about 30s (comparable to ext3 with dir_index)
[service@imap2 exp1]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000
maildir setup in 64 s
7916 writes and 2084 reads in 1532 s
real 26m37.694s
user 6m12.405s
sys 9m21.773s
ext2
[service@imap2 exp2]# time tar xzf /home/maildir_bench_data.tgz
real 3m55.649s
user 0m6.270s
sys 3m47.994s
[service@imap2 exp2]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000
maildir setup in 128 s
8060 writes and 1940 reads in 613 s
real 12m23.002s
user 6m9.274s
sys 0m58.432s
Read-intensive Results
Reiser4
[root@imap2 exp1]# /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000
writes and reads in 74 s
ext2
[root@imap2 exp2]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000
writes and reads in 136 s
real 2m20.376s
user 0m1.313s
sys 0m14.290s
ext3 vanilla comparable to ext2
[root@imap2 ext3_vanilla]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000
writes and reads in 140 s
real 4m24.904s
user 0m2.454s
sys 0m36.649s
ext3 dir_index
[root@imap2 ext3_indix]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000
writes and reads in 126 s
real 4m31.866s
user 0m2.072s
sys 0m10.200s
Summary
ext2 is the fastest on write-intesive operations 613s compared to Reiserfs (1286s), then ext3 dir_index (1441s), then Reiserfs4 (1532s). Not too surprising that ext2
wins because it is not journalled and therefore has less overhead. What is surprising is that Reiserfs4 (touted as THE fastest filesystem)
is the slowest under these conditions, even slower than the previous reiserfs release.
Reiserfs wins on read-intensive operations and surprisingly, ext2 is comparable to ext3 - even with indexed directories (dir_index).
ext2 vs xfs vs reiserfs on RAID5
Same system as described above except on RAID5 (3 x 75G, zero-channel Adaptec 2010s)
Write-intensive operations used the same script as above except write operations did not flush after every line write (no printflush).
Read-intensive operations used the same script as above.
Write-intensive Results
xfs (-b size=1024 -d su=16k,sw=3 -l logdev=/dev/sda1 [SATA])
[root@imap2 xfs]# time tar xzf /home/maildir_bench_data.tgz
real 0m44.620s
user 0m5.867s
sys 0m12.957s
[root@imap2 xfs]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000
maildir setup in 84 s
8042 writes and 1958 reads in 698 s
real 13m3.200s
user 6m6.697s
sys 3m41.503s
reiserfs (journal on /dev/sda8 [SATA])
[root@imap2 reiserfs]# time tar xzf /home/maildir_bench_data.tgz
real 2m12.850s
user 0m5.753s
sys 0m8.986s
[root@imap2 reiserfs]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000
maildir setup in 73 s
7957 writes and 2043 reads in 808 s
real 14m42.589s
user 5m45.439s
sys 4m33.096s
ext2
[root@imap2 ext2]# time tar xzf /home/maildir_bench_data.tgz
real 11m33.945s
user 0m6.670s
sys 10m11.300s
[root@imap2 ext2]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000
maildir setup in 0 s
8052 writes and 1948 reads in 520 s
real 8m40.220s
user 5m44.019s
sys 0m55.938s
Read-intensive Results
xfs
[root@imap2 xfs]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000
maildir setup in 0 s
writes and reads in 44 s
real 0m44.417s
user 0m0.969s
sys 0m0.703s
reiserfs
[root@imap2 reiserfs]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000
maildir setup in 0 s
writes and reads in 47 s
real 0m46.991s
user 0m0.963s
sys 0m0.469s
ext2
[root@imap2 ext2]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000
maildir setup in 1 s
writes and reads in 49 s
real 0m50.150s
user 0m1.003s
sys 0m0.339s
Summary
ext2 is still the fastest on write-intesive operations 520s, then xfs at 698s and reiserfs was the slowest at 808s.
xfs is faster all-around than reiserfs and I haven't heard of any reliability issues with xfs like I have with reiserfs and qmail (meta-data related).
Conclusion
In conclusion, I'm (pleasantly) surprised to discover xfs as the winner overall. Though not quite as fast as ext2 on these particular write-intensive tests,
it is much more manageable than ext2 with more features including dynamic inode allocations (ideal for Maildir/news servers).
Final note on ext3
Why did I give up so easily on ext3? Because it is choking my current production server and prompted this whole thing! I'm using the ext3 as implemented on
Linux 2.4.20 and kjournald is hogging the machine. ext3 did well on write-intensive operations (w/o dir_index) but doesn't seem to scale well
and I would not recommend it for Maildir/qmail.