Compare filesystems for the purpose of an email/IMAP server using Maildir storage format (qmail)

xfs wins for Maildirs!

ext2 vs ext3 vs reiserfs vs reiserfs4

These comparisons were performed on a Pentium IV system, Linux 2.6 (Fedora Core 2) with a single 75G U320 10K SCSI drive (Maxtor)
Write-intensive operations used this perl script
Read-intensive operations used this perl script
These scripts require two directories simulating Maildirs. I used a 77,133 entry directory for write_src and a 1,177 entry directory for read_src (entries for both dirs contained actual email).

Write-intensive Results

ext3 (-J size=256 -T news)

[service@imap2 exp2]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000

maildir setup in 103 s
8018 writes and 1982 reads in 946 s

real    17m31.489s
user    6m42.673s
sys     4m56.383s

Indexed directory (ext3) (-J size=256 -T news -O dir_index)

Remarkable results on unpacking 77K files in a single directory (28s vs. 6m25s for non-indexed ext3 above)
But slower overall

[service@imap2 exp1]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000

maildir setup in 177 s
8039 writes and 1961 reads in 1441 s

real    26m59.727s
user    6m30.147s
sys     13m19.201s

Reiserfs

[service@imap2 exp1]# time tar xzf /home/maildir_bench_data.tgz

real    2m5.876s
user    0m5.826s
sys     0m11.272s

[service@imap2 exp1]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000

maildir setup in 202 s
7962 writes and 2038 reads in 1286 s

real    24m51.175s
user    6m10.742s
sys     7m14.441s

Reiserfs4

able to unpack in about 30s (comparable to ext3 with dir_index)

[service@imap2 exp1]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000

maildir setup in 64 s
7916 writes and 2084 reads in 1532 s

real    26m37.694s
user    6m12.405s
sys     9m21.773s

ext2

[service@imap2 exp2]# time tar xzf /home/maildir_bench_data.tgz

real    3m55.649s
user    0m6.270s
sys     3m47.994s

[service@imap2 exp2]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000

maildir setup in 128 s
8060 writes and 1940 reads in 613 s

real    12m23.002s
user    6m9.274s
sys     0m58.432s

Read-intensive Results

Reiser4

[root@imap2 exp1]# /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000

 writes and  reads in 74 s

ext2

[root@imap2 exp2]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000

 writes and  reads in 136 s

 real   2m20.376s
 user   0m1.313s
 sys    0m14.290s

ext3 vanilla comparable to ext2

[root@imap2 ext3_vanilla]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000

 writes and  reads in 140 s

 real   4m24.904s
 user   0m2.454s
 sys    0m36.649s

ext3 dir_index

[root@imap2 ext3_indix]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000

 writes and  reads in 126 s

 real   4m31.866s
 user   0m2.072s
 sys    0m10.200s

Summary

ext2 is the fastest on write-intesive operations 613s compared to Reiserfs (1286s), then ext3 dir_index (1441s), then Reiserfs4 (1532s). Not too surprising that ext2 wins because it is not journalled and therefore has less overhead. What is surprising is that Reiserfs4 (touted as THE fastest filesystem) is the slowest under these conditions, even slower than the previous reiserfs release.
Reiserfs wins on read-intensive operations and surprisingly, ext2 is comparable to ext3 - even with indexed directories (dir_index).

ext2 vs xfs vs reiserfs on RAID5

Same system as described above except on RAID5 (3 x 75G, zero-channel Adaptec 2010s)
Write-intensive operations used the same script as above except write operations did not flush after every line write (no printflush).
Read-intensive operations used the same script as above.

Write-intensive Results

xfs (-b size=1024 -d su=16k,sw=3 -l logdev=/dev/sda1 [SATA])

[root@imap2 xfs]# time tar xzf /home/maildir_bench_data.tgz

real    0m44.620s
user    0m5.867s
sys     0m12.957s

[root@imap2 xfs]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000

maildir setup in 84 s
8042 writes and 1958 reads in 698 s

real    13m3.200s
user    6m6.697s
sys     3m41.503s

reiserfs (journal on /dev/sda8 [SATA])

[root@imap2 reiserfs]# time tar xzf /home/maildir_bench_data.tgz

real    2m12.850s
user    0m5.753s
sys     0m8.986s

[root@imap2 reiserfs]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10100
I/O tests
10000 of 10000

maildir setup in 73 s
7957 writes and 2043 reads in 808 s

real    14m42.589s
user    5m45.439s
sys     4m33.096s

ext2

[root@imap2 ext2]# time tar xzf /home/maildir_bench_data.tgz

real    11m33.945s
user    0m6.670s
sys     10m11.300s

[root@imap2 ext2]# time /usr/local/site/bin/maildir_io_benchmark.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000

maildir setup in 0 s
8052 writes and 1948 reads in 520 s

real    8m40.220s
user    5m44.019s
sys     0m55.938s

Read-intensive Results

xfs

[root@imap2 xfs]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000

maildir setup in 0 s
 writes and  reads in 44 s

real    0m44.417s
user    0m0.969s
sys     0m0.703s

reiserfs

[root@imap2 reiserfs]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000

maildir setup in 0 s
 writes and  reads in 47 s

real    0m46.991s
user    0m0.963s
sys     0m0.469s

ext2

[root@imap2 ext2]# time /usr/local/site/bin/maildir_io_benchmark_random_read.pl .
basedir setup
maildir setup
pcu ->10000
I/O tests
10000 of 10000

maildir setup in 1 s
 writes and  reads in 49 s

real    0m50.150s
user    0m1.003s
sys     0m0.339s

Summary

ext2 is still the fastest on write-intesive operations 520s, then xfs at 698s and reiserfs was the slowest at 808s. xfs is faster all-around than reiserfs and I haven't heard of any reliability issues with xfs like I have with reiserfs and qmail (meta-data related).

Conclusion

In conclusion, I'm (pleasantly) surprised to discover xfs as the winner overall. Though not quite as fast as ext2 on these particular write-intensive tests, it is much more manageable than ext2 with more features including dynamic inode allocations (ideal for Maildir/news servers).

Final note on ext3

Why did I give up so easily on ext3? Because it is choking my current production server and prompted this whole thing! I'm using the ext3 as implemented on Linux 2.4.20 and kjournald is hogging the machine. ext3 did well on write-intensive operations (w/o dir_index) but doesn't seem to scale well and I would not recommend it for Maildir/qmail.