Re: filesystem performance with lots of files

On Fri, 2 Dec 2005, Qingqing Zhou wrote:

>>
>> I don't have all the numbers readily available (and I didn't do all the
>> tests on every filesystem), but I found that even with only 1000
>> files/directory ext3 had some problems, and if you enabled dir_hash some
>> functions would speed up, but writing lots of files would just collapse
>> (that was the 80 min run)
>>
>
> Interesting. I would suggest test small number but bigger file would be
> better if the target is for database performance comparison. By small
> number, I mean 10^2 - 10^3; By bigger, I mean file size from 8k to 1G
> (PostgreSQL data file is at most this size under normal installation).

I agree, that round of tests was done on my system at home, and was in
response to a friend who had rsync over a local lan take > 10 hours for
<10G of data. but even so it generated some interesting info. I need to
make a more controlled run at it though.

> Let's take TPCC as an example, if we get a TPCC database of 500 files,
> each one is at most 1G (PostgreSQL has this feature/limit in ordinary
> installation), then this will give us a 500G database, which is big enough
> for your current configuration.
>
> Regards,
> Qingqing
>