Обсуждение: PostgreSQL Cache

On Mon, 29 Sep 2008, Matthew Pulis wrote:

> I need to perform some timed testing, thus need to make sure that disk cache
> does not affect me. Is clearing the OS (Ubuntu) disk cache, ( by running:
> sudo echo 3 | sudo tee /proc/sys/vm/drop_caches ) enough to do this?

What you should do is:

1) Shutdown the database server (pg_ctl, sudo service postgresql stop,
etc.)
2) sync
3) sudo echo 3 > /proc/sys/vm/drop_caches
4) Start the database server

That will clear both the database and OS cache with a minimum of junk left
behind in the process; clearing the cache without a sync is a bad idea.

Note that all of this will still leave behind whatever cache is in your
disk controller card or on the disk themselves available.  There are some
other techniques you could consider.  Add a setp 2.5 that generates a
bunch of data unused by the test, then sync again, and you've turned most
of that into useless caching.

Ideally, your test should be running against a large enough data set that
the dozens or couple of hundred megabytes that might be in those will only
add a bit of noise to whatever you're testing.  If you're not running a
larger test or going through tasts to make the caches clear, the only easy
way to make things more clear is to reboot the whole server.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Matthew Pulis wrote:
> Hi,
>
> I need to perform some timed testing, thus need to make sure that disk
> cache does not affect me. Is clearing the OS (Ubuntu) disk cache, ( by
> running:  sudo echo 3 | sudo tee /proc/sys/vm/drop_caches ) enough to
> do this? If not can you please point me to some site please since all
> I am finding is such command.

Look for methodologies for doing performance tests. A problem is that
the disk cache is an essential part that makes up for postgresql
performance. Also do not forget about overhead and inaccuracies that you
will affect your results.

In general performance tests are a rather large simulation of how your
application would use the database. It should be large enough for many
effects (such as initial cache state) to be neglected. It only provides
an average for the performance on your system configuration.
If you run it a few times more, you can compute the variation. It
provides some insight how stable your system is in handling the workload.

- Joris

A while ago I wrote a script based on Dave Plonka work
http://net.doit.wisc.edu/~plonka/fincore/

My script monitors system buffers and shared buffers
(if pg_buffercache installed) and I found it's almost useless to
check system buffers, since I got rather ridiculous numbers.

> I use it to investigate OS cacheing of PostgreSQL files and was
> surprized on 24 Gb server, total cache was about 30 Gb. How this is
> possible ?

I can send script and perl module if you want to play with.



Oleg

On Mon, 29 Sep 2008, Greg Smith wrote:

> On Mon, 29 Sep 2008, Matthew Pulis wrote:
>
>> I need to perform some timed testing, thus need to make sure that disk
>> cache
>> does not affect me. Is clearing the OS (Ubuntu) disk cache, ( by running:
>> sudo echo 3 | sudo tee /proc/sys/vm/drop_caches ) enough to do this?
>
> What you should do is:
>
> 1) Shutdown the database server (pg_ctl, sudo service postgresql stop, etc.)
> 2) sync
> 3) sudo echo 3 > /proc/sys/vm/drop_caches
> 4) Start the database server
>
> That will clear both the database and OS cache with a minimum of junk left
> behind in the process; clearing the cache without a sync is a bad idea.
>
> Note that all of this will still leave behind whatever cache is in your disk
> controller card or on the disk themselves available.  There are some other
> techniques you could consider.  Add a setp 2.5 that generates a bunch of data
> unused by the test, then sync again, and you've turned most of that into
> useless caching.
>
> Ideally, your test should be running against a large enough data set that the
> dozens or couple of hundred megabytes that might be in those will only add a
> bit of noise to whatever you're testing.  If you're not running a larger test
> or going through tasts to make the caches clear, the only easy way to make
> things more clear is to reboot the whole server.
>
> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
>
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

On Mon, Sep 29, 2008 at 02:55:52AM -0400, Greg Smith wrote:
> On Mon, 29 Sep 2008, Matthew Pulis wrote:
> >I need to perform some timed testing, thus need to make sure that
> >disk cache does not affect me. Is clearing the OS (Ubuntu) disk
> >cache, (by running: sudo echo 3 | sudo tee /proc/sys/vm/drop_caches)
> >enough to do this?
>
> 3) sudo echo 3 > /proc/sys/vm/drop_caches

I'm not sure about the rest, but shouldn't this be:

  echo 3 | sudo tee /proc/sys/vm/drop_caches

the OP's "sudo echo 3" seems redundant; "echo" doesn't need to be run
with higher privileges, only the writing process needs that.  As an
aside, it would be nicer if there was a more appropriately program than
tee but I've yet to find one.


  Sam

On Mon, 29 Sep 2008, Sam Mason wrote:

> On Mon, Sep 29, 2008 at 02:55:52AM -0400, Greg Smith wrote:
>> 3) sudo echo 3 > /proc/sys/vm/drop_caches
>
> I'm not sure about the rest, but shouldn't this be:
>  echo 3 | sudo tee /proc/sys/vm/drop_caches

I couldn't think of any reason to actually include the tee in there and
just optimized displaying the "3" out as script noise.

> As an aside, it would be nicer if there was a more appropriately program
> than tee but I've yet to find one.

What are you trying to accomplish here that tee isn't quite right for?

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

On Mon, Sep 29, 2008 at 05:53:02PM -0400, Greg Smith wrote:
> On Mon, 29 Sep 2008, Sam Mason wrote:
> > echo 3 | sudo tee /proc/sys/vm/drop_caches
> >As an aside, it would be nicer if there was a more appropriate program
> >than tee but I've yet to find one.
>
> What are you trying to accomplish here that tee isn't quite right for?

tee works for how it's being used for here; it's just wired into my
brain for tasks other than writing to files with different privileges.
I ended up with a similar invocation when I was solving a similar
problem a couple of months ago (not sure how I'd not hit it before then)
and kept searching for something more natural.

I've just done a search and using tee here appears to be somewhat
standard, so I guess that answers my question.  Sorry for the noise!


  Sam