Обсуждение: Restore of pg_dump taking a long time...

Поиск
Список
Период
Сортировка

Restore of pg_dump taking a long time...

От
"Ruairi"
Дата:
Hi,
I'm currently restoring a fairly large DB from a pg_dump and it's taking
about 12 hours to finish. The main part of this time is spent creating
indexes. Is there anyway I can speed up the restore process, or do i just
have to wait? I'm using postgres 8.1.3 on freebsd and the pg_dump is gzip'd.
I'm restoring with the following command : "zcat <gzip'd db> | psql -U
postgres <destination db>"

Regards,
Ruairi Carroll


Re: Restore of pg_dump taking a long time...

От
Tom Lane
Дата:
"Ruairi" <rcarroll@bluemetrix.com> writes:
> I'm currently restoring a fairly large DB from a pg_dump and it's taking
> about 12 hours to finish. The main part of this time is spent creating
> indexes. Is there anyway I can speed up the restore process, or do i just
> have to wait?

There's not much you can do to improve the performance of a restore
already in progress.  Next time you might want to think about kicking up
maintenance_work_mem before you start it; that's about the only thing
you can really do to speed up index builds.

(Actually, if there are lots of indexes yet to be built, increasing
the setting in postgresql.conf and SIGHUP'ing the postmaster could
be worth doing, as that should affect the builds yet to be done.)

            regards, tom lane

Re: Restore of pg_dump taking a long time...

От
"mcelroy, tim"
Дата:

I have found that the following steps have increased the speed of my restores:
1. dropdb <DBNAME>
2. createdb <DBANME>
3. Increase maintenance_work_mem as Tom mentioned.  I do this at restore/runtime
4. Increase work_mem as Tom mentioned.  I do this at restore/runtime
5. renice -20 -p <pid #> (on a Linux box)  <= This step can be performed while the restore is running but you need root privilege

Tim

 -----Original Message-----
From:   pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org]  On Behalf Of Tom Lane
Sent:   Wednesday, May 24, 2006 2:48 PM
To:     Ruairi
Cc:     pgsql-admin@postgresql.org
Subject:        Re: [ADMIN] Restore of pg_dump taking a long time...

"Ruairi" <rcarroll@bluemetrix.com> writes:
> I'm currently restoring a fairly large DB from a pg_dump and it's taking
> about 12 hours to finish. The main part of this time is spent creating
> indexes. Is there anyway I can speed up the restore process, or do i just
> have to wait?

There's not much you can do to improve the performance of a restore
already in progress.  Next time you might want to think about kicking up
maintenance_work_mem before you start it; that's about the only thing
you can really do to speed up index builds.

(Actually, if there are lots of indexes yet to be built, increasing
the setting in postgresql.conf and SIGHUP'ing the postmaster could
be worth doing, as that should affect the builds yet to be done.)

                        regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Re: Restore of pg_dump taking a long time...

От
Scott Marlowe
Дата:
On Wed, 2006-05-24 at 13:56, mcelroy, tim wrote:
> I have found that the following steps have increased the speed of my
> restores:
> 1. dropdb <DBNAME>
> 2. createdb <DBANME>
> 3. Increase maintenance_work_mem as Tom mentioned.  I do this at
> restore/runtime
> 4. Increase work_mem as Tom mentioned.  I do this at restore/runtime
> 5. renice -20 -p <pid #> (on a Linux box)  <= This step can be
> performed while the restore is running but you need root privilege

Note that if there's no other data in the database that you need to
worry about, you can also restore with fsync off.  I.e. if a database
cluster could just be re-inited should a power failure occur.

If you've got already important data you can't afford to lose in the
cluster, then do not turn off fsync.

Also, remember to turn it back on before you go into production.

Re: Restore of pg_dump taking a long time...

От
Tom Lane
Дата:
"Todd A. Cook" <tcook@blackducksoftware.com> writes:
> I have found that increasing maintenance_work_mem can decrease index
> build speed on large tables:

You should probably re-measure when 8.2 comes out; we've fixed a number
of performance issues in the sorting code that might cause that.

            regards, tom lane

Re: Restore of pg_dump taking a long time...

От
"Todd A. Cook"
Дата:
Tom Lane wrote:
> "Todd A. Cook" <tcook@blackducksoftware.com> writes:
>> I have found that increasing maintenance_work_mem can decrease index
>> build speed on large tables:
>
> You should probably re-measure when 8.2 comes out; we've fixed a number
> of performance issues in the sorting code that might cause that.

Thanks.  I'll do that.  If I can, I'll try it sooner on a build
from CVS.

-- todd

Re: Restore of pg_dump taking a long time...

От
Naomi Walker
Дата:
I would typically drop the indicies before taking the dump, and recreate
in the new instance.  I know the loads go alot faster that way.  Perhaps
the index creation would be the same.

Tom Lane wrote:
> "Ruairi" <rcarroll@bluemetrix.com> writes:
>> I'm currently restoring a fairly large DB from a pg_dump and it's taking
>> about 12 hours to finish. The main part of this time is spent creating
>> indexes. Is there anyway I can speed up the restore process, or do i just
>> have to wait?
>
> There's not much you can do to improve the performance of a restore
> already in progress.  Next time you might want to think about kicking up
> maintenance_work_mem before you start it; that's about the only thing
> you can really do to speed up index builds.
>
> (Actually, if there are lots of indexes yet to be built, increasing
> the setting in postgresql.conf and SIGHUP'ing the postmaster could
> be worth doing, as that should affect the builds yet to be done.)
>
>             regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster


--
----------------------------------------------------------------------------
Naomi Walker                          Chief Information Officer
Eldorado Computing, Inc.              nwalker@eldocomp.com 602-604-3100
----------------------------------------------------------------------------
What lies behind us and what lies before us are tiny matters compared to
what lies within us.     - William Morrow
----------------------------------------------------------------------------

-- CONFIDENTIALITY NOTICE --

Information transmitted by this e-mail is proprietary to MphasiS and/or its Customers and is intended for use only by
theindividual or entity to which it is addressed, and may contain information that is privileged, confidential or
exemptfrom disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been
forwardedto you without proper authority, you are notified that any use or dissemination of this information in any
manneris strictly prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete this
mailfrom your records. 

Re: Restore of pg_dump taking a long time...

От
"Todd A. Cook"
Дата:
Tom Lane wrote:
>
> There's not much you can do to improve the performance of a restore
> already in progress.  Next time you might want to think about kicking up
> maintenance_work_mem before you start it; that's about the only thing
> you can really do to speed up index builds.

I have found that increasing maintenance_work_mem can decrease index
build speed on large tables:

maintenance_work_mem    32768    262144     1048576     2097151
---------------------------------------------------------------
index 400M rows (min)      55        61          68          70
index  45M rows (min)    4.03      4.22        4.88        2.55

These timings were done on a machine with 8GB ram and postgresql 8.1.3
(from the RPMs on postgresql.org).  Each time is an average of three
runs; before each run, postgres was shut down and the filesystem with
/var/lib/pgsql was remounted to clear the filesystem cache.

I'm assuming that the times are increasing because there is less
memory available for the filesystem cache.  Similarly, I assume that
the runs on the smaller table with maintenance_work_mem = 2097151
are faster because everything fits in that space.

-- todd

Re: Restore of pg_dump taking a long time...

От
Jim Nasby
Дата:
On May 24, 2006, at 2:56 PM, Todd A. Cook wrote:

> Tom Lane wrote:
>> "Todd A. Cook" <tcook@blackducksoftware.com> writes:
>>> I have found that increasing maintenance_work_mem can decrease index
>>> build speed on large tables:
>> You should probably re-measure when 8.2 comes out; we've fixed a
>> number
>> of performance issues in the sorting code that might cause that.
>
> Thanks.  I'll do that.  If I can, I'll try it sooner on a build
> from CVS.

If you'll be messing around with CVS, you might also want to try this
patch: http://jim.nasby.net/misc/pgsqlcompression/compress-sort.patch
(which was written by someone else). It hacks compression into the on-
disk sort code, which has shown a 50% speed improvement on my
machine. It should be fine to use for loading a database, but you
wouldn't want to leave it in for serious use (IIRC there's some cases
it flat-out doesn't handle right now).

You could probably apply that to 8.1.4 as well if you wanted to; it
should be fine for just loading the database.
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461