Re: pg_dump directory archive format / parallel pg_dump

On 20.01.2011 17:22, Heikki Linnakangas wrote:
> (I'm working on this, no need to submit a new patch)

Ok, here's a heavily refactored version of this (also available at
git://git.postgresql.org/git/users/heikki/postgres.git, branch
pg_dump_directory). The directory format is now identical to the tar
format, except that in the directory format the files can be compressed.
Also we don't write the restore.sql file - it would be nice to have, but
pg_restore doesn't require it. We can leave that as a TODO.

I ended up writing another compression abstraction layer in
compress_io.c. It wraps fopen / gzopen etc. in a common API, so that the
caller doesn't need to care if the file is compressed or not. In
hindsight, the compression API we put in earlier didn't suit us very
well. But I guess it wasn't a complete waste, as it moved the gory
details of zlib out of the custom format code.

If compression is used, the files are created with the .gz suffix, and
include the gzip header so that you can manipulate them easily with
gzip/gunzip utilities. When reading, we accept files with or without the
.gz suffix, and you can have some files compressed and others uncompressed.

I haven't updated the documentation yet.

There's one UI thing that bothers me. The option to specify the target
directory is called --file. But it's clearly not a file. OTOH, I'd hate
to introduce a parallel --dir option just for this. Any thoughts on this?

--
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com