Right. I managed to start up Postgres by symlinking the following directories to a new mount: pg_logical, pg_subtrans, pg_wal, pg_xact.
I then created a new tablespace on the new mount, set it to be the default tablespace, and moved some of the smaller (about 30GB) tables to it. That allowed me to delete old data, do full vacuum, and move the data back to the original disk.
This is timeseries data and there is a daily process that deletes the old stuff, but vacuum full still fails to return the space to the OS. Perhaps I will get better results with table partitioning, or with TimeScaleDB.
Thank you for your help,
Igal
On Wed, May 1, 2019 at 11:08 PM Michael Nolan <htfoot@gmail.com> wrote:
Assuming you get the database back online, I would suggest you put a procedure in place to monitor disk space and alert you when it starts to get low.