Re: export to parquet
| От | George Woodring |
|---|---|
| Тема | Re: export to parquet |
| Дата | |
| Msg-id | CACi+J=QB8g3SLXHSob2eh1BBqyxYNvf07qvP+gqA-knw4PcHwQ@mail.gmail.com обсуждение исходный текст |
| Ответ на | export to parquet (Scott Ribe <scott_ribe@elevated-dev.com>) |
| Список | pgsql-general |
I don't know how many hoops you want to jump through, we use AWS and Athena to create them.
- Export table as JSON
- Put on AWS S3
- Create JSON table in Athena
- Use the JSON table to create a parquet table
The parquet files will be in S3 as well after the parquet table is created. If you are interested I can share the AWS CLI commands we use.
George Woodring
iGLASS Networks
www.iglass.net
www.iglass.net
On Wed, Aug 26, 2020 at 3:00 PM Scott Ribe <scott_ribe@elevated-dev.com> wrote:
I have no Hadoop, no HDFS. Just looking for the easiest way to export some PG tables into Parquet format for testing--need to determine what kind of space reduction we can get before deciding whether to look into it more.
Any suggestions on particular tools? (PG 12, Linux)
--
Scott Ribe
scott_ribe@elevated-dev.com
https://www.linkedin.com/in/scottribe/
В списке pgsql-general по дате отправления: