Re: Load a csv or a avro?
| От | Muhammad Ikram |
|---|---|
| Тема | Re: Load a csv or a avro? |
| Дата | |
| Msg-id | CAGeimVo_EOJO+BViDnLoxEdFoFKQkeHU=gniEQ9e2GbjAUvUHg@mail.gmail.com обсуждение исходный текст |
| Ответ на | Re: Load a csv or a avro? (Josef Šimánek <josef.simanek@gmail.com>) |
| Список | pgsql-general |
Hi,
Performance Considerations
Avro files are smaller due to compression so needing less I/O time. whereas CSV files are simpler but larger in size so read/write will need more time.
COPY command works very well with CSV files whereas ETL process is required for handling Avro.
Regards,
Muhammad Ikram
On Fri, Jul 5, 2024 at 3:03 PM Josef Šimánek <josef.simanek@gmail.com> wrote:
pá 5. 7. 2024 v 11:08 odesílatel sud <suds1434@gmail.com> napsal:
>
> Hello all,
>
> Its postgres database. We have option of getting files in csv and/or in avro format messages from another system to load it into our postgres database. The volume will be 300million messages per day across many files in batches.
>
> My question was, which format should we chose in regards to faster data loading performance ? and if any other aspects to it also should be considered apart from just loading performance?
We are able to load ~300 million rows per one day using CSV and COPY
functions (https://www.postgresql.org/docs/current/libpq-copy.html#LIBPQ-COPY-SEND).
Muhammad Ikram
В списке pgsql-general по дате отправления: