Re: Testing of a fast method to bulk insert a Pandas DataFrame into Postgres

What is the robust way to upgrade Pandas?

Regards,

David

On Monday, 4 October 2021, Adrian Klaver <adrian.klaver@aklaver.com> wrote:

On 10/4/21 9:20 AM, Shaozhong SHI wrote:

Hello, Adrian Klaver,

Pandas version is 0.23.0.

The reason the below does not work is method did not show up until pandas 0.24.0.

I used the following code:

def psql_insert_copy(table, conn, keys, data_iter):
     # gets a DBAPI connection that can provide a cursor
     dbapi_conn = conn.connection
     with dbapi_conn.cursor() as cur:
         s_buf = StringIO()
         writer = csv.writer(s_buf)
         writer.writerows(data_iter)
         s_buf.seek(0)

         columns = ', '.join('"{}"'.format(k) for k in keys)
         if table.schema:
             table_name = '{}.{}'.format(table.schema, table.name <http://table.name>)
         else:
             table_name = table.name <http://table.name>

         sql = 'COPY {} ({}) FROM STDIN WITH CSV'.format(
             table_name, columns)
         cur.copy_expert(sql=sql, file=s_buf)
engine = create_engine('postgresql+psycopg2://:5432/postgres')
try:
     df.to_sql('test1', engine, schema='public', if_exists='append', index=False, method=psql_insert_copy)

I could not find obvious reasons.

Regards,

David

--
Adrian Klaver
adrian.klaver@aklaver.com