Skip to content

Instantly share code, notes, and snippets.

View oneryalcin's full-sized avatar

Mehmet Öner Yalçın oneryalcin

View GitHub Profile
@oneryalcin
oneryalcin / description.md
Created April 4, 2017 13:28 — forked from mangecoeur/description.md
Pandas PostgresSQL support for loading to DB using fast COPY FROM method

This small subclass of the Pandas sqlalchemy-based SQL support for reading/storing tables uses the Postgres-specific "COPY FROM" method to insert large amounts of data to the database. It is much faster that using INSERT. To acheive this, the table is created in the normal way using sqlalchemy but no data is inserted. Instead the data is saved to a temporary CSV file (using Pandas' mature CSV support) then read back to Postgres using Psychopg2 support for COPY FROM STDIN.