Skip to content

Instantly share code, notes, and snippets.

@vizowl
Created April 15, 2013 09:41
Show Gist options
  • Save vizowl/5387015 to your computer and use it in GitHub Desktop.
Save vizowl/5387015 to your computer and use it in GitHub Desktop.
Pythonic way to import variable column length csv data into postgresql
import psycopg2 as dbapi
import csv
from itertools import izip, count
if __name__ == "__main__":
con = dbapi.connect(database='play')
cur = con.cursor()
data = ([str(212), str(i)] + row + (15 - len(row)) * [','] for row, i in izip(csv.reader(open('test.csv')), count()))
sql = open('test.sql').read()
t = cur.executemany(sql, data)
con.commit()
We can make this file beautiful and searchable if this error is corrected: It looks like row 2 should actually have 4 columns, instead of 10. in line 1.
100,1234,1.6,100
101,1234,9,0:6:57,10/9/2010,-40.76414, 173.98544,1,7.7,3.9
102,1234,9,50034,52,0:35:44,10/9/2010,-40.76205, 173.98703,1,1.4
103,1234,9,50034,52,7,0,456,0,0,123,0,0,83,0
104,1234,1.,3,1,00
105,1234,0:36:19,10/9/2010,3.8
BEGIN;
CREATE TABLE IF NOT EXISTS _datatable (
field_0 INTEGER,
field_1 INTEGER,
field_2 TEXT,
field_3 TEXT,
field_4 TEXT,
field_5 TEXT,
field_6 TEXT,
field_7 TEXT,
field_8 TEXT,
field_9 TEXT,
field_10 TEXT,
field_11 TEXT,
field_12 TEXT,
field_13 TEXT,
field_14 TEXT,
field_15 TEXT,
field_16 TEXT
);
INSERT INTO _datatable VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment