Created
April 15, 2013 09:41
-
-
Save vizowl/5387015 to your computer and use it in GitHub Desktop.
Pythonic way to import variable column length csv data into postgresql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import psycopg2 as dbapi | |
import csv | |
from itertools import izip, count | |
if __name__ == "__main__": | |
con = dbapi.connect(database='play') | |
cur = con.cursor() | |
data = ([str(212), str(i)] + row + (15 - len(row)) * [','] for row, i in izip(csv.reader(open('test.csv')), count())) | |
sql = open('test.sql').read() | |
t = cur.executemany(sql, data) | |
con.commit() |
We can make this file beautiful and searchable if this error is corrected: It looks like row 2 should actually have 4 columns, instead of 10. in line 1.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
100,1234,1.6,100 | |
101,1234,9,0:6:57,10/9/2010,-40.76414, 173.98544,1,7.7,3.9 | |
102,1234,9,50034,52,0:35:44,10/9/2010,-40.76205, 173.98703,1,1.4 | |
103,1234,9,50034,52,7,0,456,0,0,123,0,0,83,0 | |
104,1234,1.,3,1,00 | |
105,1234,0:36:19,10/9/2010,3.8 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BEGIN; | |
CREATE TABLE IF NOT EXISTS _datatable ( | |
field_0 INTEGER, | |
field_1 INTEGER, | |
field_2 TEXT, | |
field_3 TEXT, | |
field_4 TEXT, | |
field_5 TEXT, | |
field_6 TEXT, | |
field_7 TEXT, | |
field_8 TEXT, | |
field_9 TEXT, | |
field_10 TEXT, | |
field_11 TEXT, | |
field_12 TEXT, | |
field_13 TEXT, | |
field_14 TEXT, | |
field_15 TEXT, | |
field_16 TEXT | |
); | |
INSERT INTO _datatable VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment