Brendan Scullion brsc2909
luke
/ bulkupsert.py
Last active
March 17, 2023 19:30
I needed to upsert (insert or update) bajillions of records into postgresql. After trying various libs including upsert (which was slow as hell) I ended up doing a bit of research and trying 3 different methods. This one won. While I'm manually building the sql string no user data is passed in. Its loaded via the copy from statement as CSV. Call…
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import logging | |
| import cStringIO | |
| import csv | |
| DEBUG = False | |
| def data2csv(data): | |
| si = cStringIO.StringIO() | |
| cw = csv.writer(si, delimiter='\t',lineterminator="\n") | |
| for row in data: |