Skip to content

Instantly share code, notes, and snippets.

@evanderkoogh
Created November 20, 2017 13:29
Show Gist options
  • Save evanderkoogh/37f6adb01268e82ecd68d46fce1c7c5a to your computer and use it in GitHub Desktop.
Save evanderkoogh/37f6adb01268e82ecd68d46fce1c7c5a to your computer and use it in GitHub Desktop.
Import CSV into DynamoDB
whateverId attribute1 someotherattribute
foo bar baz
hello erwin world
const fs = require('fs')
const parse = require('csv-parse/lib/sync')
const AWS = require('aws-sdk')
AWS.config.update({region: 'ap-southeast-2'});
const docClient = new AWS.DynamoDB.DocumentClient()
const contents = fs.readFileSync('./<filename>.csv', 'utf-8')
// If you made an export of a DynamoDB table you need to remove (S) etc from header
const data = parse(contents, {columns: true})
data.forEach((item) => {
if(!item.maybeempty) delete item.maybeempty //need to remove empty items
docClient.put({TableName: '<Table>', Item: item}, (err, res) => {
if(err) console.log(err)
})
})
{
"name": "dydbimport",
"version": "1.0.0",
"main": "index.js",
"license": "MIT",
"dependencies": {
"aws-sdk": "^2.153.0",
"csv-parse": "^2.0.0"
}
}
@SaiiKiran
Copy link

What is csv-parse/lib/sync

@sleon-fmi
Copy link

sleon-fmi commented Aug 1, 2018

@SaiiKiran, csv-parse is an NPM package.

It can be installed into a project with:

npm i -S csv-parse

Here's the package page: https://www.npmjs.com/package/csv-parse.

@abdala
Copy link

abdala commented Nov 22, 2018

Add those lines if you need to set a profile using ~/.aws/config:

var credentials = new AWS.SharedIniFileCredentials({profile: 'profile-name'});
AWS.config.credentials = credentials;

@fabioce
Copy link

fabioce commented Mar 30, 2019

Hi,
I received the following error, I think there are limits in memory.
I imagine this line:
const data = parse(contents, {columns: true})
want to work in RAM all rows.
My file is 450MB
what to you think to put record lines by lines?

<--- Last few GCs --->

[25097:0x35e00f0] 114840 ms: Mark-sweep 1632.6 (1687.3) -> 1631.7 (1688.8) MB, 420.6 / 0.0 ms allocation failure GC in old space requested
[25097:0x35e00f0] 115253 ms: Mark-sweep 1631.7 (1688.8) -> 1631.6 (1658.3) MB, 412.9 / 0.0 ms last resort GC in old space requested
[25097:0x35e00f0] 115651 ms: Mark-sweep 1631.6 (1658.3) -> 1631.6 (1658.3) MB, 398.0 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1cd27f2a58b9
1: __onRow [./node_modules/csv-parse/lib/index.js:~560] [pc=0x2f17ce990a37](this=0xcabe84da141 )
2: __parse [./node_modules/csv-parse/lib/index.js:~348] [pc=0x2f17ce98e711](this=0xcabe84da141 ,nextBuf=0xcabe84da209 ,end=0x1447c5402...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x8ccf9c [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewUninitializedFixedArray(int) [node]
6: 0xd81903 [node]
7: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [node]
8: 0x2f17ce8042fd

@rubenberna
Copy link

Hi,

When processing the file, I get an error for 'Type mismatch':

'One or more parameter values were invalid: Type mismatch for key externalId expected: N actual: S'

How do you define the data type in the CSV or overcome this error altogether?

Thanks

@shlomizadok
Copy link

@rubenberna - I have had an integer column which was part of an index.
I have modified the above to

data.forEach((item) => {
  if(!item.maybeempty) delete item.maybeempty //need to remove empty items
  item.added = parseInt(item.added); // <-- Changed this to Integer solved the Type mismatch error
  docClient.put({TableName: 'ReviewTable', Item: item}, (err, res) => {
    if(err) console.log(err)
  })      
})

@BartusZak
Copy link

BartusZak commented Sep 7, 2020

node_modules\csv-parse\lib\sync.js:30
    throw err;
    ^

Error: Invalid opening quote at line 2

@bradbyte
Copy link

This should now be ...

const {parse} = require('csv-parse/sync');

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment