Skip to content

Instantly share code, notes, and snippets.

@davimacedo
Last active June 23, 2021 21:33
Show Gist options
  • Save davimacedo/c7738203a2a616b48a22fdc7c00b37d2 to your computer and use it in GitHub Desktop.
Save davimacedo/c7738203a2a616b48a22fdc7c00b37d2 to your computer and use it in GitHub Desktop.
Example of importing data with cloud functions. See a live example at https://www.back4app.com/database/davimacedo/parse-import-example
curl -X POST \
-H "X-Parse-Application-Id: LL9oIdzIkmwl5xyowQQu0fTmXyUWfet9RuAzwHfj" \
-H "X-Parse-REST-API-Key: R3S8PYQKuzeV4c8MUeO5ved46C50MEp56boDHW1O" \
-H "Content-Type: application/json" \
-d @data.json \
https://parseapi.back4app.com/functions/import
{
"className": "ExampleClass",
"rows": [
{ "ExampleColumnA": "row1columnA", "ExampleColumnB": "row1columnB" },
{ "ExampleColumnA": "row2columnA", "ExampleColumnB": "row2columnB"}
]
}
Parse.Cloud.define('import', async request => {
const className = request.params.className;
const rows = request.params.rows;
const MyClass = Parse.Object.extend(className);
var myClassObjects = [];
for (let i = 0; i < rows.length; i++) {
const myClassObject = new MyClass();
for (const column in rows[i]) {
myClassObject.set(column, rows[i][column]);
}
myClassObjects.push(myClassObject);
}
try {
await Parse.Object.saveAll(myClassObjects);
} catch (e) {
throw new Error(`Import failed: ${e}`);
}
return `Successfully imported ${myClassObjects.length} rows into ${className} class`;
});
@davimacedo
Copy link
Author

This code example is only able to include new rows. This script would not be able to update rows. As I understood you are trying to upload data with ObejctId and that would require script to update this data. That's why it is failing. You will have to change script little bit to make it update.

@archiejudd
Copy link

Hi Davi - this works great, except after importing a json file with 8 columns, 3 of them are missing at the parse dashboard. Any ideas on why this might be the case?

@archiejudd
Copy link

I have now worked out my mistake - I was using spaces and dashes in the column names! Thanks

@mpc20001
Copy link

I keep getting this error - {"code":141,"error":"Import failed: undefined"} after trying to import through curl.

@mpc20001
Copy link

nevermind, i just wasn't using my master key. It works great.

@mpc20001
Copy link

Actually I tried importing a 1mb file of about 10,000 rows and the connection reset. It worked for much smaller amounts though. What is the size limit for imports through this method?

@davimacedo
Copy link
Author

There is no specific limit, but 10,000 would reach your api 10,000 times. You would have to modify little bit the code to make it upload something like 1000 each minute.

@Mamonaku
Copy link

Mamonaku commented Jan 6, 2017

@mpc20001 keep getting the same error, while using the master key
promises.push(myClassObject.save({useMasterKey:true}));

I have client class creation set, even trying on small file..... not sure what is blocking

Log files say
Fri Jan 06 2017 12:31:53 GMT+0000 (UTC) Error: Can't set headers after they are sent. at ServerResponse.OutgoingMessage.setHeader (_http_outgoing.js:344:11) at ServerResponse.header (/usr/src/app/node_modules/express/lib/response.js:719:10) at ServerResponse.send (/usr/src/app/node_modules/express/lib/response.js:164:12) at ServerResponse.json (/usr/src/app/node_modules/express/lib/response.js:250:15) at /usr/src/app/src/back/app.js:104:9 at Layer.handle_error (/usr/src/app/node_modules/express/lib/router/layer.js:71:5) at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:310:13) at /usr/src/app/node_modules/express/lib/router/index.js:280:7 at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:330:12) at next (/usr/src/app/node_modules/express/lib/router/index.js:271:10) at Layer.handle_error (/usr/src/app/node_modules/express/lib/router/layer.js:67:12) at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:310:13) at /usr/src/app/node_modules/express/lib/router/index.js:280:7 at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:330:12) at next (/usr/src/app/node_modules/express/lib/router/index.js:271:10) at Immediate.<anonymous> (/usr/src/app/node_modules/express/lib/application.js:232:9)

@Mamonaku
Copy link

Mamonaku commented Jan 6, 2017

Seems there is a problem with the promise fulfilment. I changed the code for
Parse.Object.saveAll(myObjectArray).
Unfortunately, the code can save the entries as new objects, but can not migrate data... in other words, attempts to set the object ID results in
{"code":141,"error":"Import failed: Object not found."}

Any ideas?

@Treverr
Copy link

Treverr commented Apr 6, 2017

This does not work, I get "unexpected token /" or c or t

@nyxee
Copy link

nyxee commented Jun 18, 2017

I can't get any of this to work either..
I put main.js in my Parse sever directoy and called http://localhost:1337/parse/main.js "returns :error, unouthorize"

@aacassandra
Copy link

hi Davi, this works great, you save my day. @nyxee im tested on localhost and works. your url change to http://localhost:1337/parse/function/{name}

@davimacedo
Copy link
Author

Glad to know!

@miladabc
Copy link

Is there a better way of importing data to parse server?
Imagine thousand of rows to import, I'm worried about those for loops there.

@davimacedo
Copy link
Author

davimacedo commented Jan 5, 2020

@Milad-abbasi I've just updated the example to be compliant with the latest version of Parse and also to use Parse.Object.saveAll(). It conveniently splits the requests in batches so you can make sure it works properly. I also added the link for a living example at the Back4App Database Hub so you can just run and see the imported rows or clone, check the cloud code out and play around.

@randyheaton
Copy link

Little heads up for anyone looking. I feel like there could be a small versioning issue between the function here and the one posted in the live example. The version on this page revised on Jan 5 2020 and current as of June 23 2021 works great for me and also employs the improvements the owner describes in his Jan 5 2020 comment. Whereas the example posted in the live example fails for me at the line Parse.Promise.when(promises).

@davimacedo
Copy link
Author

Thanks for the heads up. I've just updated the live example code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment