Skip to content

Instantly share code, notes, and snippets.

@hmontazeri
Last active April 2, 2024 03:00
Show Gist options
  • Save hmontazeri/e9493c2110d4640a5d10429ccbafb616 to your computer and use it in GitHub Desktop.
Save hmontazeri/e9493c2110d4640a5d10429ccbafb616 to your computer and use it in GitHub Desktop.
get more than 1000 elements from s3 bucket (node.js)
const AWS = require('aws-sdk');
const s3 = new AWS.S3({
region: 'eu-central-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
});
async function listAllObjectsFromS3Bucket(bucket, prefix) {
let isTruncated = true;
let marker;
const elements = [];
while(isTruncated) {
let params = { Bucket: bucket };
if (prefix) params.Prefix = prefix;
if (marker) params.Marker = marker;
try {
const response = await s3.listObjects(params).promise();
response.Contents.forEach(item => {
elements.push(item.Key);
});
isTruncated = response.IsTruncated;
if (isTruncated) {
marker = response.Contents.slice(-1)[0].Key;
}
} catch(error) {
throw error;
}
}
return elements;
}
// example call
listAllObjectsFromS3Bucket('<your bucket name>', '<optional prefix>');
@hmontazeri
Copy link
Author

modified from: codingfundas

@nico355
Copy link

nico355 commented Sep 7, 2022

Excellent script! Great contribution, thank you very much

@mfrodrigo
Copy link

Excellent code! Thank you very much

@matealves
Copy link

Thank you very much, excellent code!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment