Skip to content

Instantly share code, notes, and snippets.

@luciyer
Created January 21, 2021 23:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save luciyer/5157d591b56db80b95bd1453b5a6cb5a to your computer and use it in GitHub Desktop.
Save luciyer/5157d591b56db80b95bd1453b5a6cb5a to your computer and use it in GitHub Desktop.
Batch execution against a list of items with sleep interval
class BatchQuery {
constructor(connection, queryList, batchSize = 10, sleepMs = 1000) {
this.connection = connection
this.queryList = queryList
this.batchSize = batchSize
this.sleepMs = sleepMs
this.queryResults = []
}
async singleQuery(q) {
try {
// Your query logic goes here
// const result = await connection.post('/api/endpoint/', q)
// return result
} catch (error) {
return Promise.reject(error)
}
}
async execute() {
let sleep = () => new Promise(resolve => setTimeout(resolve, this.sleepMs))
const batchedList = this.queryList.reduce((temp, x, i) => {
const subIndex = Math.floor(i / this.batchSize)
if(!temp[subIndex])
temp[subIndex] = []
temp[subIndex].push(x)
return temp
}, [])
for (const [index, batch] of batchedList.entries()) {
this.queryResults.push(await Promise.allSettled(batch.map(q => this.singleQuery(q))))
await sleep()
}
}
get results() {
return this.queryResults.flat()
}
}
const queryAll = async (connection, someList) => {
const batchQuery = new BatchQuery(connection, someList)
await batchQuery.execute()
return batchQuery.results
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment