Skip to content

Instantly share code, notes, and snippets.

@robere2
Last active May 3, 2024 15:28
Show Gist options
  • Save robere2/232691590952b1f1c3de1993b291b6eb to your computer and use it in GitHub Desktop.
Save robere2/232691590952b1f1c3de1993b291b6eb to your computer and use it in GitHub Desktop.
Fetching auction data from the Hypixel API
/**
* Robust function to fetch all auction pages from https://api.hypixel.net/v2/skyblock/auctions. If the auctions update during the time that
* this function is executing, then it will restart in order to fetch the latest data.
* @returns {Promise<Object[], Error>} Array of JSON responses from the API, with each entry being a page of auction results. The results are not necessarily in order.
* @reject {Error} Network error
* @reject {Error} If the function restarts more than five times due to updates or an API response with "success" set to false.
* @license MIT
* @author https://github.com/robere2
* @see {@link https://gist.github.com/robere2/232691590952b1f1c3de1993b291b6eb} Source
*/
async function fetchAllAuctionPages() {
let allPages = []
let tries = 0;
while(allPages.length === 0) {
// After five failed attempts, presumably more attempts aren't going to solve the problem.
if(tries++ >= 5) {
throw new Error("Tried to fetch auction results five times, but all attempts failed. This could be a Hypixel API error, " +
"or responses are not coming in fast enough to keep up with the updating results.")
}
const firstPage = await (await fetch("https://api.hypixel.net/v2/skyblock/auctions")).json()
// Last updated timestamp is used to restart the process if results update between now and when we finish all page requests
const lastUpdated = firstPage.lastUpdated;
const abortController = new AbortController();
// Request each following page concurrently
const promises = [];
for(let i = 1; i < firstPage.totalPages; i++) {
promises.push((async () => {
try {
const res = await fetch(`https://api.hypixel.net/v2/skyblock/auctions?page=${i}`, {
signal: abortController.signal
})
const page = await res.json();
// Abort and restart if the results have updated or API says request failed. Wait a moment to give the
// error some time to resolve itself.
if((page.lastUpdated !== lastUpdated || res.status === 404 || page.success === false)) {
abortController.abort()
await new Promise(resolve => setTimeout(() => resolve(), 1000));
}
return page;
} catch(e) {
if(e.name === "AbortError") {
return;
}
throw e;
}
})())
}
// Await all of the page requests to complete, and add their responses to an array with the first page
allPages = [firstPage, ...await Promise.all(promises)]
// Aborted due to failure or updated list, restart by deleting the responses
if(abortController.signal.aborted) {
allPages = [];
}
}
return allPages;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment