Respect rate-limits of SteamSpy API

steamspy.py

import json
import time
from pathlib import Path

import steamspypi


def get_cooldown():
    cooldown = 70  # 1 minute plus a cushion

    return cooldown


def get_some_sleep():
    cooldown = get_cooldown()
    print("Sleeping for {} seconds on {}".format(cooldown, time.asctime()))

    time.sleep(cooldown)

    return


def download_a_single_page(page_no=0):
    print("Downloading page={} on {}".format(page_no, time.asctime()))

    data_request = dict()
    data_request["request"] = "all"
    data_request["page"] = str(page_no)

    data = steamspypi.download(data_request)

    return data


def get_file_name(page_no):
    # Get current day as yyyymmdd format
    date_format = "%Y%m%d"
    current_date = time.strftime(date_format)
    
    file_name = "{}_steamspy_page_{}.json".format(current_date, page_no)
    
    return file_name


def download_all_pages(num_pages):
    # Download

    for page_no in range(num_pages):
        file_name = get_file_name(page_no)

        if not Path(file_name).is_file():
            page_data = download_a_single_page(page_no=page_no)

            with open(file_name, "w", encoding="utf8") as f:
                json.dump(page_data, f)

            if page_no != (num_pages - 1):
                get_some_sleep()

    # Aggregate

    data = dict()

    for page_no in range(num_pages):
        file_name = get_file_name(page_no)

        with open(file_name, "r", encoding="utf8") as f:
            page_data = json.load(f)

            data.update(page_data)

    return data


if __name__ == "__main__":
    # TODO: one would have to figure out the number of pages, it should be close to 40 as of August 2020.
    data = download_all_pages(num_pages=40)

Update to the number of pages as of 13th April, 2022: there are 53 Pages

Update to the number of pages as of 13th April, 2022: there are 53 Pages

How many pages now?

You can run the script with a larger number and figure out the correct number of pages: i) it should not be much more since it increased from 40 in August 2020 to 53 in April 2022 and we are in July 2023, ii) except for the last page, a page consists of 1000 games.

It is 64 pages now.

Thank you for the info!

You forgot to add a command to save the final JSON file after connecting all the pages together. Moreover, adding the date to the downloaded files make them unusable after running the script in the next day. Furthermore, there are some lines of code that can be removed, and the script will run the same.

You forgot to add a command to save the final JSON file after connecting all the pages together.

1. It is not important to save the aggregate as pages can be aggregated again really fast after they have been saved to disk.

Moreover, adding the date to the downloaded files make them unusable after running the script in the next day.

2. I had in mind to download data from scratch on different days. If you want to download data only once and then reuse the result on different days, you could save the aggregate as you pointed in 1), and you can choose a filename of your choosing for the saved aggregate.

Furthermore, there are some lines of code that can be removed, and the script will run the same.

3. This is an example with readability in mind. The limiting factor for performance is SteamSpy's rate-limit, so code cannot be optimized much.

First, thanks for the module and the gist, they do the job perfectly.

Is there another an API that can be used to retrieve data like "executable", "arguments", and "steam_deck_compatibility". The only thing that I have found so far is https://steamdb.info/ which does not have an API, and prohibit web scraping, and steamcmd does not work with games that are not in my library.

Is there another an API that can be used to retrieve data like "executable", "arguments", and "steam_deck_compatibility"?

I don't know for "executable" or "arguments". For "steam_deck_compatibility", you may be able to use resolved_category from:
https://store.steampowered.com/saleaction/ajaxgetdeckappcompatibilityreport?nAppID=570

I will figure something. Thanks.