Skip to content

Instantly share code, notes, and snippets.

@simonw
Created March 26, 2023 16:07
Show Gist options
  • Star 16 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save simonw/8fbade64a3606645243b3438930b8159 to your computer and use it in GitHub Desktop.
Save simonw/8fbade64a3606645243b3438930b8159 to your computer and use it in GitHub Desktop.

Fetch and Save JSON

2023-03-26 01:44:42

user: Write a new JavaScript fetch() function which calls the old one but also saves the JSON that is retrieved - if it is used to retrieve JSON - in a window.fetchedData object


assistant: To achieve this, you can create a new function called fetchAndSaveJson that wraps the native fetch function. This new function will call the original fetch function, and if the response contains JSON data, it will save the JSON data to a window.fetchedData object. Here's how you can implement this:

// Define a global object to store fetched JSON data
window.fetchedData = {};

// Define a new fetch function that wraps the native fetch
async function fetchAndSaveJson(url, options) {
  try {
    // Call the native fetch function
    const response = await fetch(url, options);

    // Check if the response has a JSON content type
    const contentType = response.headers.get('content-type');
    if (contentType && contentType.includes('application/json')) {
      // If the response is JSON, parse it and save it to the fetchedData object
      const jsonData = await response.json();
      window.fetchedData[url] = jsonData;
      return jsonData;
    }

    // If the response is not JSON, return the original response
    return response;
  } catch (error) {
    // Handle any errors that occur during the fetch
    console.error('Error fetching and saving JSON:', error);
    throw error;
  }
}

// Example usage
fetchAndSaveJson('https://jsonplaceholder.typicode.com/todos/1')
  .then(data => {
    console.log('Fetched JSON data:', data);
    console.log('Data saved in window.fetchedData:', window.fetchedData);
  })
  .catch(error => {
    console.error('Error:', error);
  });

In this implementation, the fetchAndSaveJson function takes the same arguments as the native fetch function (url and options). It then calls the native fetch function with these arguments and checks the content-type header of the response to determine if the response contains JSON data. If the response is JSON, the function parses the JSON data, saves it to the window.fetchedData object using the URL as the key, and returns the parsed JSON data. If the response is not JSON, the function returns the original response.

Note that this implementation assumes that the window.fetchedData object is used exclusively for storing fetched JSON data, and that the URLs used to fetch data are unique. If there are multiple requests to the same URL, the data from the latest request will overwrite the data from previous requests.


user: No I need to replace the old window.fetch() function entirely - I want to patch it so that I can intercept calls made by other existing code


assistant: To replace the native window.fetch function and intercept calls made by other existing code, you can create a new function that wraps the original fetch function. You can then assign this new function to window.fetch, effectively "patching" the native fetch function. Here's how you can do this:

// Define a global object to store fetched JSON data
window.fetchedData = {};

// Store a reference to the original fetch function
const originalFetch = window.fetch;

// Define a new fetch function that wraps the original fetch
window.fetch = async function(url, options) {
  try {
    // Call the original fetch function
    const response = await originalFetch(url, options);

    // Check if the response has a JSON content type
    const contentType = response.headers.get('content-type');
    if (contentType && contentType.includes('application/json')) {
      // If the response is JSON, clone the response so we can read it twice
      const responseClone = response.clone();
      // Parse the JSON data and save it to the fetchedData object
      const jsonData = await responseClone.json();
      window.fetchedData[url] = jsonData;
    }

    // Return the original response
    return response;
  } catch (error) {
    // Handle any errors that occur during the fetch
    console.error('Error fetching and saving JSON:', error);
    throw error;
  }
};

// Example usage
fetch('https://jsonplaceholder.typicode.com/todos/1')
  .then(response => response.json())
  .then(data => {
    console.log('Fetched JSON data:', data);
    console.log('Data saved in window.fetchedData:', window.fetchedData);
  })
  .catch(error => {
    console.error('Error:', error);
  });

In this implementation, we store a reference to the original fetch function in the originalFetch variable. We then define a new function that calls originalFetch with the provided arguments, checks the content-type header of the response to determine if the response contains JSON data, and saves the JSON data to the window.fetchedData object using the URL as the key. Finally, we assign this new function to window.fetch, effectively replacing the native fetch function.

Note that we use response.clone() to create a clone of the response before reading the JSON data. This is necessary because the response body can only be read once, and we want to allow the caller to read the response body as well.

This implementation assumes that the window.fetchedData object is used exclusively for storing fetched JSON data, and that the URLs used to fetch data are unique. If there are multiple requests to the same URL, the data from the latest request will overwrite the data from previous requests.


@noonething
Copy link

Thank you for this. Would there be a way to save the object to a file each time?

@Colton
Copy link

Colton commented Mar 31, 2023

Thank you for this. Would there be a way to save the object to a file each time?

@noonething if you're running this server-side (Node.js), you can use fs.writeFile to save the response object as a JSON file or something. If you're wanting to save to a file client-side, you can create a new Blob and have the user download that as a file.

Client-side:

new Blob(response.body, { type: 'application/json' });

@lauriestark
Copy link

I wanted to understand what some use cases would be for this, so I asked ChatGPT. I'm curious if its explanations were correct?

"The use case for the first implementation (fetchAndSaveJson) is to provide a custom fetch function that can save JSON data to a global object for later use. This can be useful when you need to fetch JSON data from multiple endpoints and want to save it for later use without having to refetch it. This implementation can be used in situations where you have control over the code and can modify the fetch calls to use the custom fetch function.

The use case for the second implementation (patching window.fetch) is to intercept all fetch calls made by existing code and modify the behavior of the fetch function. This can be useful when you want to add custom logic to every fetch call made in your application, such as logging, error handling, or modifying the response data. This implementation can be used in situations where you don't have control over the code that makes fetch calls, but want to modify the behavior of the fetch calls globally."

I then asked it to explain the use cases in more practical terms:

"Sure, here are some practical examples of use cases for each implementation:

fetchAndSaveJson
Suppose you are building a web application that fetches data from multiple endpoints to display various charts and tables on a dashboard. Each chart and table requires different data from different endpoints, but some of the data may overlap. To minimize the number of network requests and improve performance, you can use the fetchAndSaveJson function to fetch the data once and save it to the global fetchedData object. Then, when a chart or table needs to display the data, it can retrieve it from the fetchedData object instead of making a new network request. This can reduce the overall network traffic and improve the user experience by displaying the data more quickly.

patching window.fetch
Suppose you are working on a legacy web application that makes many fetch calls across various modules and components. The application has no error handling or logging built in, and you want to add these features without having to modify every fetch call in the codebase. You can use the second implementation to patch window.fetch and add custom error handling and logging to every fetch call made in the application. This can help you identify and diagnose issues more easily and improve the overall stability and reliability of the application. Additionally, you can modify the response data to fit your needs, such as by adding or removing fields, before passing it to the caller. This can help you customize the data to fit the specific requirements of your application."

@tedmiston
Copy link

I wanted to understand what some use cases would be for this, so I asked ChatGPT. I'm curious if its explanations were correct?

@lauriestark The author explains this himself in the blog post at https://simonwillison.net/2023/Mar/27/ai-enhanced-development/. The main use case is to create a system to archive conversations from ChatGPT.

But this post / gist were also created 2 months ago. IMO today it's easier to just use ChatGPT's built-in data export feature, although it is not real-time. In my experience, it's also oddly variable: sometimes the exports arrive in < 12 hours and sometimes they take nearly a week. No idea why that is; in the grand scheme of things it's a very tiny amount of data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment