Skip to content

Instantly share code, notes, and snippets.

@0xdevalias

0xdevalias/react-server-components-next-js-streaming-wire-format.md

Last active May 23, 2024 05:37
Show Gist options
  • Save 0xdevalias/ac465fb2f7e6fded183c2a4273d21e61 to your computer and use it in GitHub Desktop.
Save 0xdevalias/ac465fb2f7e6fded183c2a4273d21e61 to your computer and use it in GitHub Desktop.
Download ZIP
Some notes on React Server Components (specifically the streaming wire format), Next.js v13+ (specifically `__next_f`), and Webpack
Raw
react-server-components-next-js-streaming-wire-format.md

React Server Components, Next.js v13+, and Webpack: Notes on Streaming Wire Format (__next_f, etc)

Some notes on React Server Components (specifically the streaming wire format), Next.js v13+ (specifically __next_f), and Webpack.

Table of Contents

Notes

  • Figure a better way of extracting the main script URLs from the index.html/etc: 
    • const scriptTags = document.querySelectorAll('html script[src*="_next"]');
const scriptUrls = Array.from(scriptTags).map(tag => tag.src).sort()

// console.log(scriptUrls);

const joinedUrls = urls.join('\n')
console.log(joinedUrls);
copy(joinedUrls);
    • const scriptTags = document.querySelectorAll('html script[src*="_next"]');
const scriptUrls = Array.from(scriptTags).map(tag => tag.src).sort()
const joinedUrls = urls.join('\n')

// const missingIds = window.webpackChunk_N_E.map(([[id]]) => id).filter(id => !joinedUrls.includes(id)).sort()

// Note: See below code for how to calculate chunkMappings
const missingIds = 
  window.webpackChunk_N_E
    .map(([[id]]) => id)
    .filter(id => !Object.hasOwn(chunkMappings, id))
    .filter(id => !scriptUrls.some(url => url.includes(`chunks/${id}-`)))
    .sort()

const missingIdModules = window.webpackChunk_N_E.filter(([[id]]) => missingIds.includes(id))

console.log({ missingIds, chunkMappings, scriptUrls })

// If there are only 2 missingIds left, then one of them is probably yhe 'main-app-' chunk, and one is probably a chunk with a hashed name, which might be react or similar.
    • This seems to give us ~13 webpack script files at the moment; but window.webpackChunk_N_E.map(([[id]])=> id).sort() is giving us ~44 entries.. so need to figure out how to find and load the others statically..
  • Figure out how Udio handles the files, as it doesn't seem to use a buildManifest, instead seeming to use self.__next_f in the main index page
    • vercel/next.js#42170
    • https://www.reddit.com/r/nextjs/comments/173nx13/what_are_all_the_self_next_fpush_function_calls/
    • vercel/next.js#14779
    • vercel/next.js#56180 (comment)

      • those scripts are the initialization of the router state on the client. When next navigates from one page to another it does a client side transition (with a fetch to the server to get whatever new markup is required). Unlike in pages router App Router has shared layouts and the client is aware of these layouts and only fetches the part of the page that is not shared. The scripts you see here are the initialization of the client router state so that it can do these navigations. effectively. This includes preserving scroll position when hitting the back button and other aspects of the Router behavior.

    • There seems to be some code related to __next_f here:
    • The webpack file seems to just refer to a single .js file like this (and i'm not even sure if that gets used...): 
      • (d.u = function (e) {
  return "static/chunks/" + e + ".302361b2546aebf4.js";
}),
    • self.__next_f.map(f => f?.[1]).filter(f => f?.includes('static/'))
    • // const parseJSONFromEntry = entry => {
//   const jsonPart = entry.substring(entry.indexOf('[') + 1, entry.lastIndexOf(']'));
//   try {
//     return JSON.parse(`[${jsonPart}]`);
//   } catch (e) {
//     console.error("Failed to parse JSON for entry: ", entry);
//     return [];  // Return an empty array or null as per error handling choice
//   }
// };

// // Get build ID/etc
// const buildId = self.__next_f
//   .map(f => f?.[1])
//   .filter(f => f?.includes('buildId'))
//   .flatMap(f => f.trim().split('\n'))
//   .flatMap(parseJSONFromEntry)
//   .map(f => Array.isArray(f) ? f.flat() : f)
//   .map(f => f?.[3]?.buildId)
//   .filter(Boolean)?.[0]
    • // Source: https://gist.github.com/0xdevalias/ac465fb2f7e6fded183c2a4273d21e61#react-server-components-nextjs-v13-and-webpack-notes-on-streaming-wire-format-__next_f-etc

const parseJSONFromEntry = entry => {
  const jsonPart = entry.substring(entry.indexOf('[') + 1, entry.lastIndexOf(']'));
  try {
    return JSON.parse(`[${jsonPart}]`);
  } catch (e) {
    console.error("Failed to parse JSON for entry: ", entry);
    return [];  // Return an empty array or null as per error handling choice
  }
};

// Function to transform dependencies into a simpler, directly accessible format
function transformDependencies(dependencies) {
  return Object.values(dependencies).reduce((acc, currentDeps) => {
    Object.entries(currentDeps).forEach(([moduleId, path]) => {
      // If the paths match, skip to the next entry
      if (acc?.[moduleId] === path) return
      
      if (!acc[moduleId]) {
        // If this module ID has not been encountered yet, initialize it with the current path
        acc[moduleId] = path;
      } else if (typeof acc[moduleId] === 'string' && acc[moduleId] !== path) {
        // If the current path for this module ID is different from the existing one,
        // and the existing one is a string, transform it into an array containing both paths.
        const oldPath = acc[moduleId];
        acc[moduleId] = [oldPath, path];
      } else if (Array.isArray(acc[moduleId]) && !acc[moduleId].includes(path)) {
        // If the existing entry for this module ID is an array and does not already include the current path,
        // add the current path to the array.
        acc[moduleId].push(path);
      } else {
        // Log any unhandled cases for further investigation. This could be used to catch any unexpected data structures or duplicates.
        console.error('Unhandled case', { acc, currentDeps, moduleId, path });
      }
    });
    return acc;
  }, {});
}

// Get _next script urls
const scriptTags = document.querySelectorAll('html script[src*="_next"]');
const scriptUrls = Array.from(scriptTags).map(tag => tag.src).sort()
// console.log(scriptUrls);

// Get imports/etc (v3)
const moduleDependencies = 
  self.__next_f
    .map(f => f?.[1])
    .filter(f => f?.includes('static/'))
    .flatMap(f => f.split('\n'))
    .map(parseJSONFromEntry)
    .filter(f => Array.isArray(f) ? f.length > 0 : !!f)
    .map(f => {
      if (!Array.isArray(f?.[1])) { return f } else {
        // Convert alternating key/value array to an object
        const keyValueArray = f[1];
        const keyValuePairs = [];
        for (let i = 0; i < keyValueArray.length; i += 2) {
          keyValuePairs.push([keyValueArray[i], keyValueArray[i + 1]]);
        }
        f[1] = Object.fromEntries(keyValuePairs);
        return f;
      }
    })
    .filter(f => Array.isArray(f) && f.length === 3 && typeof f?.[1] === 'object')
    .reduce((acc, [moduleId, dependencies, _]) => {
      acc[moduleId] = dependencies;
      return acc;
    }, {});

const chunkMappings = transformDependencies(moduleDependencies)

const uniqueChunkPaths = Array.from(new Set(Object.values(chunkMappings))).sort()

const dynamicChunkUrls = uniqueChunkPaths
  .map(path => `https://www.udio.com/_next/${path}`)
  .sort()

const chunkUrls = Array.from(new Set([...scriptUrls, ...dynamicChunkUrls])).sort()

const chunkUrlsJoined = chunkUrls.join('\n')

const buildId = self.__next_f
  .map(f => f?.[1])
  .filter(f => f?.includes('buildId'))
  .flatMap(f => f.trim().split('\n'))
  .flatMap(parseJSONFromEntry)
  .map(f => Array.isArray(f) ? f.flat() : f)
  .map(f => f?.[3]?.buildId)
  .filter(Boolean)?.[0]

console.log({
  scriptUrls,
  moduleDependencies,
  chunkMappings,
  uniqueChunkPaths,
  dynamicChunkUrls,
  chunkUrls,
  buildId,
})
console.log(chunkUrlsJoined)
copy(chunkUrlsJoined)
console.log('Chunk URLs (joined) copied to clipboard')
    • // Get imports/etc (v2)

// const parseJSONFromEntry = entry => {
//   const jsonPart = entry.substring(entry.indexOf('[') + 1, entry.lastIndexOf(']'));
//   try {
//     return JSON.parse(`[${jsonPart}]`);
//   } catch (e) {
//     console.error("Failed to parse JSON for entry: ", entry);
//     return [];  // Return an empty array or null as per error handling choice
//   }
// };

// // Get imports/etc (v2)
// self.__next_f
//   .map(f => f?.[1])
//   .filter(f => f?.includes('static/'))
//   .flatMap(f => f.split('\n'))
//   .map(parseJSONFromEntry)
//   .filter(f => Array.isArray(f) ? f.length > 0 : !!f)
//   .map(f => {
//     if (!Array.isArray(f?.[1])) { return f } else {
//       // Convert alternating key/value array to an object
//       const keyValueArray = f[1];
//       const keyValuePairs = [];
//       for (let i = 0; i < keyValueArray.length; i += 2) {
//         keyValuePairs.push([keyValueArray[i], keyValueArray[i + 1]]);
//       }
//       f[1] = Object.fromEntries(keyValuePairs);
//       return f;
//     }
//   })
    • // Get imports/etc (v1)

// const parseJSONFromEntry = entry => {
//   const jsonPart = entry.substring(entry.indexOf('[') + 1, entry.lastIndexOf(']'));
//   try {
//     return JSON.parse(`[${jsonPart}]`);
//   } catch (e) {
//     console.error("Failed to parse JSON for entry: ", entry);
//     return [];  // Return an empty array or null as per error handling choice
//   }
// };

// // Get imports/etc (v1)
// self.__next_f
//   .map(f => f?.[1])
//   .filter(f => f?.includes('static/'))
//   .flatMap(f => f.split('\n'))
//   .map(parseJSONFromEntry)
//   .filter(f => Array.isArray(f) ? f.length > 0 : !!f)

See Also

Announcement Tweet

My Other Related Deepdive Gist's and Projects

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment