Some notes and tools for reverse engineering / deobfuscating / unminifying obfuscated web app code

_deobfuscating-unminifying-obfuscated-web-app-code.md

Deobfuscating / Unminifying Obfuscated Web App Code

Table of Contents

• PoC
• Tools
  • Unsorted
  • wakaru
  • webcrack
  • ast-grep
  • Restringer
  • debundle + related
  • joern
• Blogs / Articles / etc
• Libraries / Helpers
  • Unsorted
  • Recast + related
  • estools + related
  • Babel
  • semantic / tree-sitter + related
  • Shift AST
  • swc
  • esbuild
  • Source Maps
  • Visualisation/etc
• Browser Based Code Editors / IDEs
  • CodeMirror
  • monaco-editor
• Obfuscation / Deobfuscation
  • Variable Name Mangling
• Symbolic / Concolic Execution
• Profiling
• Unsorted
• My ChatGPT Research / Conversations
• See Also
  • My Other Related Deepdive Gist's and Projects

Other files in this gist:

• chrome-devtools-sources-extension.md

  • Chrome DevTools 'Sources' Extension

• fingerprinting-minified-javascript-libraries.md

  • Fingerprinting Minified JavaScript Libraries

PoC

• https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples

Tools

Unsorted

• https://eslint.org/docs/
  • https://eslint.org/docs/latest/extend/custom-rules#the-context-object

    • The context object is the only argument of the create method in a rule.

    • As the name implies, the context object contains information that is relevant to the context of the rule.

  • https://eslint.org/docs/latest/extend/custom-rules#applying-fixes

    • Applying Fixes If you’d like ESLint to attempt to fix the problem you’re reporting, you can do so by specifying the fix function when using context.report(). The fix function receives a single argument, a fixer object, that you can use to apply a fix.

    • Important: The meta.fixable property is mandatory for fixable rules. ESLint will throw an error if a rule that implements fix functions does not export the meta.fixable property.

    • The fixer object has the following methods:

      • insertTextAfter(nodeOrToken, text): Insert text after the given node or token.
      • insertTextAfterRange(range, text): Insert text after the given range.
      • insertTextBefore(nodeOrToken, text): Insert text before the given node or token.
      • insertTextBeforeRange(range, text): Insert text before the given range.
      • remove(nodeOrToken): Remove the given node or token.
      • removeRange(range): Remove text in the given range.
      • replaceText(nodeOrToken, text): Replace the text in the given node or token.
      • replaceTextRange(range, text): Replace the text in the given range.

      A range is a two-item array containing character indices inside the source code. The first item is the start of the range (inclusive) and the second item is the end of the range (exclusive). Every node and token has a range property to identify the source code range they represent.

      The above methods return a fixing object. The fix() function can return the following values:

      • A fixing object.
      • An array which includes fixing objects.
      • An iterable object which enumerates fixing objects. Especially, the fix() function can be a generator.

      If you make a fix() function which returns multiple fixing objects, those fixing objects must not overlap.

  • https://eslint.org/docs/latest/extend/code-path-analysis

    • Code Path Analysis Details

    • ESLint’s rules can use code paths. The code path is execution routes of programs. It forks/joins at such as if statements.

    • Program is expressed with several code paths. A code path is expressed with objects of two kinds: CodePath and CodePathSegment.

    • CodePath expresses whole of one code path. This object exists for each function and the global. This has references of both the initial segment and the final segments of a code path.

    • CodePathSegment is a part of a code path. A code path is expressed with plural CodePathSegment objects, it’s similar to doubly linked list. Difference from doubly linked list is what there are forking and merging (the next/prev are plural).

    • There are seven events related to code paths, and you can define event handlers by adding them alongside node visitors in the object exported from the create() method of your rule.

• https://prettier.io/
  • https://github.com/prettier/prettier

    • Prettier is an opinionated code formatter. It enforces a consistent style by parsing your code and re-printing it with its own rules that take the maximum line length into account, wrapping code when necessary.

    • https://prettier.io/docs/en/options#parser

      • Parser Specify which parser to use. Prettier automatically infers the parser from the input file path, so you shouldn’t have to change this setting.

    • https://prettier.io/docs/en/api.html

      • API If you want to run Prettier programmatically, check this page out.

• https://github.com/beautify-web/js-beautify

  • Beautifier for javascript

  • This little beautifier will reformat and re-indent bookmarklets, ugly JavaScript, unpack scripts packed by Dean Edward’s popular packer, as well as partly deobfuscate scripts processed by the npm package javascript-obfuscator.

  • https://beautifier.io/
• https://github.com/shapesecurity/unminify

  • Reverse many of the transformations applied by minifiers and naïve obfuscators

  • https://github.com/shapesecurity/unminify/#safety-levels
  • https://unminify.io/
• https://github.com/lelinhtinh/de4js

  • JavaScript Deobfuscator and Unpacker

  • https://lelinhtinh.github.io/de4js/
  • https://github.com/lelinhtinh/de4js/blob/master/userscript/de4js_helper.user.js
• http://www.jsnice.org/

  • Statistical renaming, type inference and deobfuscation

  • https://www.sri.inf.ethz.ch/research/plml

    • Machine Learning for Code This project combines programming languages and machine learning for building statistical programming engines -- systems built on top of machine learning models of large codebases. These are new kinds of engines which can provide statistically likely solutions to problems that are difficult or impossible to solve with traditional techniques.

    • JSNice JSNice de-obfuscates JavaScript programs. JSNice is a popular system in the JavaScript commmunity used by tens of thousands of programmers, worldwide

• https://github.com/spaceraccoon/webpack-exploder/

  • Unpack the source code of React and other Webpacked apps!

  • https://github.com/spaceraccoon/webpack-exploder/blob/master/index.html#L225-L286
    • This basically just extracts the original files from a sourcemap *.map file
  • https://spaceraccoon.github.io/webpack-exploder/
• https://github.com/goto-bus-stop/webpack-unpack

  • extract modules from a bundle generated by webpack

  • https://github.com/goto-bus-stop/webpack-unpack/blob/master/index.js
• https://github.com/goto-bus-stop/amd-unpack

  • extract modules from a bundled AMD project using define/require functions

• https://github.com/gchq/CyberChef

  • The Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis

  • https://gchq.github.io/CyberChef/
    • Javascrpt Parser: https://gchq.github.io/CyberChef/#recipe=JavaScript_Parser(false,false,false,false,false)
• https://github.com/dandavison/delta

  • A syntax-highlighting pager for git, diff, and grep output

  • (the package is called "git-delta" in most package managers, but the executable is just delta)

  • https://dandavison.github.io/delta/introduction.html
• https://github.com/Wilfred/difftastic

  • Difftastic is a structural diff tool that compares files based on their syntax.

  • https://difftastic.wilfred.me.uk/introduction.html

    • Difftastic is a structural diff tool that understands syntax. It supports over 30 programming languages and when it works, it's fantastic.

• https://github.com/prettydiff/prettydiff

  • Beautifier and language aware code comparison tool for many languages. It also minifies and a few other things

  • https://prettydiff.com/#projects-prettydiff

    • When I first became a developer at Travelocity I would sometimes needs to compare code in different environments where some code existed in its original condition and in other cases was minified. Existing diff tools could not solve for that sort of comparison, and at that time existing JavaScript beautifiers had trouble with complex data structures. So I integrated a web-based diff tool with an existing beautifier and minifier. As the features, capabilities, and requests upon the application grew I eventually wrote my own diff algorithm and beautifiers for the various supported languages.

• https://github.com/Vunovati/astii

  • A JavaScript AST-aware diff and patch toolset

  • When comparing two JavaScript files, standard diff tools compare the two files line-by-line and output the lines on which the files differ. This tool does not compare the characters of the source files directly but their abstract representation - their abstract syntax trees.

  • This enables you to have more meaningfull diffs between files which may be very simmilar but have different source code formatting.

  • When patching, astii patch will regenerate (original --> AST --> generate) the source file and patch it with the provided diff.

• https://www.blueclosure.com/product/bc-detect

  • BC Detect (DOMinatorPro NG) helps security testers to analyse and automatically discover DOM Based Cross Site Scripting issues thanks to its Hybrid IAST Engine together with the Smart Fuzzer module.

  • https://www.blueclosure.com/page/features
• https://github.com/kuizuo/js-deobfuscator

  • JS obfuscated code restoration Let confusion no longer be a stumbling block in reverse analysis https://js-deobfuscator.vercel.app/

wakaru

• https://github.com/pionxzh/wakaru

  • Javascript decompiler, unpacker and unminify toolkit

  • Wakaru is the Javascript decompiler for modern frontend. It brings back the original code from a bundled and transpiled source.

  • https://wakaru.vercel.app/
  • https://github.com/pionxzh/wakaru/blob/main/packages/unpacker/src/unpack.ts
    • Uses jscodeshift.withParser('babylon')
    • First checks for webpack with getModulesFromWebpack, then tries getModulesFromBrowserify, and finally falls back to a single module if those both failed
    • https://github.com/pionxzh/wakaru/blob/b06d0a7d8682042700398f0bbba1d76839fb57cf/packages/playground/src/pages/Uploader.vue#L80-L158
      • async function startUnpack(code: string)

        • // TODO: Move to worker
          const { modules, moduleIdMapping } = unpack(code)
          const unpackedModules = modules.map<TransformedModule>((module) => {
              const { id, isEntry, code, tags } = module
              return {
                  id,
                  isEntry,
                  code,
                  transformed: code,
                  import: module.import,
                  export: module.export,
                  tags,
              }
          })

  • https://github.com/pionxzh/wakaru/tree/main/packages/unpacker/src/extractors
    • https://github.com/pionxzh/wakaru/blob/main/packages/unpacker/src/extractors/browserify/index.ts
    • https://github.com/pionxzh/wakaru/tree/main/packages/unpacker/src/extractors/webpack
  • https://github.com/pionxzh/wakaru/tree/main/packages/unminify

    • @wakaru/unminify This package offers a comprehensive set of transformation rules designed to unminify and enhance the readability of code.

      It covered most of patterns that are used by the following tools:

      • Terser (Check the Terser Progress)
      • Babel (Check the Babel Progress)
      • SWC (Check the SWC Progress)
      • TypeScript

    • https://github.com/pionxzh/wakaru/tree/main/packages/unminify#smart-rename

      • Rename minified identifiers with heuristic rules.

    • https://github.com/pionxzh/wakaru/blob/main/packages/unminify/src/transformations/smart-rename.ts
      • handleDestructuringRename, handleFunctionParamsRename, handlePropertyRename, handleReactRename, getElementName
    • https://github.com/pionxzh/wakaru/blob/main/packages/unminify/src/utils/identifier.ts#L28-L75
      • generateName, getUniqueName
  • pionxzh/wakaru#32

    • Scoping issue

    • The current identifier renaming is not 100% accurate. By inspecting the unpacking snapshot, you can tell that some variable was wrongly renamed to export or require during the unpacking process. Mostly because ast-types are giving us wrong scope information, and it's no longer well-maintained. We need to either patch it or find an alternative.

  • pionxzh/wakaru#34

    • support un-mangle identifiers

    • Hi, currently we don't have the un-mangle feature yet. You can use other tools for the first pass, and let wakaru handle syntax-related unminification. I will transform this issue to a feature request, so that we can track the progress of it here.

    • For now, we have smart-rename that can guess the variable name based on the context. I would like to expand it to cover some other generic cases.

  • pionxzh/wakaru#36

    • jsx formats that cannot be handled

  • pionxzh/wakaru#35

    • Split code and save progress?

  • pionxzh/wakaru#37

    • Add CLI tool

  • pionxzh/wakaru#45

    • wakaru IDE

webcrack

• https://github.com/j4k0xb/webcrack

  • Deobfuscate obfuscator.io, unminify and unpack bundled javascript

  • https://webcrack.netlify.app/ (Playground)
    • https://webcrack.netlify.app/docs/
    • https://webcrack.netlify.app/docs/guide/introduction.html
  • https://github.com/j4k0xb/webcrack/tree/master/src/transforms
  • https://github.com/j4k0xb/webcrack/tree/master/src/deobfuscator
  • j4k0xb/webcrack#3

    • Awesome project - looking for help?

    • I have a project idea I'd like to discuss around unmangling variable names actually, that I think you'll find very interesting as well. Let me know if you'd like to discuss more and we could have a call I'll show you a POC for that next week so you can let me know your thoughts.

  • j4k0xb/webcrack#21

    • rename short identifiers

  • j4k0xb/webcrack#10

    • hardcoded React UMD global

    • The current Sketchy implementation only decompiles React JSX when the code utilizes the UMD global, which is not effective since the majority of React websites incorporate the library within their bundle.

      To make the decompilation process more effective and adaptable to different React websites, I recommend a more dynamic approach by identifying the React library being used in the compiled code, instead of hardcoding the use of 'React'. This can possibly be achieved by finding the variable name assigned to the React library and using that in the matchers.

  • j4k0xb/webcrack#6

    • (0, fn)(...args) type of calls

  • j4k0xb/webcrack#24

    • optimisation ideas

    • There are some js to c/wasm transpilers but

  • https://github.com/e9x/krunker-decompiler

    • Krunker Decompiler

    • Powered by webcrack

    • https://github.com/e9x/krunker-decompiler/blob/master/src/libDecompile.ts
    • https://github.com/e9x/krunker-decompiler/blob/master/src/libRenameVars.ts

ast-grep

• https://github.com/ast-grep/ast-grep

  • A CLI tool for code structural search, lint and rewriting. Written in Rust

  • ast-grep is a AST-based tool to search code by pattern code. Think it as your old-friend grep but it matches AST nodes instead of text. You can write patterns as if you are writing ordinary code. It will match all code that has the same syntactical structure. You can use $ sign + upper case letters as wildcard, e.g. $MATCH, to match any single AST node. Think it as REGEX dot ., except it is not textual.

  • https://github.com/ast-grep/ast-grep/tree/main/npm

    • @ast-grep/cli ast-grep(sg) is a CLI tool for code structural search, lint, and rewriting.

    • https://github.com/ast-grep/ast-grep/tree/main/crates
    • https://github.com/ast-grep/ast-grep/blob/main/benches
      • https://github.com/ast-grep/ast-grep/blob/main/benches/bench.ts#L37-L101
  • https://ast-grep.github.io/
  • https://ast-grep.github.io/playground.html
    • https://ast-grep.github.io/reference/playground.html

      • ast-grep Playground Manual The ast-grep playground is an online tool that allows you to try out ast-grep without installing anything on your machine. You can write code patterns and see how they match your code in real time. You can also apply rewrite rules to modify your code based on the patterns.

        The playground is a great way to learn ast-grep, debug patterns/rules, report bugs and showcase ast-grep's capabilities.

  • https://ast-grep.github.io/guide/introduction.html
  • https://ast-grep.github.io/reference/languages.html

    • List of Languages with Built-in Support

  • https://ast-grep.github.io/reference/cli.html
  • https://ast-grep.github.io/reference/api.html

    • API Reference ast-grep currently has an experimental API for Node.js

  • https://ast-grep.github.io/guide/tooling-overview.html#editor-integration
    • https://marketplace.visualstudio.com/items?itemName=ast-grep.ast-grep-vscode&ssr=false#overview

      • ast-grep-vscode VSCode extension package for ast-grep language server

  • ast-grep/ast-grep#80

    • Using ast-grep as a library

    • https://ast-grep.github.io/guide/api-usage/py-api.html

      • Python API ast-grep's Python API is powered by PyO3. You can write Python to programmatically inspect and change syntax trees.

  • ast-grep/ast-grep#524

    • More powerful tree-sitter analysis

      • https://github.com/microsoft/vscode-anycode

        • Anycode A Tree-sitter-based language extension that inaccurately implements popular features like "Outline & Breadcrumbs", "Go to Symbol in Workspace", "Document Highlights" and more. This extension should be used when running in enviroments that don't allow for running actual language services, like https://github.dev or https://vscode.dev.

          • https://code.visualstudio.com/docs/editor/editingevolved#_go-to-symbol
          • https://microsoft.github.io/monaco-editor/typedoc/interfaces/languages.DocumentSymbolProvider.html
  • ast-grep/ast-grep#334

    • [pipedream] Add control flow / data dependency operators to ast-grep

    • inspired by docs.joern.io/code-property-graph, or more specifically, docs.joern.io/cpgql/data-flow-steps

    • State of Art:

      • arxiv.org/pdf/2208.07461.pdf

        • parser: tree-sitter
        • cfg/data flow: built for Python

      • nickgregory.me/post/2022/07/02/go-code-as-a-graph

        • kallsyms/go-graph
        • parser: go
        • cfg/dataflow: golangx/tool
        • golang team's tooling is crazy...

      • docs.rs/tree-sitter-graph/0.10.4/tree_sitter_graph/reference

        • parser: tree-sitter
        • cfg/data flow: N/A. It only supports graph construction.

      • joern

        • parser: graal based. e.g. graalJS or custom parser generator
        • e.g. Python joernio/joern@7e66155/joern-cli/frontends/pysrc2cpg/pythonGrammar.jj#L10
        • cfg: based on a generic-ast. joernio/joern@7e66155/joern-cli/frontends/x2cpg/src/main/scala/io/joern/x2cpg

      Not related but worth looking

      • eslint: eslint/eslint@main/lib/linter/code-path-analysis/code-path-analyzer.js
      • oxc: Boshen/oxc@main/crates/oxc_semantic2/src/reference.rs
      • ruff: astral-sh/ruff@main/crates/ruff_python_semantic/src

  • https://www.reddit.com/r/rust/comments/13eg738/meet_astgrep_a_rustbased_tool_for_code_searching/

Restringer

• https://github.com/PerimeterX/restringer

  • Restringer

  • Deobfuscate Javascript and reconstruct strings. Simplify cumbersome logic where possible while adhering to scope limitations.

  • https://restringer.tech/ (Playground)
  • https://github.com/PerimeterX/restringer/tree/main/src/processors

    • Processors Processors are a collection of methods meant to prepare the script for obfuscation, removing anti-debugging traps and performing any required modifications before (preprocessors) or after (postprocessors) the main deobfuscation process.

      The processors are created when necessary and are lazily loaded when a specific obfuscation type was detected which requires these additional processes.

  • https://github.com/PerimeterX/restringer/tree/main/src/modules/safe
  • https://github.com/PerimeterX/restringer/tree/main/src/modules/unsafe
  • https://github.com/PerimeterX/obfuscation-detector

    • Obfuscation Detector

    • Detect different types of JS obfuscation by their AST structure.

  • https://github.com/PerimeterX/flast

    • flAST - FLat Abstract Syntax Tree Flatten an Abstract Syntax Tree by placing all the nodes in a single flat array.

    • Provides a flat Abstract Syntax Tree and an Arborist to trim and modify the tree

debundle + related

• https://github.com/1egoman/debundle

  • A javascript debundler. Takes a Browserify or Webpack bundle and recreates the initial, pre-bundled source

  • https://github.com/1egoman/debundle/blob/master/DOCS.md

    • Configuration Documentation Configuration is stored in a json file with an object at its root. The headings below represent the keys in that object

  • 1egoman/debundle#23

    • Potential reownership?

      • I actually recently built a better version of this same tool (a much more reliable, version that requires much less configuration)

      • I'll push up this new version to a separate, orphan branch on this repository.

      • Here's that v2 branch: https://github.com/1egoman/debundle/tree/v2

        • Debundle, V2 I needed to somewhat recently reverse engineer a few webpack bundles. I rebuilt debundle to be a bit easier to use and to do a few more things automatically. I give no support to this code right now - it's not being published on npm, any issues will be closed, etc.

          However, I think, it's a much more streamlined way of doing things.

  • 1egoman/debundle#27

    • Support for webpackJsonp

  • https://github.com/scil/reliable-debundle

    • A javascript debundler. Takes a Browserify or Webpack bundle and recreates the initial, pre-bundled source

    • forked from 1egoman/debundle

    • This branch is 89 commits ahead of 1egoman/debundle:master

      • https://github.com/1egoman/debundle/compare/master...scil:reliable-debundle:master
        • https://github.com/hectorqin/debundle

          • Modified from https://github.com/1egoman/debundle

            The dependencies have been upgraded, some compatibility issues have been corrected, and there are other bugs, but it works and supports es6 syntax. The completeness and accuracy of the split files are not guaranteed

  • https://github.com/TheFireBlast/debundle

    • A javascript debundler. Takes a Webpack bundle and recreates the initial, pre-bundled source.

    • forked from 1egoman/debundle

    • This branch is 26 commits ahead of 1egoman/debundle:master

      • https://github.com/1egoman/debundle/compare/master...TheFireBlast:debundle:master
• https://github.com/topics/debundle
  • https://github.com/Xmader/retidy

    • Extract, unminify, and beautify ("retidy") each file from a webpack/parcel bundle (JavaScript reverse engineering)

  • https://github.com/nickw444/rn-debundle

    • A very basic debundler for Javascript bundles compiled with React Native's bundler.

      Debundles a large React Native bundle by walking the compiled AST and extracts individual module declarations and writes them to their own modules & attempts to resolve dependeny import relationships.

joern

• https://joern.io/

  • The Bug Hunter's Workbench

  • Query: Uncover attack surface, sloppy coding practices, and variants of known vulnerabilities using an interactive code analysis shell. Joern supports C, C++, LLVM bitcode, x86 binaries (via Ghidra), JVM bytecode (via Soot), and Javascript. Python, Java source code, Kotlin, and PHP support coming soon.

  • Automate: Wrap your queries into custom code scanners and share them with the community or run existing Joern-based scanners in your CI.

  • Integrate: Use Joern as a library to power your own code analysis tools or as a component via the REST API.

  • https://github.com/joernio/joern

    • Open-source code analysis platform for C/C++/Java/Binary/Javascript/Python/Kotlin based on code property graphs.

    • Joern is a platform for analyzing source code, bytecode, and binary executables. It generates code property graphs (CPGs), a graph representation of code for cross-language code analysis. Code property graphs are stored in a custom graph database. This allows code to be mined using search queries formulated in a Scala-based domain-specific query language. Joern is developed with the goal of providing a useful tool for vulnerability discovery and research in static program analysis.

  • https://docs.joern.io/

    • Joern is a platform for robust analysis of source code, bytecode, and binary code. It generates code property graphs, a graph representation of code for cross-language code analysis. Code property graphs are stored in a custom graph database. This allows code to be mined using search queries formulated in a Scala-based domain-specific query language. Joern is developed with the goal of providing a useful tool for vulnerability discovery and research in static program analysis.

    • The core features of Joern are:

      • Robust parsing. Joern allows importing code even if a working build environment cannot be supplied or parts of the code are missing.
      • Code Property Graphs. Joern creates semantic code property graphs from the fuzzy parser output and stores them in an in-memory graph database. SCPGs are a language-agnostic intermediate representation of code designed for query-based code analysis.
      • Taint Analysis. Joern provides a taint-analysis engine that allows the propagation of attacker-controlled data in the code to be analyzed statically.
      • Search Queries. Joern offers a strongly-typed Scala-based extensible query language for code analysis based on Gremlin-Scala. This language can be used to manually formulate search queries for vulnerabilities as well as automatically infer them using machine learning techniques.
      • Extendable via CPG passes. Code property graphs are multi-layered, offering information about code on different levels of abstraction. Joern comes with many default passes, but also allows users to add passes to include additional information in the graph, and extend the query language accordingly.

    • https://docs.joern.io/code-property-graph/
    • https://docs.joern.io/cpgql/data-flow-steps/
    • https://docs.joern.io/export/

      • Joern can create the following graph representations for C/C++ code:

        • Abstract Syntax Trees (AST)
        • Control Flow Graphs (CFG)
        • Control Dependence Graphs (CDG)
        • Data Dependence Graphs (DDG)
        • Program Dependence graphs (PDG)
        • Code Property Graphs (CPG14)
        • Entire graph, i.e. convert to a different graph format (ALL)

Blogs / Articles / etc

• https://thejunkland.com/blog/using-llms-to-reverse-javascript-minification.html

  • Using LLMs to reverse JavaScript variable name minification

  • This blog introduces a novel way to reverse minified Javascript using large language models (LLMs) like ChatGPT and llama2 while keeping the code semantically intact. The code is open source and available at Github project Humanify

    • https://github.com/jehna/humanify

      • Un-minify Javascript code using ChatGPT

      • This tool uses large language modeles (like ChatGPT & llama2) and other tools to un-minify Javascript code. Note that LLMs don't perform any structural changes – they only provide hints to rename variables and functions. The heavy lifting is done by Babel on AST level to ensure code stays 1-1 equivalent.

      • jehna/humanify#3

        • Consider using pionxzh/wakaru instead of/alongside webcrack

      • https://github.com/jehna/humanify/blob/main/src/index.ts
      • https://github.com/jehna/humanify/blob/main/src/humanify.ts
      • https://github.com/jehna/humanify/blob/main/src/openai/openai.ts#L28-L82
      • https://github.com/jehna/humanify/blob/main/src/openai/rename-variables-and-functions.ts#L9-L26
      • https://github.com/jehna/humanify/blob/main/src/openai/is-reserved-word.ts1
      • https://github.com/jehna/humanify/blob/main/src/local-rename.ts
        • https://github.com/jehna/humanify/blob/main/src/mq.ts
        • https://github.com/jehna/humanify/tree/main/local-inference
        • https://github.com/jehna/humanify/blob/main/local-inference/inference-server.py
        • https://github.com/jehna/humanify/blob/main/local-inference/rename.py
• https://blog.apify.com/chatgpt-reverse-engineer-code/

  • Unlocking JavaScript secrets: reverse engineering code with ChatGPT

• https://book.hacktricks.xyz/network-services-pentesting/pentesting-web/code-review-tools#static-analysis
  • HackTricks-wiki/hacktricks#743

    • Some useful tool additions for 'Code Review Tools -> JavaScript -> Static Analysis -> Deobfuscate/Unpack'

• https://www.digitalocean.com/community/tutorials/js-traversing-ast
• https://medium.com/@yuexing0921/a-brief-introduction-of-various-javascript-parsers-103e32c4d7d2
• https://jotadeveloper.medium.com/abstract-syntax-trees-on-javascript-534e33361fc7
• https://infosecwriteups.com/javascript-parser-to-create-abstract-syntax-tree-ast-acorn-be9bbfe91bed
• https://itnext.io/ast-for-javascript-developers-3e79aeb08343
• https://steakenthusiast.github.io/

  • Learning Reverse Engineering by Example

  • https://steakenthusiast.github.io/2022/05/21/Deobfuscating-Javascript-via-AST-An-Introduction-to-Babel/

    • An Introduction to Javascript Obfuscation & Babel

  • https://steakenthusiast.github.io/2022/05/22/Deobfuscating-Javascript-via-AST-Manipulation-Various-String-Concealing-Techniques/

    • Deobfuscating Javascript via AST: Reversing Various String Concealing Techniques

  • https://steakenthusiast.github.io/2022/05/28/Deobfuscating-Javascript-via-AST-Manipulation-Converting-Bracket-Notation-Dot-Notation-for-Property-Accessors/

    • Deobfuscating Javascript via AST: Converting Bracket Notation => Dot Notation for Property Accessors

  • https://steakenthusiast.github.io/2022/05/28/Deobfuscating-Javascript-via-AST-Manipulation-Constant-Folding/

    • Deobfuscating Javascript via AST: Constant Folding/Binary Expression Simplification

  • https://steakenthusiast.github.io/2022/05/31/Deobfuscating-Javascript-via-AST-Replacing-References-to-Constant-Variables-with-Their-Actual-Value/

    • Deobfuscating Javascript via AST: Replacing References to Constant Variables with Their Actual Value

  • https://steakenthusiast.github.io/2022/06/04/Deobfuscating-Javascript-via-AST-Removing-Dead-or-Unreachable-Code/

    • Deobfuscating Javascript via AST: Removing Dead or Unreachable Code

  • https://steakenthusiast.github.io/2022/06/14/Deobfuscating-Javascript-via-AST-Deobfuscating-a-Peculiar-JSFuck-style-Case/

    • Deobfuscating Javascript via AST: A Peculiar JSFuck-esque Case

• https://www.trickster.dev/post/

  • Trickster Dev: Code level discussion of web scraping, gray hat automation, growth hacking and bounty hunting

  • https://www.trickster.dev/post/understanding-abstract-syntax-trees/

    • Understanding Abstract Syntax Trees

  • https://www.trickster.dev/post/javascript-obfuscation-techniques-by-example/

    • Javascript obfuscation techniques by example

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-the-first-steps/

    • JavaScript AST manipulation with Babel: the first steps

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-extracting-hardcoded-data/

    • JavaScript AST manipulation with Babel: extracting hardcoded data

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-removing-unreachable-code/

    • JavaScript AST manipulation with Babel: removing unreachable code

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-defeating-string-array-mapping/

    • JavaScript AST manipulation with Babel: defeating string array mapping

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-transform-prototyping-and-plugin-development/

    • JavaScript AST manipulation with Babel: transform prototyping and plugin development

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-3-ways-to-create-nodes-and-subtrees/

    • JavaScript AST manipulation with Babel: 3 ways to create nodes and subtrees

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-untangling-scope-confusion/

    • JavaScript AST manipulation with Babel: untangling scope confusion

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-ast-modification-apis/

    • JavaScript AST manipulation with Babel: AST modification APIs

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-constant-folding-and-propagation/

    • JavaScript AST manipulation with Babel: constant folding and propagation

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-reducing-indirection-undoing-string-concealing/

    • JavaScript AST manipulation with Babel: reducing indirection, undoing string concealing

  • https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-reducing-nestedness-unflattening-the-cfg/

    • JavaScript AST manipulation with Babel: reducing nestedness, unflattening the CFG

  • https://www.trickster.dev/post/dont-jsfuck-with-me-part-1/

    • Don’t JSFuck with me: Part 1

  • https://www.trickster.dev/post/dont-jsfuck-with-me-part-2/

    • Don’t JSFuck with me: Part 2

  • https://www.trickster.dev/post/dont-jsfuck-with-me-part-3/

    • Don’t JSFuck with me: Part 3

  • https://www.trickster.dev/post/understanding-javascript-packers/

    • Understanding JavaScript packers

  • https://www.trickster.dev/post/self-defending-js-code-and-debugger-traps/

    • Self-defending JS code and debugger traps

  • https://www.trickster.dev/post/restringer-modular-javascript-deobfuscator/

    • Restringer: modular JavaScript deobfuscator

  • https://www.trickster.dev/post/solving-a-simple-js-challenge-with-sandboxing/

    • Solving a simple JS challenge with sandboxing

• https://raz0r.name/articles/using-codeql-to-detect-client-side-vulnerabilities-in-web-applications/

  • Using CodeQL to detect client-side vulnerabilities in web applications

Libraries / Helpers

Unsorted

• https://astexplorer.net/
  • https://github.com/fkling/astexplorer

    • A web tool to explore the ASTs generated by various parsers.

    • https://github.com/fkling/astexplorer/tree/master/website
      • Seems to be written in React, which is nice!
    • fkling/astexplorer#625

      • State of the Project

    • fkling/astexplorer#304

      • Path to astexplorer v3

    • fkling/astexplorer#70

      • Make astexplorer standalone / embeddable

    • fkling/astexplorer#646

      • Any plans to encapsulate parsers as NPM package?

    • fkling/astexplorer#510

      • Feature request: Tree of JavaScript scopes (eslint-scope)

    • fkling/astexplorer#310

      • Show JSON path

    • fkling/astexplorer#688

      • Add support for semantic / tree-sitter

      • fkling/astexplorer#622

        • feat: web-tree-sitter

• https://github.com/sxzz/ast-explorer

  • AST Explorer - For most popular front-end languages and parsers

  • https://ast.sxzz.moe/
• https://github.com/dsherret/ts-ast-viewer

  • TypeScript AST viewer

  • https://ts-ast-viewer.com/
• https://github.com/rajasegar/ast-builder

  • Build your ASTs directly from code

  • Build your Abstract Syntax Trees (AST) directly by writing code. Simply type in your code and get the right jscodeshift api to build your AST.

  • WARNING: This repository is moved here https://github.com/rajasegar/ast-tooling/tree/master/apps/ast-builder

  • https://github.com/rajasegar/ast-tooling/tree/master/packages/ast-node-builder
  • https://www.hangaroundtheweb.com/posts/ast-builder-building-ast-nodes-from-code/

    • AST Builder - Building AST nodes from code

• https://github.com/estree/estree

  • The ESTree Spec

• https://github.com/jquery/esprima

  • ECMAScript parsing infrastructure for multipurpose analysis

• https://github.com/eslint/espree

  • An Esprima-compatible JavaScript parser

  • Espree started out as a fork of Esprima v1.2.2, the last stable published released of Esprima before work on ECMAScript 6 began. Espree is now built on top of Acorn, which has a modular architecture that allows extension of core functionality. The goal of Espree is to produce output that is similar to Esprima with a similar API so that it can be used in place of Esprima.

  • The primary goal is to produce the exact same AST structure and tokens as Esprima, and that takes precedence over anything else. (The AST structure being the ESTree API with JSX extensions.) Separate from that, Espree may deviate from what Esprima outputs in terms of where and how comments are attached, as well as what additional information is available on AST nodes. That is to say, Espree may add more things to the AST nodes than Esprima does but the overall AST structure produced will be the same.

    Espree may also deviate from Esprima in the interface it exposes.

• https://github.com/eslint/eslint-scope

  • eslint-scope: ECMAScript scope analyzer

  • ESLint Scope is the ECMAScript scope analyzer used in ESLint. It is a fork of escope.

  • https://github.com/MarcosNASA/telEScope

    • TelEScope.js is the first part of the «3 pilars project» inspired by Kyle Simpson

    • Parses scopes (from eslint-scope) into a data structure that allows building scope chain visualization tools

      • https://github.com/MarcosNASA/bubbl.es

        • Bubbl.es is the first part of the «3 pilars project» inspired by Kyle Simpson

        • Visualize the JS scope chain as colored bubbles. Powered by telEScope

        • https://jsbubbl.es/
        • https://jsbubbl.es/theory
        • https://jsbubbl.es/bubbles
• https://github.com/acornjs/acorn

  • A tiny, fast JavaScript parser, written completely in JavaScript

• https://github.com/acornjs/acorn-jsx

  • Alternative, faster React.js JSX parser

• https://github.com/davidbonnet/astring

  • Tiny and fast JavaScript code generator from an ESTree-compliant AST

• https://github.com/goto-bus-stop/estree-assign-parent

  • assign .parent properties to all nodes in an AST

• https://github.com/goto-bus-stop/scope-analyzer

  • simple scope analysis for javascript ASTs. tracks scopes and collects references to variables.

• https://github.com/afiore/arboreal

  • Javascript tree-traversal and manipulation microlibrary

  • This repository has been archived by the owner on Apr 4, 2020. It is now read-only.

  • https://github.com/shonzilla/arboreal
    • https://github.com/afiore/arboreal/compare/master...shonzilla:arboreal:master
• https://github.com/coderaiser/putout

  • Pluggable and configurable JavaScript Linter and code transformer with built-in ESLint and Babel support for js, jsx typescript, flow, markdown, yaml and json. Write declarative codemods in a simplest possible way

  • Putout is JavaScript Linter, pluggable and configurable code transformer based on Babel with built-in ESLint. It has a lot of transformations that keeps your codebase in a clean state, removing any code smell and making code readable according to best practices.

  • https://github.com/coderaiser/putout#-built-in-transformations
• https://codsen.com/os/#ast-libraries

  • AST Libraries

    • ast-compare: Compare anything: AST, objects, arrays, strings and nested thereof
      • https://github.com/codsen/codsen/tree/main/packages/ast-compare
    • ast-contains-only-empty-space: Does AST contain only empty space?
      • https://github.com/codsen/codsen/tree/main/packages/ast-contains-only-empty-space
    • ast-deep-contains: Like t.same assert on array of objects, where element order doesn’t matter.
      • https://github.com/codsen/codsen/tree/main/packages/ast-deep-contains
    • ast-delete-object: Delete all plain objects in AST if they contain a certain key/value pair
      • https://github.com/codsen/codsen/tree/main/packages/ast-delete-object
    • ast-get-object: Getter/setter for nested parsed HTML AST’s, querying objects by key/value pairs
      • https://github.com/codsen/codsen/tree/main/packages/ast-get-object
    • ast-get-values-by-key: Extract values and paths from AST by keys OR set them by keys
      • https://github.com/codsen/codsen/tree/main/packages/ast-get-values-by-key
    • ast-is-empty: Find out, is nested array/object/string/AST tree is empty
      • https://github.com/codsen/codsen/tree/main/packages/ast-is-empty
    • ast-loose-compare: Compare anything: AST, objects, arrays and strings
      • https://github.com/codsen/codsen/tree/main/packages/ast-loose-compare
    • ast-monkey: Traverse and edit AST
      • https://github.com/codsen/codsen/tree/main/packages/ast-monkey
    • ast-monkey-traverse: Utility library to traverse AST
      • https://github.com/codsen/codsen/tree/main/packages/ast-monkey-traverse
    • ast-monkey-traverse-with-lookahead: Utility library to traverse AST, reports upcoming values
      • https://github.com/codsen/codsen/tree/main/packages/ast-monkey-traverse-with-lookahead
    • ast-monkey-util: Utility library of AST helper functions
      • https://github.com/codsen/codsen/tree/main/packages/ast-monkey-util

  • https://github.com/codsen/codsen

    • a monorepo of npm packages

    • Please visit codsen.com for an overview and full documentation of all packages

    • https://github.com/codsen/codsen/tree/main/packages
• https://github.com/scottrogowski/code2flow

  • Pretty good call graphs for dynamic languages

  • Code2flow generates call graphs for dynamic programming language. Code2flow supports Python, JavaScript, Ruby, and PHP.

    The basic algorithm is simple:

    • Translate your source files into ASTs.
    • Find all function definitions.
    • Determine where those functions are called.
    • Connect the dots.

    Code2flow is useful for:

    • Untangling spaghetti code.
    • Identifying orphaned functions.
    • Getting new developers up to speed.

    Code2flow provides a pretty good estimate of your project's structure. No algorithm can generate a perfect call graph for a dynamic language – even less so if that language is duck-typed. See the known limitations in the section below.

  • https://github.com/scottrogowski/code2flow#known-limitations
  • https://github.com/scottrogowski/code2flow#why-is-it-impossible-to-generate-a-perfect-call-graph
    • Though in that toy example instance.. we could attempt to capture/output the possible states/differences/conditions/etc that could be achieved.. if this doesn't already handle it
• https://github.com/Persper/js-callgraph

  • Construct approximate static call graph for JavaScript & Typescript

  • This project implements a field-based call graph construction algorithm for JavaScript as described in

    A. Feldthaus, M. Schäfer, M. Sridharan, J. Dolby, F. Tip. Efficient Construction of Approximate Call Graphs for JavaScript IDE Services. In ICSE, 2013. This repo builds upon Max Schaefer's original acg.js

    • https://github.com/xiemaisi/acg.js
• https://github.com/whyboris/TypeScript-Call-Graph

  • CLI to generate an interactive graph of functions and calls from your TypeScript files

  • https://github.com/Deskbot/TS-Call-Graph

    • A program that generates a graph of the methods and attributes of a TypeScript class

• https://github.com/julianjensen/ast-flow-graph

  • ast-flow-graph

  • Creates a CFG from JavaScript source code.

  • This module will read one or more JavaScript source files and produce CFGs (Control Flow Graphs) of the code.

  • Uses espree, escope, estraverse, etc
  • https://github.com/isaacs/yallist

    • Yet Another Linked List

      There are many doubly-linked list implementations like it, but this one is mine.

      For when an array would be too big, and a Map can't be iterated in reverse order.

  • https://github.com/julianjensen/traversals

    • Small module for graph traversals, supporting DFS and BFS with niceties added for pre- and post-order, including their reverses.

    • Some notes from ChatGPT:

      • Provides a small module designed for performing graph traversal operations, specifically Depth-First Search (DFS) and Breadth-First Search (BFS). It includes additional features such as pre-order and post-order traversals, as well as their reverse versions, to enhance the functionality of these standard graph traversal techniques.

  • https://github.com/julianjensen/dominators

    • Various dominator tree algorithms

    • It implements two different methods for finding the immediate dominators of a graph.

    • Some notes from ChatGPT:

      • A dominator tree is a concept used in computer science, particularly in the field of compiler design and program analysis. To understand a dominator tree, let's first look at the concept of dominators in a control flow graph (CFG).

        In a CFG, which represents the flow of control in a program, a node ( A ) is said to dominate another node ( B ) if every path from the start node of the graph to ( B ) must go through ( A ). In other words, ( A ) dominates ( B ) if ( A ) is always encountered before ( B ) when traversing the graph from the start node.

        The concept becomes more nuanced with the idea of immediate dominators. An immediate dominator of a node ( B ) is the last dominator on any path from the start node to ( B ).

        Now, a dominator tree is a tree structure that represents these dominance relationships within a CFG. In this tree:

        • Each node corresponds to a node in the original CFG.
        • There is a directed edge from node ( A ) to node ( B ) if ( A ) is the immediate dominator of ( B ) in the CFG.

        A dominator tree generator, therefore, is a tool or an algorithm that constructs the dominator tree from a given control flow graph. This tool is essential in optimizing compilers and in various program analysis tasks, where understanding the dominance relationships helps in transformations like loop optimization, dead code elimination, and more sophisticated analyses like static single assignment (SSA) form conversion.

        This concept is closely related to computer science and software engineering, particularly in areas concerning compiler construction and code optimization. Given your background in software engineering and ethical hacking, this knowledge could be particularly useful in understanding code structure and flow, especially when analyzing or optimizing complex software systems.

• https://github.com/dochne/wappalyzer

  • Wappalyzer identifies technologies on websites, such as CMS, web frameworks, ecommerce platforms, JavaScript libraries, analytics tools and more.

  • The last commit of Wappalyzer before it went private

  • Specification A long list of regular expressions is used to identify technologies on web pages. Wappalyzer inspects HTML code, as well as JavaScript variables, response headers and more. Patterns (regular expressions) are kept in src/technologies/

    • https://github.com/dochne/wappalyzer/tree/main/src/technologies
    • https://github.com/dochne/wappalyzer/blob/main/src/groups.json
    • https://github.com/dochne/wappalyzer/blob/main/src/categories.json
    • https://github.com/dochne/wappalyzer/blob/471c2fb0b093973c098bd1855b89c8cde4997479/src/js/index.js#L133
      • Loads all of the technologies .json files: chrome.runtime.getURL(`technologies/${character}.json`)
      • Deletes some keys from the loaded data:

        • Object.keys(technologies).forEach((name) => {
            delete technologies[name].description
            delete technologies[name].cpe
            delete technologies[name].pricing
            delete technologies[name].website
          })

      • calls setTechnologies(technologies)
        • https://github.com/dochne/wappalyzer/blob/471c2fb0b093973c098bd1855b89c8cde4997479/src/js/wappalyzer.js#L348-L425
          • Re-maps a bunch of the .json keys to different in-memory names/etc
      • analyzeJs: Analyse JavaScript variables
      • analyzeDom: Analyse DOM nodes
      • analyzeManyToMany

        • Object.keys(technology[type]).reduce((technologies, key) => {
            const patterns = technology[type][key] || []
            const values = items[key] || []
          
            patterns.forEach((_pattern) => {
              const pattern = (subtypes || []).reduce(
                (pattern, subtype) => pattern[subtype] || {},
                _pattern
              )
          
              values.forEach((value) => {
                const startTime = Date.now()
          
                const matches = pattern.regex.exec(value)
          
                if (matches) {
                  technologies.push({

    • We could download and filter these .json files to find the js patterns used similar to this:

      • ⇒ wget https://github.com/dochne/wappalyzer/raw/main/src/technologies/a.json
        
        ⇒ jq 'to_entries | map(select(.value.js != null) | {key: .key, value: .value.js}) | from_entries' a.json
        
        # ..snip..
        #
        #  "anime.js": {
        #    "anime.version": "^([\\d\\.]+)$\\;version:\\1"
        #  },
        #  "authorize.net": {
        #    "config.authorizenet_public_client_key": ""
        #  }
        #}

• https://github.com/lebab/lebab

  • Turn your ES5 code into readable ES6. Lebab does the opposite of what Babel does.

  • Lebab transpiles your ES5 code to ES6/ES7. It does exactly the opposite of what Babel does.

  • https://lebab.github.io/ (Playground)
• https://github.com/EricSmekens/jsep

  • jsep: A Tiny JavaScript Expression Parser jsep is a simple expression parser written in JavaScript. It can parse JavaScript expressions but not operations. The difference between expressions and operations is akin to the difference between a cell in an Excel spreadsheet vs. a proper JavaScript program.

  • https://ericsmekens.github.io/jsep/
• https://github.com/oxc-project/oxc

  • Oxc The Oxidation Compiler is creating a suite of high-performance tools for JavaScript and TypeScript.

    Oxc is building a parser, linter, formatter, transpiler, minifier, resolver ... all written in Rust.

  • https://github.com/oxc-project/oxc#-ast-and-parser

    • AST and Parser Oxc maintains its own AST and parser, which is by far the fastest and most conformant JavaScript and TypeScript (including JSX and TSX) parser written in Rust.

    • While many existing JavaScript tools rely on estree as their AST specification, a notable drawback is its abundance of ambiguous nodes. This ambiguity often leads to confusion during development with estree.

      The Oxc AST differs slightly from the estree AST by removing ambiguous nodes and introducing distinct types. For example, instead of using a generic estree Identifier, the Oxc AST provides specific types such as BindingIdentifier, IdentifierReference, and IdentifierName. This clear distinction greatly enhances the development experience by aligning more closely with the ECMAScript specification.

  • https://oxc-project.github.io/
    • https://oxc-project.github.io/oxc/playground/
    • https://oxc-project.github.io/docs/guide/usage/parser.html

    • Parser

      • 2x faster then SWC parser
      • By far the fastest and most conformant JavaScript and TypeScript (including JSX and TSX) parser written in Rust

      • https://github.com/oxc-project/bench-javascript-parser-written-in-rust

        • Benchmark for Oxc, Swc and Rome parser

    • The umbrella crate oxc exports all public crates from this repository

    • The AST and parser crates oxc_ast and oxc_parser are production ready

      • https://docs.rs/oxc_parser/latest/oxc_parser/#conformance

        • The parser parses all of Test262 and most of Babel and TypeScript parser conformance tests.

          • https://github.com/oxc-project/oxc/tree/main/tasks/coverage
  • https://oxc-project.github.io/javascript-parser-in-rust/

    • Write a JavaScript Parser in Rust

      Rust, JavaScript, and parsers are all hard to learn, let's combine these three and challenge ourselves to write a JavaScript parser in Rust.

      This will be the guide for you if you are interested in learning Rust, parsers, or would like to contribute to oxc, swc or Biome in the near future.

      The guide will cover all the basic topics of writing a JavaScript parser in rust. The tutorials will explain some topics in more depth.

    • https://oxc-project.github.io/javascript-parser-in-rust/blog/
      • https://oxc-project.github.io/javascript-parser-in-rust/blog/rome/

        • Rome Tools Rome uses a different set of techniques for parsing JavaScript and TypeScript. This tutorial summarizes them in learning order for better understanding.

        • The Rome codebase was rewritten from TypeScript to Rust, see Rome will be rewritten in Rust The decision was made after talking to the author of rslint and rust-analyzer rust-analyzer proved that IDE-centric tools built around concrete syntax tree are possible rslint proved that it is possible to write a JavaScript parser in Rust, with the same base libraries as rust-analyzer

        • The base library is called rowan, see overview of rowan Rowan, also known as red-green trees, is named after the real green rowan tree that makes red berries The origin of red-green trees is described in this blog post, by the authors of the C# programming language The whole point of rowan is to define a lossless concrete syntax tree (CST) that describes all the details of the source code and provides a set of traversal APIs (parent, children, siblings, etc) Read the advantage of having a CST over an AST: Pure AST based linting sucks CST provides the ability to build a fully recoverable parser

  • https://craftinginterpreters.com/

    • Ever wanted to make your own programming language or wondered how they are designed and built? If so, this book is for you.

    • Crafting Interpreters contains everything you need to implement a full-featured, efficient scripting language. You’ll learn both high-level concepts around parsing and semantics and gritty details like bytecode representation and garbage collection. Your brain will light up with new ideas, and your hands will get dirty and calloused. It’s a blast.

      Starting from main(), you build a language that features rich syntax, dynamic typing, garbage collection, lexical scope, first-class functions, closures, classes, and inheritance. All packed into a few thousand lines of clean, fast code that you thoroughly understand because you write each one yourself.

• https://github.com/kaleidawave/ezno

  • A JavaScript compiler and TypeScript checker written in Rust with a focus on static analysis and runtime performance

  • https://kaleidawave.github.io/posts/introducing-ezno/

    • Introducing Ezno

    • Ezno is an experimental compiler I have been working on and off for a while. In short, it is a JavaScript compiler featuring checking, correctness and performance for building full-stack (rendering on the client and server) websites.

  • https://kaleidawave.github.io/posts/ezno-23/

    • Ezno in '23

• https://github.com/boa-dev/boa

  • Boa is an embeddable and experimental Javascript engine written in Rust. Currently, it has support for some of the language.

  • This is an experimental Javascript lexer, parser and interpreter written in Rust. Currently, it has support for some of the language.

  • https://boajs.dev/

    • Boa is an experimental Javascript lexer, parser and compiler written in Rust. Currently, it has support for some of the language. It can be embedded in Rust projects fairly easily and also used from the command line. Boa also exists to serve as a Rust implementation of the EcmaScript specification, there will be areas where we can utilise Rust and its fantastic ecosystem to make a fast, concurrent and safe engine.

    • https://boajs.dev/boa/playground/
    • https://boajs.dev/posts/2022-10-24-boa-usage/

      • Adding a JavaScript interpreter to your Rust project

    • https://boajs.dev/boa/dev/bench/

      • Boa Benchmarks

      • https://github.com/boa-dev/boa/blob/main/boa_engine/benches/full.rs#L9
        • https://github.com/bheisler/criterion.rs

          • Statistics-driven benchmarking library for Rust

          • Criterion.rs helps you write fast code by detecting and measuring performance improvements or regressions, even small ones, quickly and accurately. You can optimize with confidence, knowing how each change affects the performance of your code.

          • https://docs.rs/criterion/latest/criterion/
    • https://boajs.dev/boa/test262/

      • EcmaScript conformance test results for Boa

• https://github.com/rust-analyzer/rowan

  • Rowan is a library for lossless syntax trees, inspired in part by Swift's libsyntax.

• https://github.com/mozilla-spidermonkey/jsparagus

  • Experimental JS parser-generator project.

  • jsparagus - A JavaScript parser written in Rust jsparagus is intended to replace the JavaScript parser in Firefox.

  • https://github.com/mozilla-spidermonkey/jsparagus#benchmarking

    • Benchmarking

  • https://github.com/mozilla-spidermonkey/jsparagus#limitations

    • Limitations

• https://github.com/ratel-rust/ratel-core

  • High performance JavaScript to JavaScript compiler with a Rust core

  • https://maciej.codes/ratel-wasm/ (REPL / Playground)
  • https://github.com/ratel-rust/ratel-cli

    • A command-line interface for Ratel, a high performance JavaScript to JavaScript compiler with a Rust core

Recast + related

• https://github.com/benjamn/recast

  • JavaScript syntax tree transformer, nondestructive pretty-printer, and automatic source map generator

  • https://github.com/benjamn/recast/blob/master/lib/options.ts

    • All Recast API functions take second parameter with configuration options, documented in options.js

  • https://github.com/benjamn/recast/tree/master/parsers
    • acorn, babel-ts, babel, babylon, esprima, flow, typescript
  • https://github.com/benjamn/ast-types

    • Esprima-compatible implementation of the Mozilla JS Parser API

    • https://github.com/benjamn/ast-types#ast-traversal

      • AST Traversal Because it understands the AST type system so thoroughly, this library is able to provide excellent node iteration and traversal mechanisms.

    • https://github.com/benjamn/ast-types#nodepath

      • NodePath The NodePath object passed to visitor methods is a wrapper around an AST node, and it serves to provide access to the chain of ancestor objects (all the way back to the root of the AST) and scope information.

    • https://github.com/benjamn/ast-types#scope

      • Scope The object exposed as path.scope during AST traversals provides information about variable and function declarations in the scope that contains path.node. See scope.ts for its public interface, which currently includes .isGlobal, .getGlobalScope(), .depth, .declares(name), .lookup(name), and .getBindings()

        • https://github.com/benjamn/ast-types/blob/master/src/scope.ts
    • https://github.com/benjamn/ast-types#custom-ast-node-types

      • Custom AST Node Types The ast-types module was designed to be extended. To that end, it provides a readable, declarative syntax for specifying new AST node types, based primarily upon the require("ast-types").Type.def function

      • The def syntax is used to define all the default AST node types found in babel-core.ts, babel.ts, core.ts, es-proposals.ts, es6.ts, es7.ts, es2020.ts, esprima.ts, flow.ts, jsx.ts, type-annotations.ts, and typescript.ts, so you have no shortage of examples to learn from.

        • https://github.com/benjamn/ast-types/tree/master/src/def
          • babel-core.ts, babel.ts, core.ts, es-proposals.ts, es2016.ts, es2017.ts, es2018.ts, es2019.ts, es2020.ts, es2021.ts, es2022.ts, es6.ts, esprima.ts, flow.ts, jsx.ts, type-annotations.ts, typescript.ts
          • operators/: core.ts, es2016.ts, es2020.ts, es2021.ts
• https://github.com/facebook/jscodeshift

  • A JavaScript codemod toolkit

  • jscodeshift is a toolkit for running codemods over multiple JavaScript or TypeScript files. It provides:

    • A runner, which executes the provided transform for each file passed to it. It also outputs a summary of how many files have (not) been transformed.
    • A wrapper around recast, providing a different API. Recast is an AST-to-AST transform tool and also tries to preserve the style of original code as much as possible.

  • facebook/jscodeshift#500

    • Bringing jscodeshift up to date

    • The biggest issue is with recast. This library hasn't really had a lot of maintenance for the last couple of years, and there's something like 150+ issues and 40+ pull requests waiting to be merged. It seems like 80% of the issues that are logged against jscodeshift are actually recast issues. In order to fix the jscodeshift's outstanding issues, either recast itself needs to fix them or jscodeshift will need to adopt/create its own fork of recast to solve them. For the past year and a half or so putout's main developer has been maintaining a fork of recast and adding a lot of fixes to it. It might be worthwhile to look at switching to @putout/recast as opposed to the recast upstream. I've also been working on a fork of @putout/recast for evcodeshift that adds a few other things to make evcodeshift transforms more debuggable in vscode.

    • https://github.com/putoutjs/recast
      • https://github.com/putoutjs/printer

      • Prints Babel AST to readable JavaScript. For ESTree use estree-to-babel.

        • Similar to Recast, but twice faster, also simpler and easier in maintenance, since it supports only Babel.
        • As opinionated as Prettier, but has more user-friendly output and works directly with AST.
        • Like ESLint but works directly with Babel AST.
        • Easily extendable with help of Overrides.

  • What can be said about recast can probably also be said to a lesser degree about ast-types

• https://github.com/codemod-js/codemod

  • codemod rewrites JavaScript and TypeScript using babel plugins

• https://github.com/unjs/magicast

  • Programmatically modify JavaScript and TypeScript source codes with a simplified, elegant and familiar syntax powered by recast and babel.

estools + related

• https://github.com/estools
  • https://github.com/estools/escope

    • Escope: ECMAScript scope analyzer

    • https://github.com/mazurov/escope-demo

      • Escope library: Scope Objects Visualization

      • https://mazurov.github.io/escope-demo/
      • https://github.com/lizhihao132/escope-demo/

        • Scope Objects Visualization (By Esprima, Acorn, escope, eslint-scope)

        • Note: This is a fork of the original escope-demo repo that seems to add new features/etc
        • https://lizhihao132.github.io/escope-demo/
    • https://github.com/mazurov/eslevels

      • ECMAScript scope levels analyzer based on escope library

      • ECMAScript scope levels analyzer based on escope library. The original purpose of this library is to enable scope context coloring in javascript editors (for SublimeText in first order).

      • https://github.com/mazurov/eslevels-demo

        • JavaScript scope coloring based on esprima toolbox

        • This is a simple web application created to show features of eslevels javascript library — am ECMAScript scope levels analyzer based on escope library which original purpose was to enable scope context coloring in javascript editors (SublimeText in first order).

        • https://mazurov.github.io/eslevels-demo/
      • https://github.com/mazurov/sublime-levels

        • SublimeText plugin for scope context coloring (ST2/ST3)

  • https://github.com/estools/esquery

    • ECMAScript AST query library

    • ESQuery is a library for querying the AST output by Esprima for patterns of syntax using a CSS style selector system

    • https://estools.github.io/esquery/
    • See also: https://eslint.org/docs/latest/extend/selectors
  • https://github.com/phenomnomnominal/tsquery

    • TypeScript AST query library

    • TSQuery is a port of the ESQuery API for TypeScript! TSQuery allows you to query a TypeScript AST for patterns of syntax using a CSS style selector system.

    • https://github.com/urish/tsquery-playground

      • Playground for TSQuery

      • https://tsquery-playground.firebaseapp.com/
  • https://github.com/estools/esutils

    • utility box for ECMAScript language tools

  • https://github.com/estools/estraverse

    • Estraverse (estraverse) is ECMAScript traversal functions from esmangle project.

  • https://github.com/estools/esrecurse

    • Esrecurse (esrecurse) is ECMAScript recursive traversing functionality.

  • https://github.com/estools/escodegen

    • ECMAScript code generator

  • https://github.com/inikulin/esotope

    • ECMAScript code generator on steroids

    • This project has been started as a fork of escodegen with intention to speed up the original code. escodegen is a great project, however it was a constant bottleneck in our project, where we are doing a real-time JavaScript code instrumentation. When nearly 70% of the original code was rewritten, it became clear that it cannot be issued as a PR to the original repo and I decided to leave it as a standalone project. Currently esotope is x2 times faster than escodegen in node v0.10.x, and x4.5 times faster in node v0.11.x (benchmark). However in production we've seen x10 times performance gain in some cases.

  • https://github.com/estools/estemplate

    • Proper (AST-based) JavaScript code templating with source maps support.

    • This module allows to generate JavaScript AST from code template and AST nodes as substitutions. This is more proper way of code templating since it works on AST not on code string, and thus preserves locations which allow to generate source maps in future.

  • https://github.com/estools/esshorten

    • Shorten (mangle) names in JavaScript code

    • esshorten provides name mangler, this shorten names in JavaScript code. mangler accepts JavaScript AST and generate modified AST with shortened names.

  • https://github.com/estools/esmangle

    • esmangle is mangler / minifier for Mozilla Parser API AST

  • https://github.com/estools/espurify

    • Clone AST without extra properties

    • Leaves properties defined in The ESTree Spec (formerly known as Mozilla SpiderMonkey Parser API) only. Also note that extra informations (such as loc, range and raw) are eliminated too.

• https://github.com/ariya/esrefactor

  • esrefactor (BSD licensed) is a little helper library for ECMAScript refactoring.

Babel

• https://babeljs.io/
  • https://github.com/babel/website

    • The Babel documentation website

  • https://babeljs.io/repl
    • https://github.com/babel/website/tree/main/js/repl
    • https://github.com/babel/sandboxes

      • Babel repl-like codesandbox

    • Not sure any of these are the official ones.. but noting here anyway:
      • https://bvaughn.github.io/babel-repl/
        • https://github.com/bvaughn/babel-repl

          • React powered Babel REPL

  • https://babeljs.io/docs/babel-parser

    • @babel/parser

    • The Babel parser (previously Babylon) is a JavaScript parser used in Babel

    • Heavily based on acorn and acorn-jsx

    • https://babeljs.io/docs/babel-parser#api
    • https://babeljs.io/docs/babel-parser#output

      • The Babel parser generates AST according to Babel AST format. It is based on ESTree spec with the following deviations...

        • https://github.com/babel/babel/blob/main/packages/babel-parser/ast/spec.md

      • AST for JSX code is based on Facebook JSX AST

        • https://github.com/facebook/jsx/blob/main/AST.md
    • https://babeljs.io/docs/babel-parser#plugins
      • https://babeljs.io/docs/babel-parser#language-extensions

        • Language extensions

      • https://babeljs.io/docs/babel-parser#ecmascript-proposals

        • ECMAScript proposals

      • https://babeljs.io/docs/babel-parser#latest-ecmascript-features

        • The following features are already enabled on the latest version of @babel/parser, and cannot be disabled because they are part of the language. You should enable these features only if you are using an older version.

    • https://github.com/babel/babel/blob/main/packages/babel-parser
      • https://github.com/babel/babel/blob/main/packages/babel-parser/src/util/scope.ts

        • The functions in this module keep track of declared variables in the current scope in order to detect duplicate variable names.

        • currentScope()
          inFunction()
          inStaticBlock()
          declareName(name: string, bindingType: BindingTypes, loc: Position)
          // etc

        • https://github.com/babel/babel/blob/main/packages/babel-parser/src/util/scope.ts#L24-L31

          • var: Set<string> = new Set(); // A set of var-declared names in the current lexical scope
            lexical: Set<string> = new Set(); // A set of lexically-declared names in the current lexical scope
            functions: Set<string> = new Set(); // A set of lexically-declared FunctionDeclaration names in the current lexical scope`

  • https://babeljs.io/docs/babel-traverse

    • @babel/traverse

    • We can use it alongside the babel parser to traverse and update nodes

    • https://github.com/babel/babel/tree/main/packages/babel-traverse
      • https://github.com/babel/babel/blob/main/packages/babel-traverse/src/traverse-node.ts#L8-L20

        • Traverse the children of given node

      • https://github.com/babel/babel/blob/main/packages/babel-traverse/src/scope/index.ts#L380-L394

        • export default class Scope {
            uid;
            path: NodePath;
            block: t.Pattern | t.Scopable;
            labels;
            inited;
            bindings: { [name: string]: Binding };
            references: { [name: string]: true };
            globals: { [name: string]: t.Identifier | t.JSXIdentifier };
            uids: { [name: string]: boolean };
            data: { [key: string | symbol]: unknown };
            crawling: boolean;

        • rename(oldName: string, newName?: string, /* Babel 7 - block?: t.Pattern | t.Scopable */)
          dump()
          getProgramParent() // Walk up to the top of the scope tree and get the `Program`.
          getFunctionParent() // Walk up the scope tree until we hit either a Function or return null.
          getBlockParent() // Walk up the scope tree until we hit either a BlockStatement/Loop/Program/Function/Switch or reach the very top and hit Program.
          getPatternParent() // Walk up from a pattern scope (function param initializer) until we hit a non-pattern scope, then returns its block parent
          getAllBindings(): Record<string, Binding> // Walks the scope tree and gathers **all** bindings.
          getAllBindingsOfKind(...kinds: string[]): Record<string, Binding> // Walks the scope tree and gathers all declarations of `kind`.
          getBinding(name: string): Binding | undefined
          getOwnBinding(name: string): Binding | undefined
          parentHasBinding(name: string, opts?: { noGlobals?: boolean; noUids?: boolean })
          // etc

      • https://github.com/babel/babel/blob/main/packages/babel-traverse/src/path/index.ts#L36-L51
        • class NodePath<T extends t.Node = t.Node> {

        • declare parent: t.ParentMaps[T["type"]];
          declare hub: HubInterface;
          declare data: Record<string | symbol, unknown>;
          // TraversalContext is configured by setContext
          declare context: TraversalContext;
          declare scope: Scope;

        • debug(message: string) // Generates a debug message with the context of the path location
          toString() // Generates the code for this path
          // etc

  • https://babeljs.io/docs/babel-generator

    • @babel/generator

    • Turns an AST into code.

  • https://babeljs.io/docs/babel-template

    • @babel/template

    • When calling template as a function with a string argument, you can provide placeholders which will get substituted when the template is used.

  • https://babeljs.io/docs/babel-code-frame

    • @babel/code-frame

    • Babel Code Frame is a package in the Babel toolchain that generates errors containing a "code frame" which points to specific source locations in the code, aiding in debugging and error analysis

  • https://babeljs.io/docs/babel-types
    • https://babeljs.io/docs/babel-types#aliases
      • https://babeljs.io/docs/babel-types#scopable

        • A cover of FunctionParent and BlockParent.

        • https://babeljs.io/docs/babel-types#functionparent

          • A cover of AST nodes that start an execution context with new VariableEnvironment. In other words, they define the scope of var declarations. FunctionParent did not include Program since Babel 7.

        • https://babeljs.io/docs/babel-types#blockparent

          • A cover of AST nodes that start an execution context with new LexicalEnvironment. In other words, they define the scope of let and const declarations.

  • https://babeljs.io/docs/babel-helper-module-imports

    • @babel/helper-module-imports

  • https://babeljs.io/docs/babel-helper-validator-identifier

    • @babel/helper-validator-identifier is a utility package for parsing JavaScript keywords and identifiers. It provides several helper functions for identifying valid identifier names and detecting reserved words and keywords.

  • https://babeljs.io/docs/babel-helper-environment-visitor

    • @babel/helper-environment-visitor is a utility package that provides a current this context visitor.

semantic / tree-sitter + related

• https://github.com/github/semantic

  • Parsing, analyzing, and comparing source code across many languages

• https://github.com/tree-sitter/tree-sitter

  • An incremental parsing system for programming tools

  • https://github.com/tree-sitter/tree-sitter/blob/master/lib/binding_web/

    • WebAssembly bindings to the Tree-sitter parsing library.

  • https://github.com/tree-sitter/tree-sitter/tree/master/cli

    • The Tree-sitter CLI allows you to develop, test, and use Tree-sitter grammars from the command line.

  • https://tree-sitter.github.io/tree-sitter/

    • Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited.

    • https://tree-sitter.github.io/tree-sitter/playground
    • https://tree-sitter.github.io/tree-sitter/using-parsers
    • https://tree-sitter.github.io/tree-sitter/creating-parsers
    • https://tree-sitter.github.io/tree-sitter/syntax-highlighting
      • https://github.com/tree-sitter/tree-sitter/tree/master/highlight
    • https://tree-sitter.github.io/tree-sitter/code-navigation-systems

      • Tree-sitter can be used in conjunction with its tree query language as a part of code navigation systems. An example of such a system can be seen in the tree-sitter tags command, which emits a textual dump of the interesting syntactic nodes in its file argument. A notable application of this is GitHub’s support for search-based code navigation. This document exists to describe how to integrate with such systems, and how to extend this functionality to any language with a Tree-sitter grammar.

      • https://tree-sitter.github.io/tree-sitter/code-navigation-systems#tagging-and-captures

        • Tagging is the act of identifying the entities that can be named in a program. We use Tree-sitter queries to find those entities. Having found them, you use a syntax capture to label the entity and its name.

        • The essence of a given tag lies in two pieces of data: the role of the entity that is matched (i.e. whether it is a definition or a reference) and the kind of that entity, which describes how the entity is used (i.e. whether it’s a class definition, function call, variable reference, and so on). Our convention is to use a syntax capture following the @role.kind capture name format, and another inner capture, always called @name, that pulls out the name of a given identifier.

        • A more sophisticated query can be found in the JavaScript Tree-sitter repository

      • https://tree-sitter.github.io/tree-sitter/using-parsers#pattern-matching-with-queries

        • Many code analysis tasks involve searching for patterns in syntax trees. Tree-sitter provides a small declarative language for expressing these patterns and searching for matches.

        • https://tree-sitter.github.io/tree-sitter/using-parsers#query-syntax

          • A query consists of one or more patterns, where each pattern is an S-expression that matches a certain set of nodes in a syntax tree. The expression to match a given node consists of a pair of parentheses containing two things: the node’s type, and optionally, a series of other S-expressions that match the node’s children.

        • https://tree-sitter.github.io/tree-sitter/using-parsers#the-query-api
• https://github.com/tree-sitter/node-tree-sitter

  • Node.js bindings for tree-sitter

• https://www.npmjs.com/package/web-tree-sitter

  • WebAssembly bindings to the Tree-sitter parsing library.

  • https://github.com/tree-sitter/tree-sitter/tree/master/lib/binding_web

    • WebAssembly bindings to the Tree-sitter parsing library

  • https://crates.io/crates/tree-sitter-javascript
    • https://github.com/tree-sitter/tree-sitter-javascript

    • JavaScript and JSX grammar for tree-sitter. For TypeScript, see tree-sitter-typescript.

• https://github.com/afnanenayet/diffsitter

  • A tree-sitter based AST difftool to get meaningful semantic diffs

  • diffsitter is very much a work in progress and nowhere close to production ready (yet). Contributions are always welcome!

  • diffsitter creates semantically meaningful diffs that ignore formatting differences like spacing. It does so by computing a diff on the AST (abstract syntax tree) of a file rather than computing the diff on the text contents of the file.

  • diffsitter uses the parsers from the tree-sitter project to parse source code. As such, the languages supported by this tool are restricted to the languages supported by tree-sitter.

  • https://news.ycombinator.com/item?id=27875333

Shift AST

• https://github.com/shapesecurity/shift-spec

  • Shift AST Specification

  • https://shift-ast.org/
• https://github.com/shapesecurity/shift-parser-js

  • ECMAScript parser that produces a Shift format AST

• https://github.com/shapesecurity/shift-codegen-js

  • Code generator for Shift format ASTs

• https://github.com/shapesecurity/shift-scope-js
  • Scope analyser for the Shift AST
• https://github.com/shapesecurity/shift-reducer-js

  • Reducer for the Shift AST format

• https://github.com/shapesecurity/shift-fuzzer-js

  • Generate random valid Shift format ASTs

• https://github.com/codemodsquad/astx

  • Super powerful structural search and replace for JavaScript and TypeScript to automate your refactoring

swc

• https://swc.rs/

  • SWC is an extensible Rust-based platform for the next generation of fast developer tools. It's used by tools like Next.js, Parcel, and Deno, as well as companies like Vercel, ByteDance, Tencent, Shopify, and more.

    SWC can be used for both compilation and bundling. For compilation, it takes JavaScript / TypeScript files using modern JavaScript features and outputs valid code that is supported by all major browsers.

    SWC is 20x faster than Babel on a single thread and 70x faster on four cores.

  • https://swc.rs/playground
    • This can show both transformed code, as well as the AST
    • https://github.com/swc-project/swc-playground

      • The SWC playground

      • Two editors powered by Monaco Editor for editing input code and showing output code.

  • https://swc-css.netlify.app/
    • https://github.com/g-plane/swc-css-playground

      • Playground for SWC CSS

      • Currently there's only AST viewer

  • https://swc.rs/docs/usage/core

    • @swc/core

    • These are the core SWC APIs mainly useful for build tool authors

    • https://swc.rs/docs/usage/core#parse

      • parse Returns Promise<Script | Module>

  • https://swc.rs/docs/usage/wasm

    • @swc/wasm-web This modules allows you to synchronously transform code inside the browser using WebAssembly.

    • https://codesandbox.io/examples/package/@swc/wasm-web

      • Use this online @swc/wasm-web playground to view and fork @swc/wasm-web example apps and templates on CodeSandbox

    • Some example usages of @swc/wasm-web
      • https://github.com/fkling/astexplorer/blob/master/website/src/parsers/js/swc.js#L3
      • https://github.com/sxzz/ast-explorer/blob/main/composables/language/javascript.ts#L74-L109
    • swc-project/swc#3713

      • Running a plugin with @swc/wasm-web

• https://github.com/swc-project/swc

  • Rust-based platform for the Web

  • SWC (stands for Speedy Web Compiler) is a super-fast TypeScript / JavaScript compiler written in Rust. It's a library for Rust and JavaScript at the same time. If you are using SWC from Rust, see rustdoc and for most users, your entry point for using the library will be parser.

    • https://rustdoc.swc.rs/swc/

      • The main crate of the swc project

    • https://rustdoc.swc.rs/swc_ecma_parser/

      • EcmaScript/TypeScript parser for the rust programming language.

      • https://rustdoc.swc.rs/swc_ecmascript/
      • https://rustdoc.swc.rs/swc_ecma_ast/
      • https://rustdoc.swc.rs/swc_ecma_visit/
      • https://rustdoc.swc.rs/swc_ecma_codegen/
      • https://rustdoc.swc.rs/swc_ecma_dep_graph/
      • https://rustdoc.swc.rs/swc_ecma_minifier/

        • JavaScript minifier implemented in rust

      • https://rustdoc.swc.rs/swc_ecma_transforms/
      • etc
    • https://rustdoc.swc.rs/swc_estree_compat/
      • https://rustdoc.swc.rs/swc_estree_ast/
        • https://rustdoc.swc.rs/swc_estree_compat/babelify/trait.Babelify.html
        • https://rustdoc.swc.rs/swc_estree_compat/swcify/trait.Swcify.html

          • Used to convert a babel ast node to

    • https://rustdoc.swc.rs/swc_html_parser/
      • https://rustdoc.swc.rs/swc_html_ast/
    • https://rustdoc.swc.rs/swc_css_parser/
      • https://rustdoc.swc.rs/swc_css_ast/
    • https://rustdoc.swc.rs/swc_xml_parser/
      • https://rustdoc.swc.rs/swc_xml_ast/

        • AST definitions for XML

    • https://rustdoc.swc.rs/wasmparser/

      • A simple event-driven library for parsing WebAssembly binary files (or streams). The parser library reports events as they happen and only stores parsing information for a brief period of time, making it very fast and memory-efficient. The event-driven model, however, has some drawbacks. If you need random access to the entire WebAssembly data-structure, this is not the right library for you. You could however, build such a data-structure using this library.

      • https://rustdoc.swc.rs/wast/

        • A crate for low-level parsing of the WebAssembly text formats: WAT and WAST. This crate is intended to be a low-level detail of the wat crate, providing a low-level parsing API for parsing WebAssembly text format structures. The API provided by this crate is very similar to syn and provides the ability to write customized parsers which may be an extension to the core WebAssembly text format. For more documentation see the parser module.

          • https://rustdoc.swc.rs/wast/parser/

            • Traits for parsing the WebAssembly Text format This module contains the traits, abstractions, and utilities needed to define custom parsers for WebAssembly text format items.

            • The top-level parse function can be used to fully parse AST fragments

• https://www.christopherbiscardi.com/how-to-print-a-javascript-ast-using-swc-and-rust

  • How to print a JavaScript AST using SWC and Rust

• https://blog.logrocket.com/writing-webpack-plugins-rust-using-swc/

  • Writing webpack plugins in Rust using SWC for faster builds

  • In this tutorial, we’ll create a simple custom Rust-based SWC plugin and compile it to Wasm, so it can be used in a webpack build using swc-loader. We’ll build our simple plugin and import it in a JavaScript project using webpack, configure it as a plugin running with the swc-loader within webpack, and then check that the plugin was run and that it worked.

• https://chat.openai.com/c/1ab6cfcd-3fd5-43d8-b13e-a604239450ca
  • ChatGPT chat showing example of creating rust parsing code (using swc_ecma_parser, JsValue::from_serde, etc), compiling to webassembly with wasm-pack, and using it from JavaScript
  • https://github.com/rustwasm/wasm-pack

    • This tool seeks to be a one-stop shop for building and working with rust- generated WebAssembly that you would like to interop with JavaScript, in the browser or with Node.js. wasm-pack helps you build rust-generated WebAssembly packages that you could publish to the npm registry, or otherwise use alongside any javascript packages in workflows that you already use

    • https://rustwasm.github.io/wasm-pack/book/
• https://github.com/coderaiser/swc-to-babel

  • convert SWC to Babel AST

  • https://github.com/coderaiser/estree-to-babel

    • convert estree ast to babel

    • Convert ESTree-compatible JavaScript AST to Babel AST.

    • https://github.com/j4k0xb/estree-to-babel/tree/perf
      • https://github.com/coderaiser/estree-to-babel/compare/master...j4k0xb:estree-to-babel:perf
        • Replaces cherow with meriyah

        • When using meriyah and estree-to-babel, its up to 2.8x faster than @babel/parser alone.

          • https://github.com/meriyah/meriyah

            • 100% compliant, self-hosted javascript parser with high focus on both performance and stability. Stable and already used in production.

            • https://meriyah.github.io/meriyah/ (Playground)

esbuild

• https://github.com/evanw/esbuild

  • An extremely fast bundler for the web

  • Written in Golang
  • https://esbuild.github.io/
    • https://esbuild.github.io/faq/#upcoming-roadmap

      • I am not planning to include these features in esbuild's core itself:

        • ..snip..
        • An API for custom AST manipulation
        • ..snip..

        I hope that the extensibility points I'm adding to esbuild (plugins and the API) will make esbuild useful to include as part of more customized build workflows, but I'm not intending or expecting these extensibility points to cover all use cases.

        • https://esbuild.github.io/plugins/
        • https://esbuild.github.io/api/
        • https://news.ycombinator.com/item?id=29004200

          • ESBuild does not support any AST transforms directly

            You can add it, via plugins, but its a serious limitation for a project like Next.js which require's these types of transforms

            You also end up with diminishing returns with the more plugins in you add to esbuild, and I imagine its worse with js plugins than it is with go based ones, none the less, you have zero access to it directly

          • It is trivial to write extensions for esbuild. We've written extensive plugins to perform ast transformations that all run, collectively, in under 0.5 seconds. Make a plugin, add acorn and escodegen.

            • This implies that the plugins are doing the AST transformation outside of esbuild itself (likely still running in JS), so wouldn't really benefit from the fact that esbuild is written in golang like I was hoping.
        • evanw/esbuild#2172

          • Forking esbuild to build an AST plugin tool

          • The internal AST is not designed for this use case at all, and it’s not a use case that I’m going to spend time supporting (so I’m not going to spend time documenting exactly how to do it). I recommend using some other tool if you want to do AST-level stuff, especially because keeping a hack like this working over time as esbuild changes might be a big pain for you.

          • If it really want to do this with esbuild, know that the AST is not cleanly abstracted and is only intended for use with esbuild (e.g. uses a lot of internal data structures, has implicit invariants regarding symbols and tree shaking, does some weird things for performance reasons).

Source Maps

• https://github.com/mozilla/source-map

  • Consume and generate source maps

  • This is a library to generate and consume the source map format described here

  • https://github.com/mozilla/source-map#examples
  • https://github.com/mozilla/source-map#api
• https://github.com/parcel-bundler/source-map

  • A fast source map manipulation, generation and consumption library written in Rust and Node.js

  • https://github.com/parcel-bundler/source-map#why-did-we-write-this-library

    • Why did we write this library Parcel is a performance conscious bundler, and therefore we like to optimise Parcel's performance as much as possible.

      Our original source-map implementation used mozilla's source-map and a bunch of javascript and had issues with memory usage and serialization times (we were keeping all mappings in memory using JS objects and write/read it using JSON for caching).

      This implementation has been written from scratch in Rust minimizing the memory usage, by utilizing indexes for sources and names and optimizing serialization times by using Buffers instead of JSON for caching.

• https://github.com/denandz/sourcemapper

  • Extract JavaScript source trees from Sourcemap files

  • Sourcemapper is a bit of golang to parse a sourcemap, as generated by webpack or similar, and spit out the original JavaScript files, recreating the source tree based on the file paths in the sourcemap.

  • https://pulsesecurity.co.nz/articles/javascript-from-sourcemaps

    • Extracting JavaScript from SourceMaps

• https://github.com/paazmaya/shuji

  • Reverse engineering JavaScript and CSS sources from sourcemaps

• https://github.com/Anthonyzou/Sourcemap-Unpack

  • Unpack source maps

• https://github.com/lydell/source-map-visualize

  • Quickly open an online source map visualization with local files

• https://github.com/sokra/source-map-visualization

  • Just a simple hacky visualisation of SourceMaps

  • https://sokra.github.io/source-map-visualization/#typescript

Visualisation/etc

• https://github.com/keeyipchan/esgoggles

  • Browser IDE for reading and exploring javascript code

    • Demo 1 -- style 1: https://keeyipchan.github.io/esgoggles/demo/style1.html (use Chrome)

    • Demo 2 -- style 2 with variable scopes: https://keeyipchan.github.io/esgoggles/demo/style2.html (use Chrome)

• https://github.com/Bogdan-Lyashenko/js-code-to-svg-flowchart

  • js2flowchart - a visualization library to convert any JavaScript code into beautiful SVG flowchart. Learn other’s code. Design your code. Refactor code. Document code. Explain code.

  • Imagine a library which takes any JS code and generate SVG flowchart from it, works on client and server. Allows you easily adjust styles scheme for your context or demonstrate your code logic from different abstractions levels. Highlighting, destructing whole blocks, custom modifiers for your needs etc.

  • https://bogdan-lyashenko.github.io/js-code-to-svg-flowchart/docs/live-editor/

Browser Based Code Editors / IDEs

In addition to the links directly below, also make sure to check out the various online REPL/playground tools linked under various other parts of this page too (eg. babel, swc, etc):

• https://github.com/microsoft/TypeScript-Website/tree/v2/packages/playground

  • This is the JS tooling which powers the https://www.typescriptlang.org/play/

  • It is more or less vanilla DOM-oriented JavaScript with as few dependencies as possible. Originally based on the work by Artem Tyurin but now it's diverged far from that fork.

    • https://github.com/agentcooper/typescript-play
  • https://github.com/microsoft/TypeScript-Website/tree/v2/packages/sandbox

    • The TypeScript Sandbox is the editor part of the TypeScript Playground. It's effectively an opinionated fork of monaco-typescript with extra extension points so that projects like the TypeScript Playground can exist.

  • https://github.com/microsoft/TypeScript-Playground-Samples

    • Examples of TypeScript Playground Plugins for you to work from

    • This is a series of example plugins, which are extremely well documented and aim to give you samples to build from depending on what you want to build.

    • • TS Compiler API: Uses @typescript/vfs to set up a TypeScript project in the browser, and then displays all of the top-level functions as AST nodes in the sidebar.
      • TS Transformers Demo: Uses a custom TypeScript transformer when emitting JavaScript from the current file in the Playground.
      • Using a Web-ish npm Dependency: Uses a dependency which isn't entirely optimised for running in a web page, but doesn't have too big of a dependency tree that it this becomes an issue either
      • Presenting Information Inline: Using a fraction of the extensive Monaco API (monaco is the text editor at the core of the Playground) to showcase what parts of a TypeScript file would be removed by a transpiler to make it a JS file.

CodeMirror

• https://codemirror.net/

  • CodeMirror is a code editor component for the web. It can be used in websites to implement a text input field with support for many editing features, and has a rich programming interface to allow further extension.

  • CodeMirror is open source under a permissive license (MIT).

  • A full parser package, often with language-specific integration and extension code, exists for the following languages

    • https://github.com/codemirror/lang-javascript

      • JavaScript language support for the CodeMirror code editor

    • etc

  • There is also a collection of CodeMirror 5 modes that can be used, and a list of community-maintained language packages. If your language is not listed above, you may still find a solution there.

    • https://github.com/codemirror/legacy-modes

      • Collection of ported legacy language modes for the CodeMirror code editor

    • https://codemirror.net/docs/community/#language
  • https://codemirror.net/docs/community/

    • Community Packages

    • This page lists CodeMirror-related packages maintained by the wider community.

• https://github.com/codemirror/dev

  • Development repository for the CodeMirror editor project

  • This is the central repository for CodeMirror. It holds the bug tracker and development scripts.

    If you want to use CodeMirror, install the separate packages from npm, and ignore the contents of this repository. If you want to develop on CodeMirror, this repository provides scripts to install and work with the various packages.

• https://github.com/uiwjs/react-codemirror

  • CodeMirror 6 component for React

  • https://uiwjs.github.io/react-codemirror/

monaco-editor

• https://microsoft.github.io/monaco-editor/

  • The Monaco Editor is the code editor that powers VS Code.

  • It is licensed under the MIT License and supports Edge, Chrome, Firefox, Safari and Opera. The Monaco editor is not supported in mobile browsers or mobile web frameworks.

  • https://microsoft.github.io/monaco-editor/docs.html
  • https://microsoft.github.io/monaco-editor/playground.html
  • https://microsoft.github.io/monaco-editor/monarch.html

    • This document describes how to create a syntax highlighter using the Monarch library. This library allows you to specify an efficient syntax highlighter, using a declarative lexical specification (written as a JSON value). The specification is expressive enough to specify sophisticated highlighters with complex state transitions, dynamic brace matching, auto-completion, other language embeddings, etc. as shown in the 'advanced' topic sections of this document.

  • See also: https://code.visualstudio.com/docs/editor/vscode-web
• https://github.com/microsoft/monaco-editor

  • A browser based code editor

    • The Monaco Editor is the fully featured code editor from VS Code. Check out the VS Code docs to see some of the supported features.

      • https://code.visualstudio.com/docs/editor/editingevolved
        • https://code.visualstudio.com/docs/editor/editingevolved#_go-to-definition

          • Go to Definition

          • If a language supports it, you can go to the definition of a symbol by pressing F12.

        • https://code.visualstudio.com/docs/editor/editingevolved#_go-to-type-definition

          • Go to Type Definition

          • Some languages also support jumping to the type definition of a symbol by running the Go to Type Definition command from either the editor context menu or the Command Palette. This will take you to the definition of the type of a symbol.

        • https://code.visualstudio.com/docs/editor/editingevolved#_go-to-implementation

          • Go to Implementation

          • Languages can also support jumping to the implementation of a symbol by pressing ⌘F12. For an interface, this shows all the implementors of that interface and for abstract methods, this shows all concrete implementations of that method.

        • https://code.visualstudio.com/docs/editor/editingevolved#_go-to-symbol

          • Go to Symbol

          • You can navigate symbols inside a file with ⇧⌘O. By typing : the symbols will be grouped by category. Press Up or Down and navigate to the place you want.

        • https://code.visualstudio.com/docs/editor/editingevolved#_open-symbol-by-name

          • Open symbol by name

          • Some languages support jumping to a symbol across files with ⌘T. Type the first letter of a type you want to navigate to, regardless of which file contains it, and press Enter.

        • https://code.visualstudio.com/docs/editor/editingevolved#_peek

          • Peek

          • We think there's nothing worse than a big context switch when all you want is to quickly check something. That's why we support peeked editors. When you execute a Go to References search (via ⇧F12), or a Peek Definition (via ⌥F12), we embed the result inline

        • https://code.visualstudio.com/docs/editor/editingevolved#_bracket-matching

          • Bracket matching

          • Matching brackets will be highlighted as soon as the cursor is near one of them.

        • https://code.visualstudio.com/docs/editor/editingevolved#_reference-information

          • Reference information

          • Some languages like C# support inline reference information, that is updated live. This allows you to quickly analyze the impact of your edit or the popularity of your specific method or property throughout your project:

        • https://code.visualstudio.com/docs/editor/editingevolved#_rename-symbol

          • Rename symbol

          • Some languages support rename symbol across files. Press F2 and then type the new desired name and press Enter. All usages of the symbol will be renamed, across files.

        • https://code.visualstudio.com/docs/editor/editingevolved#_inlay-hints

          • Inlay Hints

          • Some languages provide inlay hints: that is additional information about source code that is rendered inline. This is usually used to show infered types.

        • etc

Obfuscation / Deobfuscation

• https://github.com/terser/terser

  • JavaScript parser, mangler and compressor toolkit for ES6+

  • https://terser.org/
  • https://try.terser.org/ (REPL, Playground)
  • https://terser.org/docs/options/#compress-options

    • Compress options

    • keep_classnames, keep_fnames, module, toplevel, etc
  • https://terser.org/docs/options/#mangle-options

    • Mangle options

    • keep_classnames, keep_fnames, module, toplevel, etc
  • https://terser.org/docs/api-reference/

    • mangle (default true) — pass false to skip mangling names, or pass an object to specify mangle options

    • module (default false) — Use when minifying an ES6 module. "use strict" is implied and names can be mangled on the top scope. If compress or mangle is enabled then the toplevel option will be enabled.

    • nameCache (default null) - pass an empty object {} or a previously used nameCache object if you wish to cache mangled variable and property names across multiple invocations of minify(). Note: this is a read/write property. minify() will read the name cache state of this object and update it during minification so that it may be reused or externally persisted by the user.

    • keep_classnames (default: undefined) - pass true to prevent discarding or mangling of class names. Pass a regular expression to only keep class names matching that regex.

    • keep_fnames (default: false) - pass true to prevent discarding or mangling of function names. Pass a regular expression to only keep function names matching that regex. Useful for code relying on Function.prototype.name. If the top level minify option keep_classnames is undefined it will be overridden with the value of the top level minify option keep_fnames.

  • https://github.com/terser/terser/blob/main/lib/minify.js#L176-L200

    • var toplevel;
      if (files instanceof AST_Toplevel) {
          toplevel = files;
      } else {
          if (typeof files == "string") {
              files = [ files ];
          }

    • // disable rename on harmony due to expand_names bug in for-of loops
      // https://github.com/mishoo/UglifyJS2/issues/2794
      if (0 && options.rename) {
        toplevel.figure_out_scope(options.mangle);
        toplevel.expand_names(options.mangle);
      }

      • https://github.com/terser/terser/blob/master/lib/scope.js#L203-L479
        • AST_Scope.DEFMETHOD("figure_out_scope", function(options, { parent_scope = null, toplevel = this } = {}) {
      • https://github.com/terser/terser/blob/master/lib/scope.js#L930-L966
        • AST_Toplevel.DEFMETHOD("expand_names", function(options) {
      • https://github.com/terser/terser/blob/master/lib/scope.js#L907-L928
        • AST_Toplevel.DEFMETHOD("find_colliding_names", function(options) {

    • if (options.compress) {
        toplevel = new Compressor(options.compress, {
          mangle_options: options.mangle
        }).compress(toplevel);
      }

      • https://github.com/terser/terser/blob/master/lib/compress/index.js#L213
        • class Compressor extends TreeWalker {
      • TODO: add reference to compress function here

    • if (options.mangle) toplevel.figure_out_scope(options.mangle);`

      • https://github.com/terser/terser/blob/master/lib/scope.js#L203-L479
        • AST_Scope.DEFMETHOD("figure_out_scope", function(options, { parent_scope = null, toplevel = this } = {}) {

    • if (options.mangle) {
        base54.reset();
        toplevel.compute_char_frequency(options.mangle);
        toplevel.mangle_names(options.mangle);
      }

      • https://github.com/terser/terser/blob/master/lib/scope.js#L1015-L1061
        • base54
      • https://github.com/terser/terser/blob/master/lib/scope.js#L973-L1013
        • AST_Toplevel.DEFMETHOD("compute_char_frequency", function(options) {
      • https://github.com/terser/terser/blob/master/lib/scope.js#L806-L905
        • AST_Toplevel.DEFMETHOD("mangle_names", function(options) {

    • if (options.mangle && options.mangle.properties) {
        toplevel = mangle_properties(toplevel, options.mangle.properties);
      }

      • https://github.com/terser/terser/blob/master/lib/propmangle.js#L152
        • function mangle_private_properties(ast, options) {
      • https://github.com/terser/terser/blob/master/lib/propmangle.js#L216-L425
        • function mangle_properties(ast, options, annotated_props = find_annotated_props(ast)) {

        • options = defaults(options, {
            // ..snip..
            nth_identifier: base54,
            // ..snip..
          }, true);

        • var nth_identifier = options.nth_identifier;
        • https://github.com/terser/terser/blob/master/lib/propmangle.js#L381-L407
          • function mangle(name) {

          • // debug mode: use a prefix and suffix to preserve readability, e.g. o.foo -> o._$foo$NNN_.
            var debug_mangled = "_$" + name + "$" + debug_name_suffix + "_";

          • if (!mangled) {
                do {
                    mangled = nth_identifier.get(++cname);
                } while (!can_mangle(mangled));
            }
            
            cache.set(name, mangled);

        • https://github.com/terser/terser/blob/master/lib/propmangle.js#L409-L424
          • function mangleStrings(node) {
• https://github.com/javascript-obfuscator/javascript-obfuscator

  • A powerful obfuscator for JavaScript and Node.js

  • https://obfuscator.io/
• https://github.com/MichaelXF/js-confuser

  • JS-Confuser is a JavaScript obfuscation tool to make your programs impossible to read.

  • https://js-confuser.com/
  • https://github.com/MichaelXF/js-confuser/blob/master/docs/Integrity.md

    • JSConfuser can detect changes to the source and prevent execution. If the code is determined modified, the tampered code will not run.

    JavaScript has a sneaky method to view the source code any function. Calling .toString() on any function reveals the raw source code. Integrity hashes the code during obfuscation phase and embeds an IF-statement within the code. We used an additional regex to remove spaces, newlines, braces, and commas to ensure the hash isn't too sensitive.

• https://github.com/ben-sb/javascript-deobfuscator

  • General purpose JavaScript deobfuscator

  • https://deobfuscate.io/
• https://github.com/ben-sb/obfuscator-io-deobfuscator

  • A deobfuscator for scripts obfuscated by Obfuscator.io

  • https://obf-io.deobfuscate.io/
• https://github.com/relative/synchrony

  • javascript cleaner & deobfuscator (primarily javascript-obfuscator/obfuscator.io)

  • https://deobfuscate.relative.im/

Variable Name Mangling

• https://hex-rays.com/blog/igors-tip-of-the-week-34-dummy-names/

  • In IDA’s disassembly, you may have often observed names that may look strange and cryptic on first sight: sub_73906D75, loc_40721B, off_40A27C and more. In IDA’s terminology, they’re called dummy names. They are used when a name is required by the assembly syntax but there is nothing suitable available

  • https://www.hex-rays.com/products/ida/support/idadoc/609.shtml

    • IDA Help: Names Representation

    • Dummy names are automatically generated by IDA. They are used to denote subroutines, program locations and data. Dummy names have various prefixes depending on the item type and value

• https://binary.ninja/2023/09/15/3.5-expanded-universe.html#automatic-variable-naming

  • Automatic Variable Naming One easy way to improve decompilation output is to come up with better default names for variables. There’s a lot of possible defaults you could choose and a number of different strategies are seen throughout different reverse engineering tools. Prior to 3.5, Binary Ninja left variables named based on their origin. Stack variables were var_OFFSET, register-based variables were reg_COUNTER, and global data variables were (data_). While this scheme isn’t changing, we’re being much more intelligent about situations where additional information is available.

    For example, if a variable is passed to a function and a variable name is available, we can now make a much better guess for the variable name. This is most obvious in binaries with type libraries.

  • This isn’t the only style of default names. Binary Ninja also will name loop counters with simpler names like i, or j, k, etc (in the case of nested loops)

• Webpack
  • https://webpack.js.org/configuration/optimization/#optimizationchunkids

    • optimization.chunkIds

    • Tells webpack which algorithm to use when choosing chunk ids.

    • • 'natural': Numeric ids in order of usage.
      • 'named': Readable ids for better debugging.
      • 'deterministic': Short numeric ids which will not be changing between compilation. Good for long term caching. Enabled by default for production mode.
      • 'size': Numeric ids focused on minimal initial download size.
      • 'total-size': numeric ids focused on minimal total download size.

  • https://webpack.js.org/configuration/optimization/#optimizationmangleexports

    • optimization.mangleExports allows to control export mangling. By default optimization.mangleExports: 'deterministic' is enabled in production mode and disabled elsewise.

    • • 'size': Short names - usually a single char - focused on minimal download size.
      • 'deterministic': Short names - usually two chars - which will not change when adding or removing exports. Good for long term caching.
      • true: Same as 'deterministic'
      • false: Keep original name. Good for readability and debugging.

  • https://webpack.js.org/configuration/optimization/#optimizationmanglewasmimports
    • optimization.mangleWasmImports

      • When set to true tells webpack to reduce the size of WASM by changing imports to shorter strings. It mangles module and export names.

  • https://webpack.js.org/configuration/optimization/#optimizationminimize
    • optimization.minimize

      • Tell webpack to minimize the bundle using the TerserPlugin or the plugin(s) specified in optimization.minimizer

        • https://webpack.js.org/plugins/terser-webpack-plugin/

          • TerserWebpackPlugin

          • This plugin uses terser to minify/minimize your JavaScript.

          • https://webpack.js.org/plugins/terser-webpack-plugin/#terseroptions
            • keep_classnames, keep_fnames, mangle, nameCache, etc
          • https://github.com/terser/terser
  • https://webpack.js.org/configuration/optimization/#optimizationminimizer

    • optimization.minimizer

    • Allows you to override the default minimizer by providing a different one or more customized TerserPlugin instances.

  • https://webpack.js.org/configuration/optimization/#optimizationmoduleids

    • optimization.moduleIds

    • Tells webpack which algorithm to use when choosing module ids. Setting optimization.moduleIds to false tells webpack that none of built-in algorithms should be used, as custom one can be provided via plugin.

    • • natural: Numeric ids in order of usage.
      • named: Readable ids for better debugging.
      • deterministic: Module names are hashed into small numeric values.
      • size: Numeric ids focused on minimal initial download size.

    • deterministic option is useful for long term caching, but still results in smaller bundles compared to hashed. Length of the numeric value is chosen to fill a maximum of 80% of the id space. By default a minimum length of 3 digits is used when optimization.moduleIds is set to deterministic. To override the default behaviour set optimization.moduleIds to false and use the webpack.ids.DeterministicModuleIdsPlugin
• https://github.com/estools/esshorten

  • Shorten (mangle) names in JavaScript code

  • https://github.com/estools/esshorten/blob/master/lib/esshorten.js#L88-L138
    • NameGenerator, mangle, mangleLabels
  • https://github.com/estools/esshorten/blob/master/lib/utility.js#L64-L92
    • generateNextName

Symbolic / Concolic Execution

• https://en.wikipedia.org/wiki/Symbolic_execution

  • In computer science, symbolic execution (also symbolic evaluation or symbex) is a means of analyzing a program to determine what inputs cause each part of a program to execute. An interpreter follows the program, assuming symbolic values for inputs rather than obtaining actual inputs as normal execution of the program would. It thus arrives at expressions in terms of those symbols for expressions and variables in the program, and constraints in terms of those symbols for the possible outcomes of each conditional branch. Finally, the possible inputs that trigger a branch can be determined by solving the constraints.

  • https://en.wikipedia.org/wiki/Symbolic_execution#Tools
  • https://en.wikipedia.org/wiki/Symbolic_execution#See_also

    • Abstract interpretation

    • Symbolic simulation

    • Symbolic computation

    • Concolic testing

    • Control-flow graph

    • Dynamic recompilation

• https://en.wikipedia.org/wiki/Concolic_testing

  • Concolic testing (a portmanteau of concrete and symbolic, also known as dynamic symbolic execution) is a hybrid software verification technique that performs symbolic execution, a classical technique that treats program variables as symbolic variables, along a concrete execution (testing on particular inputs) path. Symbolic execution is used in conjunction with an automated theorem prover or constraint solver based on constraint logic programming to generate new concrete inputs (test cases) with the aim of maximizing code coverage. Its main focus is finding bugs in real-world software, rather than demonstrating program correctness.

  • Implementation of traditional symbolic execution based testing requires the implementation of a full-fledged symbolic interpreter for a programming language. Concolic testing implementors noticed that implementation of full-fledged symbolic execution can be avoided if symbolic execution can be piggy-backed with the normal execution of a program through instrumentation. This idea of simplifying implementation of symbolic execution gave birth to concolic testing.

  • An important reason for the rise of concolic testing (and more generally, symbolic-execution based analysis of programs) in the decade since it was introduced in 2005 is the dramatic improvement in the efficiency and expressive power of SMT Solvers. The key technical developments that lead to the rapid development of SMT solvers include combination of theories, lazy solving, DPLL(T) and the huge improvements in the speed of SAT solvers. SMT solvers that are particularly tuned for concolic testing include Z3, STP, Z3str2, and Boolector.

    • https://en.wikipedia.org/wiki/Satisfiability_modulo_theories

      • In computer science and mathematical logic, satisfiability modulo theories (SMT) is the problem of determining whether a mathematical formula is satisfiable. It generalizes the Boolean satisfiability problem (SAT) to more complex formulas involving real numbers, integers, and/or various data structures such as lists, arrays, bit vectors, and strings. The name is derived from the fact that these expressions are interpreted within ("modulo") a certain formal theory in first-order logic with equality (often disallowing quantifiers). SMT solvers are tools that aim to solve the SMT problem for a practical subset of inputs. SMT solvers such as Z3 and cvc5 have been used as a building block for a wide range of applications across computer science, including in automated theorem proving, program analysis, program verification, and software testing.

    • https://en.wikipedia.org/wiki/Boolean_satisfiability_problem#Algorithms_for_solving_SAT
  • https://en.wikipedia.org/wiki/Concolic_testing#Algorithm

    • Essentially, a concolic testing algorithm operates as follows:

      • Classify a particular set of variables as input variables. These variables will be treated as symbolic variables during symbolic execution. All other variables will be treated as concrete values.
      • Instrument the program so that each operation which may affect a symbolic variable value or a path condition is logged to a trace file, as well as any error that occurs.
      • Choose an arbitrary input to begin with.
      • Execute the program.
      • Symbolically re-execute the program on the trace, generating a set of symbolic constraints (including path conditions).
      • Negate the last path condition not already negated in order to visit a new execution path. If there is no such path condition, the algorithm terminates.
      • Invoke an automated satisfiability solver on the new set of path conditions to generate a new input. If there is no input satisfying the constraints, return to step 6 to try the next execution path.
      • Return to step 4.

      There are a few complications to the above procedure:

      • The algorithm performs a depth-first search over an implicit tree of possible execution paths. In practice programs may have very large or infinite path trees – a common example is testing data structures that have an unbounded size or length. To prevent spending too much time on one small area of the program, the search may be depth-limited (bounded).
      • Symbolic execution and automated theorem provers have limitations on the classes of constraints they can represent and solve. For example, a theorem prover based on linear arithmetic will be unable to cope with the nonlinear path condition xy = 6. Any time that such constraints arise, the symbolic execution may substitute the current concrete value of one of the variables to simplify the problem. An important part of the design of a concolic testing system is selecting a symbolic representation precise enough to represent the constraints of interest.

  • https://en.wikipedia.org/wiki/Concolic_testing#Tools
    • Jalangi is an open-source concolic testing and symbolic execution tool for JavaScript. Jalangi supports integers and strings.
• https://github.com/Z3Prover/z3

  • The Z3 Theorem Prover

  • https://github.com/Z3Prover/z3/wiki

    • Z3 is an SMT solver and supports the SMTLIB format.

      • https://smtlib.cs.uiowa.edu/

        • SMT-LIB is an international initiative aimed at facilitating research and development in Satisfiability Modulo Theories (SMT).

        • Documents describing the SMT-LIB input/output language for SMT solvers and its semantics;

        • etc
  • https://microsoft.github.io/z3guide/

    • Online Z3 Guide

    • https://github.com/microsoft/z3guide

      • Tutorials and courses for Z3

      • https://microsoft.github.io/z3guide/docs/logic/intro/

        • Introduction Z3 is a state-of-the art theorem prover from Microsoft Research. It can be used to check the satisfiability of logical formulas over one or more theories. Z3 offers a compelling match for software analysis and verification tools, since several common software constructs map directly into supported theories.

          The main objective of the tutorial is to introduce the reader on how to use Z3 effectively for logical modeling and solving. The tutorial provides some general background on logical modeling, but we have to defer a full introduction to first-order logic and decision procedures to text-books in order to develop an in depth understanding of the underlying concepts. To clarify: a deep understanding of logical modeling is not necessarily required to understand this tutorial and modeling with Z3, but it is necessary to understand for writing complex models.

      • https://microsoft.github.io/z3guide/programming/Z3%20JavaScript%20Examples/

        • Z3 JavaScript The Z3 distribution comes with TypeScript (and therefore JavaScript) bindings for Z3. In the following we give a few examples of using Z3 through these bindings. You can run and modify the examples locally in your browser.

• https://github.com/Samsung/jalangi2

  • Dynamic analysis framework for JavaScript

  • Jalangi2 is a framework for writing dynamic analyses for JavaScript. Jalangi1 is still available at https://github.com/SRA-SiliconValley/jalangi, but we no longer plan to develop it. Jalangi2 does not support the record/replay feature of Jalangi1. In the Jalangi2 distribution you will find several analyses:

    • an analysis to track NaNs.
    • an analysis to check if an undefined is concatenated to a string.
    • Memory analysis: a memory-profiler for JavaScript and HTML5.
    • DLint: a dynamic checker for JavaScript bad coding practices.
    • JITProf: a dynamic JIT-unfriendly code snippet detection tool.
    • analysisCallbackTemplate.js: a template for writing a dynamic analysis.
    • and more ...

    See our tutorial slides for a detailed overview of Jalangi and some client analyses.

  • https://github.com/Samsung/jalangi2#usage

    • Usage

    • Analysis in node.js with on-the-fly instrumentation

    • Analysis in node.js with explicit one-file-at-a-time offline instrumentation

    • Analysis in a browser using a proxy and on-the-fly instrumentation

• https://github.com/SRA-SiliconValley/jalangi

  • This repository has been archived by the owner on Dec 9, 2017. It is now read-only.

  • We encourage you to switch to Jalangi2 available at https://github.com/Samsung/jalangi2. Jalangi2 is a framework for writing dynamic analyses for JavaScript. Jalangi2 does not support the record/replay feature of Jalangi1. Jalangi1 is still available from this website, but we no longer plan to develop it.

  • Jalangi is a framework for writing heavy-weight dynamic analyses for JavaScript. Jalangi provides two modes for dynamic program analysis: an online mode (a.k.a direct or inbrowser analysis mode)and an offilne mode (a.k.a record-replay analysis mode). In both modes, Jalangi instruments the program-under-analysis to insert callbacks to methods defined in Jalangi. An analysis writer implements these methods to perform custom dynamic program analysis. In the online mode, Jalangi performs analysis during the execution of the program. An analysis in online mode can use shadow memory to attach meta information with every memory location. The offilne mode of Jalangi incorporates two key techniques: 1) selective record-replay, a technique which enables to record and to faithfully replay a user-selected part of the program, and 2) shadow values and shadow execution, which enables easy implementation of heavy-weight dynamic analyses. Shadow values allow an analysis to attach meta information with every value. In the distribution you will find several analyses:

    • concolic testing,
    • an analysis to track origins of nulls and undefined,
    • an analysis to infer likely types of objects fields and functions,
    • an analysis to profile object allocation and usage,
    • a simple form of taint analysis,
    • an experimental pure symbolic execution engine (currently undocumented)

Profiling

• See also, the notes from ChatGPT (Ref) that I captured in this comment: pionxzh/wakaru#35 (comment)
  • https://nodejs.org/en/docs/guides/diagnostics-flamegraph
  • https://github.com/davidmarkclements/0x

    • single-command flamegraph profiling 🔥 Discover the bottlenecks and hot paths in your code, with flamegraphs

  • https://github.com/clinicjs/node-clinic

    • Clinic.js diagnoses your Node.js performance issues

    • https://clinicjs.org/

      • Tools to help diagnose and pinpoint Node.js performance issues

      • https://clinicjs.org/doctor/

        • Clinic.js Doctor: Diagnose performance issues in your Node.js applications

      • https://clinicjs.org/bubbleprof/

        • Clinic.js Bubbleprof: Bubbleprof is a new, completely unique, approach to profiling your Node.js code

        • https://clinicjs.org/blog/introducing-bubbleprof/

          • Introducing Bubbleprof - a novel approach to Node.js async profiling

      • https://clinicjs.org/flame/

        • Clinic.js Flame: Uncovers the bottlenecks and hot paths in your code with flamegraphs

      • https://clinicjs.org/heapprofiler/
        • Clinic.js HeapProfiler: Uncovers memory allocations by functions with Flamegraphs.

Unsorted

• https://github.com/bytecodealliance/ComponentizeJS

  • ESM -> WebAssembly Component creator, via a SpiderMonkey JS engine embedding

  • Provides a Mozilla SpiderMonkey embedding that takes as input a JavaScript source file and a WebAssembly Component WIT World, and outputs a WebAssembly Component binary with the same interface.

  • https://bytecodealliance.org/articles/making-javascript-run-fast-on-webassembly

    • Making JavaScript run fast on WebAssembly

    • We should be clear here—if you’re running JavaScript in the browser, it still makes the most sense to simply deploy JS. The JS engines within the browsers are highly tuned to run the JS that gets shipped to them.

  • https://github.com/bytecodealliance/wizer

    • The WebAssembly Pre-Initializer Don't wait for your Wasm module to initialize itself, pre-initialize it! Wizer instantiates your WebAssembly module, executes its initialization function, and then snapshots the initialized state out into a new WebAssembly module. Now you can use this new, pre-initialized WebAssembly module to hit the ground running, without making your users wait for that first-time set up code to complete.

      The improvements to start up latency you can expect will depend on how much initialization work your WebAssembly module needs to do before it's ready. Some initial benchmarking shows between 1.35 to 6.00 times faster instantiation and initialization with Wizer, depending on the workload

• https://wingolog.org/archives/2022/08/18/just-in-time-code-generation-within-webassembly

  • just-in-time code generation within webassembly

• https://github.com/WebAssembly/wabt

  • The WebAssembly Binary Toolkit

  • WABT (we pronounce it "wabbit") is a suite of tools for WebAssembly, including:

    • wat2wasm: translate from WebAssembly text format to the WebAssembly binary format
    • wasm2wat: the inverse of wat2wasm, translate from the binary format back to the text format (also known as a .wat)
    • wasm-objdump: print information about a wasm binary. Similiar to objdump.
    • wasm-interp: decode and run a WebAssembly binary file using a stack-based interpreter
    • wasm-decompile: decompile a wasm binary into readable C-like syntax.
    • wat-desugar: parse .wat text form as supported by the spec interpreter (s-expressions, flat syntax, or mixed) and print "canonical" flat format
    • wasm2c: convert a WebAssembly binary file to a C source and header
    • wasm-strip: remove sections of a WebAssembly binary file
    • wasm-validate: validate a file in the WebAssembly binary format
    • wast2json: convert a file in the wasm spec test format to a JSON file and associated wasm binary files
    • wasm-stats: output stats for a module
    • spectest-interp: read a Spectest JSON file, and run its tests in the interpreter

    These tools are intended for use in (or for development of) toolchains or other systems that want to manipulate WebAssembly files. Unlike the WebAssembly spec interpreter (which is written to be as simple, declarative and "speccy" as possible), they are written in C/C++ and designed for easier integration into other systems. Unlike Binaryen these tools do not aim to provide an optimization platform or a higher-level compiler target; instead they aim for full fidelity and compliance with the spec (e.g. 1:1 round-trips with no changes to instructions).

My ChatGPT Research / Conversations

These are private chat links, so won't work for others, and are included here only for my reference:

• Webpack Minification Noise

See Also

My Other Related Deepdive Gist's and Projects

• https://github.com/0xdevalias/chatgpt-source-watch : Analyzing the evolution of ChatGPT's codebase through time with curated archives and scripts.
• Debugging Electron Apps (and related memory issues) (0xdevalias gist)
• Reverse Engineering Webpack Apps (0xdevalias gist)
• Reverse Engineered Webpack Tailwind-Styled-Component (0xdevalias gist)
• Reverse engineering ChatGPT's frontend web app + deep dive explorations of the code (0xdevalias gist)
• Bypassing Cloudflare, Akamai, etc (0xdevalias gist)
• devalias' Beeper CSS Hacks (0xdevalias gist)
• Reverse Engineering Golang (0xdevalias' gist)
• Reverse Engineering on macOS (0xdevalias' gist)

chrome-devtools-sources-extension.md

Chrome DevTools 'Sources' Extension

Originally articulated here:

• j4k0xb/webcrack#29

  • add Chrome DevTools extension that allows the web IDE to be used within DevTools

• pionxzh/wakaru#76

  • add Chrome DevTools extension that allows the web IDE to be used within DevTools

Overview

The following is as I originally wrote it on the above issues, copied here for posterity:

First off, I LOVE the new v2.11.0 update, and the changes to the web IDE to use monaco to support 'references', 'go to declaration', etc.

This issue is about an idea I've had and wanted to work on myself for a while, but haven't got around to it yet. Often when exploring web apps, I make pretty heavy use of Chrome DevTools' 'Search' / 'Sources' / 'Network' / etc tabs, the debugger, and the console / utilities APIs. While there have been some nice improvements to the 'Sources' tab over the years (pretty printing, syntax highlighting, code folding, etc); one area I have really wished it was able to support for a long time now is 'references' / 'go to definition' / similar.

A thought I had in this space is that, while I obviously can't outright replace the 'Sources' tab (which I believe is based on CodeMirror), it should be possible to create a Chrome DevTools Extension that adds a new tab/panel that does similar to the current 'Sources' tab, but using monaco as it's base, and enabling features like 'references' / 'go to definition' / etc within that.

Useful Chrome Extension APIs

Overview of extending DevTools:

• https://developer.chrome.com/docs/extensions/how-to/devtools/extend-devtools

  • Extending DevTools DevTools extensions add functionality to Chrome DevTools by accessing DevTools-specific extension APIs through a DevTools page added to the extension.

Some of the Chrome Extension API's that would be useful/enable this:

• https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow

  • Use the chrome.devtools.inspectedWindow API to interact with the inspected window: obtain the tab ID for the inspected page, evaluate the code in the context of the inspected window, reload the page, or obtain the list of resources within the page.

  • Use the getResources call and the onResourceContent event to obtain the list of resources (documents, stylesheets, scripts, images etc) within the inspected page. The getContent and setContent methods of the Resource class along with the onResourceContentCommitted event may be used to support modification of the resource content, for example, by an external editor.

  • https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#executing-code

    • The execution context of the code being evaluated includes the Developer Tools console API. For example, the code can use inspect and $0.

    • https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#method-eval

      • Evaluates a JavaScript expression in the context of the main frame of the inspected page.

      • This can also be used to target different frameURLs, different scriptExecutionContexts, etc
  • https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#method-getResources

    • Retrieves the list of resources from the inspected page.

    • https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#type-Resource

      • A resource within the inspected page, such as a document, a script, or an image.

      • getContent: Gets the content of the resource. (potentially encoded, Currently, only base64 is supported.).

      • setContent: Sets the content of the resource. Only resources with the text type are currently supported.

  • https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#method-reload

    • Reloads the inspected page.

    • The most interesting part of this API is probably reloadOptions -> injectedScript (though userAgent might be useful sometimes too):

      • injectedScript: If specified, the script will be injected into every frame of the inspected page immediately upon load, before any of the frame's scripts.

      • userAgent: If specified, the string will override the value of the User-Agent HTTP header that's sent while loading the resources of the inspected page. The string will also override the value of the navigator.userAgent property that's returned to any scripts that are running within the inspected page.

  • https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#event-onResourceAdded

    • Fired when a new resource is added to the inspected page.

  • https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#event-onResourceContentCommitted

    • Fired when a new revision of the resource is committed (e.g. user saves an edited version of the resource in the Developer Tools).

• https://developer.chrome.com/docs/extensions/reference/api/devtools/panels

  • Use the chrome.devtools.panels API to integrate your extension into Developer Tools window UI: create your own panels, access existing panels, and add sidebars.

  • https://developer.chrome.com/docs/extensions/reference/api/devtools/panels#type-SourcesPanel

    • Represents the Sources panel

    • I hadn't stumbled upon this before, but it sounds like it's possible to access the existing 'Sources' panel; can react to 'when an object is selected in the panel' (not sure what 'object' refers to in this case), can create a sidebar pane, etc
  • https://developer.chrome.com/docs/extensions/reference/api/devtools/panels#method-create

    • Creates an extension panel

    • This was my main original thought, in creating a new custom panel
  • https://developer.chrome.com/docs/extensions/reference/api/devtools/panels#method-openResource

    • Requests DevTools to open a URL in a Developer Tools panel.

    • Can seemingly be used to interact with existing panels
  • https://developer.chrome.com/docs/extensions/reference/api/devtools/panels#method-setOpenResourceHandler

    • Specifies the function to be called when the user clicks a resource link in the Developer Tools window. To unset the handler, either call the method with no parameters or pass null as the parameter.

• https://developer.chrome.com/docs/extensions/reference/api/devtools/network
  • Use the chrome.devtools.network API to retrieve the information about network requests displayed by the Developer Tools in the Network panel.
  • Probably less specifically relevant to the functionality I was thinking of, but could be some value/crossover there, so figured I'd mention it. onNavigated / onRequestFinished might be useful in some cases.
• https://developer.chrome.com/docs/extensions/reference/api/devtools/recorder

  • Use the chrome.devtools.recorder API to customize the Recorder panel in DevTools. devtools.recorder API is a preview feature that allows you to extend the Recorder panel in Chrome DevTools.

  • Again, probably not as useful, but linking here in case.
• https://developer.chrome.com/docs/extensions/reference/api/debugger

  • The chrome.debugger API serves as an alternate transport for Chrome's remote debugging protocol. Use chrome.debugger to attach to one or more tabs to instrument network interaction, debug JavaScript, mutate the DOM and CSS, etc.

  • For security reasons, the chrome.debugger API does not provide access to all Chrome DevTools Protocol Domains. The available domains are: Accessibility, Audits, CacheStorage, Console, CSS, Database, Debugger, DOM, DOMDebugger, DOMSnapshot, Emulation, Fetch, IO, Input, Inspector, Log, Network, Overlay, Page, Performance, Profiler, Runtime, Storage, Target, Tracing, WebAudio, and WebAuthn.

  • This can do all sorts of powerful stuff, though some of the original reasons I was thinking of needing to use it (eg. getting access to the page's script contents/etc) are apparently already covered in other more specific APIs (eg. chrome.devtools.inspectedWindow). Still worth being aware of it and the features it has though, as there is some pretty cool stuff here!
  • https://chromedevtools.github.io/devtools-protocol/

Then there are also all of the 'standard' Chrome Extension APIs as well, which can do a lot of cool stuff:

• https://developer.chrome.com/docs/extensions/reference/api

A few of which could be useful for this feature:

• https://developer.chrome.com/docs/extensions/reference/api/runtime

  • Use the chrome.runtime API to retrieve the service worker, return details about the manifest, and listen for and respond to events in the app or extension lifecycle. You can also use this API to convert the relative path of URLs to fully-qualified URLs.

  • Some more esotetic, but interesting aspects of this, are the ability to communicate with other externsions and/or locally installed programs using connect / onConnectExternal / onMessageExternal / connectNative / sendMessageNative:
    • https://developer.chrome.com/docs/extensions/reference/api/runtime#method-connect

      • Attempts to connect listeners within an extension/app (such as the background page), or other extensions/apps. This is useful for content scripts connecting to their extension processes, inter-app/extension communication, and web messaging. Note that this does not connect to any listeners in a content script. Extensions may connect to content scripts embedded in tabs via tabs.connect.

    • https://developer.chrome.com/docs/extensions/reference/api/runtime#event-onConnectExternal

      • Fired when a connection is made from another extension (by runtime.connect).

    • https://developer.chrome.com/docs/extensions/reference/api/runtime#event-onMessageExternal

      • Fired when a message is sent from another extension/app (by runtime.sendMessage).

    • https://developer.chrome.com/docs/extensions/reference/api/runtime#method-connectNative

      • Connects to a native application in the host machine. See Native Messaging for more information.

    • https://developer.chrome.com/docs/extensions/reference/api/runtime#method-sendNativeMessage

      • Send a single message to a native application.

    • https://developer.chrome.com/docs/extensions/reference/api/runtime#event-onConnectNative

      • Fired when a connection is made from a native application. Currently only supported on Chrome OS.

      • While this is only supported from Chrome OS, I believe the browser can always initiate the connection to a native app.
• https://developer.chrome.com/docs/extensions/reference/api/permissions

  • Use the chrome.permissions API to request declared optional permissions at run time rather than install time, so users understand why the permissions are needed and grant only those that are necessary.

• https://developer.chrome.com/docs/extensions/reference/api/contextMenus

  • Use the chrome.contextMenus API to add items to Google Chrome's context menu. You can choose what types of objects your context menu additions apply to, such as images, hyperlinks, and pages.

• https://developer.chrome.com/docs/extensions/reference/api/sidePanel

  • Use the chrome.sidePanel API to host content in the browser's side panel alongside the main content of a webpage.

• https://developer.chrome.com/docs/extensions/reference/api/storage

  • Use the chrome.storage API to store, retrieve, and track changes to user data. The Storage API provides an extension-specific way to persist user data and state. It's similar to the web platform's storage APIs (IndexedDB, and Storage), but was designed to meet the storage needs of extensions.

• https://developer.chrome.com/docs/extensions/reference/api/notifications
  • Use the chrome.notifications API to create rich notifications using templates and show these notifications to users in the system tray.

And some that are a little more esoteric, but might still be interesting/useful:

• https://developer.chrome.com/docs/extensions/reference/api/scripting

  • Use the chrome.scripting API to execute script in different contexts.

• https://developer.chrome.com/docs/extensions/reference/api/userScripts

  • Use the userScripts API to execute user scripts in the User Scripts context.

• https://developer.chrome.com/docs/extensions/reference/api/webNavigation

  • Use the chrome.webNavigation API to receive notifications about the status of navigation requests in-flight.

• https://developer.chrome.com/docs/extensions/reference/api/offscreen

  • Use the offscreen API to create and manage offscreen documents.

fingerprinting-minified-javascript-libraries.md

Fingerprinting Minified JavaScript Libraries

Table of Contents

• See Also
• Initial ChatGPT Conversation / Notes
• Thoughts / comments as I've articulated them elsewhere
  • On j4k0xb/webcrack
    • Issue 21: rename short identifiers
  • On pionxzh/wakaru
    • Issue 34: support un-mangle identifiers

See Also

• pionxzh/wakaru#41

  • Module detection

• pionxzh/wakaru#34

  • support un-mangle identifiers

Initial ChatGPT Conversation / Notes

Source: https://chat.openai.com/c/d9b7b64f-aa93-474e-939f-79e376e6d375

Me:

Are there any tools that allow me to identify potentially minified javascript/etc libraries within a webpack or similar bundle?

ChatGPT:

Identifying and fingerprinting minified JavaScript modules within a webpack bundle, regardless of their minification method, can be a challenging task due to the nature of minification and bundling. However, there are approaches and tools you can use to achieve this:

1. Custom Fingerprinting Script: Given your skills in IT and software engineering, you might consider writing a custom script to analyze the bundles. This script can use heuristics like function and variable name patterns, known code snippets, or specific algorithm implementations to identify libraries. The script can parse the bundle, apply these heuristics, and attempt to match segments of the code with known library patterns.

2. Entropy Analysis: Minified code typically has higher entropy than non-minified code. By analyzing the entropy of different sections of your bundle, you might be able to identify boundaries between different libraries or modules.

3. AST Analysis: Abstract Syntax Tree (AST) analysis involves parsing the JavaScript code into a tree structure, which can then be analyzed. You could use AST to look for specific patterns that match known library code structures.

4. Binary Analysis Tools: Tools like Binwalk, which are traditionally used for analyzing binary files, can sometimes be repurposed for script analysis. They work by scanning files for embedded files and executable code.

5. Regular Expression Matching: This involves creating regular expressions that match known patterns from popular libraries. While this approach can be time-consuming and may require regular updates, it can be effective for well-known and widely used libraries.

6. Machine Learning Approaches: With advancements in machine learning, you can train a model to recognize patterns in minified code. This requires a substantial dataset of minified and unminified code to train the model.

7. Community Databases: Sometimes, the community might maintain databases of fingerprints of popular libraries. These can be used to match against your bundle.

8. Source Map Reversal: If source maps are available, you can reverse-engineer them to get the original source code, making it easier to identify the libraries.

Remember, the effectiveness of these methods depends on various factors, including the degree of minification, the presence of source maps, and the specific bundling process used. Additionally, given the potential complexity, this might be a time-consuming task, but your background in software engineering and ethical hacking should provide a solid foundation for tackling this challenge.

Me:

Let's work with the following ideas:

Custom Fingerprinting Script: Given your skills in IT and software engineering, you might consider writing a custom script to analyze the bundles. This script can use heuristics like function and variable name patterns, known code snippets, or specific algorithm implementations to identify libraries. The script can parse the bundle, apply these heuristics, and attempt to match segments of the code with known library patterns.

AST Analysis: Abstract Syntax Tree (AST) analysis involves parsing the JavaScript code into a tree structure, which can then be analyzed. You could use AST to look for specific patterns that match known library code structures.

Community Databases: Sometimes, the community might maintain databases of fingerprints of popular libraries. These can be used to match against your bundle.

--

Let's assume that there isn't currently a community database, but that once we figure out an effective way of creating and matching these fingerprints, that we could open source one.

Since we want this to work regardless of whether the library code has been minified or not, we need to take that into account.

How should we approach this?

ChatGPT:

Creating a system to fingerprint JavaScript libraries within a webpack bundle, especially considering the challenges posed by minification, is a multi-step process. The goal is to create a system that can analyze JavaScript code, extract identifiable patterns, and match these against a database of known library fingerprints. Here's a proposed approach:

Establishing a Fingerprinting Methodology

1. Identify Unique Patterns: Begin by identifying patterns that remain consistent in a library, even after minification. These could include:

   • Unique function signatures.
   • Specific algorithm implementations.
   • Characteristic variable naming schemes that survive minification.
   • Constant values or strings used by the library.

2. Heuristics for Detection: Develop heuristics to detect these patterns, such as specific function name and parameter combinations, unique method chaining styles, or specific API calls.

Abstract Syntax Tree (AST) Analysis

1. Parse JavaScript Bundle: Utilize tools like Esprima, Acorn, or Babel to convert JavaScript code into AST, enabling effective analysis of code structure and content.

2. Pattern Recognition in AST: Implement algorithms for matching parsed code against defined fingerprint criteria, focusing on structural and syntactical patterns and heuristics established earlier. This involves looking for specific node types (like function declarations, variable declarations, etc.) that match the identified patterns.

3. Handling Minification: Design heuristics and pattern recognition to be resilient to common minification techniques like variable renaming and whitespace removal.

Building and Maintaining the Fingerprint Database

1. Collecting Fingerprints: Start with popular libraries, analyzing unminified sources to establish baseline fingerprints.

2. Database Structure and Contributions: Organize the database with library entries, fingerprints, and metadata, and set up a system for validating and integrating community contributions.

Matching Mechanism and Script Integration

1. Matching Algorithm: Develop an algorithm to compare AST nodes against database fingerprints.

2. Handling Ambiguities: Design the system to handle cases where a definitive match isn't possible. This could involve ranking potential matches or providing a probability score.

3. Script Functionality: Ensure the script can process a JavaScript bundle, parse it, and match it against fingerprints. Test the matching algorithm with both minified and unminified versions of known libraries to ensure accuracy.

Handling Minification Variations

1. Minification Resilience: Ensure your fingerprints are resilient to common minification techniques, like variable renaming and whitespace removal.

2. Testing Across Minifiers: Test your system with code minified by different tools (e.g., UglifyJS, Terser) to ensure reliability.

User Interface and Accessibility

1. Ease of Use: Develop a user-friendly interface for both contributing to the database and using the script to identify libraries.

2. Integration with Development Tools: Consider how your tool could integrate with existing development workflows, such as bundlers or IDEs.

Open Sourcing and Community Engagement

1. Open Source the Tool: Release the tool and database as open source, encouraging community contributions of new fingerprints and algorithm improvements.

2. Documentation and Support Channels: Provide thorough documentation and establish channels for community support, feedback, and contributions.

Continuous Improvement and Adaptation

1. Regular Updates and Algorithm Refinement: Keep the database up-to-date with new library versions and continuously refine matching algorithms based on user feedback and new JavaScript development trends.

2. Performance Optimization: Optimize for performance, especially when analyzing large bundles, using efficient AST traversal and optimized pattern matching algorithms.

Legal and Ethical Considerations

1. Respecting Copyright: Ensure that the use of library fingerprints complies with legal and ethical standards, respecting the intellectual property rights of library authors.

This project aligns well with your background in IT and software engineering. The main challenge lies in the nuances of JavaScript minification and the diverse coding patterns across different libraries, but a focused and iterative approach can lead to a robust and useful tool for the community.

Thoughts / comments as I've articulated them elsewhere

On j4k0xb/webcrack

Issue 21: rename short identifiers

When I was exploring this concept in my own deobfuscation PoC project, I was exploring to make the variable names unique + have them add sort of semantic information about their source/scope.

Eg. if it was an arg to a function, it might be arg_1. Or potentially if the function is foo, it might end up as foo_arg_1

It looks like most of the PoC code I was playing with was local/in a pretty messy/hacky state, but I did find a link in it to an online REPL I was playing around with some of it in. Not sure how outdated that code is, but it might be useful:

• https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples

There were a number of different AST parsers I was playing around with, but I think that this babel code may have been the latest (not sure which one):

• https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples#babel_v1_3.js
• https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples#babel_v1_3_clean.js

Within those files, I believe the functions getNameFromPath, getPrefix (and older commented out functions getTypePrefix, getPrefix

---

Edit: Came across this in another issue here:

I published my decompiler that I used in the above example. I think it might be a good reference for adding this feature. https://github.com/e9x/krunker-decompiler

Originally posted by @e9x in j4k0xb/webcrack#10 (comment)

And looking at it's libRenameVars code seems to be taking a vaguely similar approach to how I was looking at doing things in my original PoC that I described above:

• https://github.com/e9x/krunker-decompiler/blob/master/src/libRenameVars.ts
  • getVarPrefix will set a prefix based on the type (eg. func, arg, Class, imported, var)
  • getName generates a new variable name that does not conflict with existing names or reserved keywords
  • generateName generates a new name for a variable considering its scope, type, and the context in which it is used (e.g., whether it's a class, a function variable, etc.). It employs various AST manipulations to ensure the generated name is appropriate and does not conflict with existing names.

A more generalised summary/overview (via ChatGPT):

Certainly, the code implements a sophisticated algorithm for renaming variables in a JavaScript program, adhering to several high-level rules and strategies:

1. Type-Specific Prefixing:

   • The getVarPrefix function assigns specific prefixes to variable names based on their type (e.g., "func" for function names, "arg" for parameters). This approach helps in identifying the role of a variable just by its name.

2. Avoiding Reserved Keywords:

   • The script includes a comprehensive list of reserved JavaScript keywords. If a variable's name matches a reserved keyword, it is prefixed with an underscore to prevent syntax errors.

3. Unique Naming with Context Consideration:

   • The generateName function ensures that each variable gets a unique name that doesn't conflict with other variables in its scope. It also considers the context in which a variable is used. For example, if a variable is part of a class, it may receive a name that reflects this context, using pascalCase or camelCase as appropriate.

4. Handling Special Cases:

   • The script contains logic to handle special cases, such as variables that are function expressions (isFuncVar) or class instances (isClass). This affects the naming convention applied to these variables.

5. Randomness with Mersenne Twister:

   • A Mersenne Twister is used to generate random elements for variable names, ensuring that the names are not only unique within the scope of the program but also less predictable.

6. AST-Based Renaming:

   • The script analyzes the Abstract Syntax Tree (AST) of the program to understand the structure and scope of variables. This analysis guides the renaming process, ensuring that the new names are consistent with the variable's usage and position in the code.

7. Scope Analysis with ESLint Scope:

   • By leveraging eslint-scope, the script can accurately determine the scope of each variable. This is crucial in avoiding name collisions and ensuring that the renaming respects lexical scoping rules in JavaScript.

8. Consideration for Exported and Assigned Variables:

   • The script pays special attention to variables that are exported or assigned in specific ways (e.g., through Object.defineProperty). It ensures that these variables receive names that are appropriate for their roles.

In summary, the script uses a combination of type-based naming conventions, context consideration, randomness, AST analysis, and scope analysis to systematically rename variables in a JavaScript program. This approach aims to enhance readability, avoid conflicts, and maintain the logical structure of the program.

Originally posted by @0xdevalias in j4k0xb/webcrack#21 (comment)

---

And for an even cooler/more extreme version of improving variable naming; I just came across this blog post / project from @jehna that makes use of webcrack + ChatGPT for variable renaming:

• https://thejunkland.com/blog/using-llms-to-reverse-javascript-minification.html

  • Using LLMs to reverse JavaScript variable name minification This blog introduces a novel way to reverse minified Javascript using large language models (LLMs) like ChatGPT and llama2 while keeping the code semantically intact. The code is open source and available at Github project Humanify.

• https://github.com/jehna/humanify

  • Un-minify Javascript code using ChatGPT

  • This tool uses large language modeles (like ChatGPT & llama2) and other tools to un-minify Javascript code. Note that LLMs don't perform any structural changes – they only provide hints to rename variables and functions. The heavy lifting is done by Babel on AST level to ensure code stays 1-1 equivalent.

Originally posted by @0xdevalias in j4k0xb/webcrack#21 (comment)

---

I came across another tool today that seemed to have a start on implementing some 'smart rename' features:

• https://github.com/pionxzh/wakaru
  • https://wakaru.vercel.app/

Digging through the code lead me to this:

• https://github.com/pionxzh/wakaru/tree/main/packages/unminify#smart-rename

  • Rename minified identifiers with heuristic rules.

• https://github.com/pionxzh/wakaru/blob/main/packages/unminify/src/transformations/smart-rename.ts
  • handleDestructuringRename, handleFunctionParamsRename, handlePropertyRename, handleReactRename, getElementName
• https://github.com/pionxzh/wakaru/blob/main/packages/unminify/src/utils/identifier.ts#L28-L75
  • generateName, getUniqueName
• https://github.com/pionxzh/wakaru/blob/master/packages/unminify/src/transformations/__tests__/smart-rename.spec.ts

There's also an issue there that seems to be exploring how to improve 'unmangling variable names' as well:

• pionxzh/wakaru#34

Which I wrote the following extra thoughts on:

I just finished up writing some thoughts/references for variable renaming on the webcrack repo, that could also be a useful idea for here. (see quotes below)

When I was exploring PoC ideas for my own project previously, I was looking to generate a file similar to the 'module map' that this project is using; but instead of just for the names of modules, I wanted to be able to use it to provide a 'variable name map'. Though because the specific variables used in webpack/etc can change between builds, my thought was that first 'normalising' them to a 'known format' based on their context would make sense to do first.

That could then be letter enhanced/expanded by being able to pre-process these 'variable name mappings' for various open source projects in a way that could then be applied 'automagically' without the end user needing to first create them.

It could also be enhanced by similar techniques such as what the humanify project does, by using LLMs/similar to generate suggested variable name mappings based on the code.

My personal ideal end goal for a feature like that would then allow me to use it within an IDE-like environment, where I can rename variables 'as I explore', knowing that the mappings/etc will be kept up to date.

Originally posted by @0xdevalias in pionxzh/wakaru#34 (comment)

Originally posted by @0xdevalias in j4k0xb/webcrack#21 (comment)

---

Another link from my reference notes that I forgot to include earlier; my thoughts on how to rename otherwise unknown variables are based on similar concepts that are used in reverse engineering tools such as IDA:

• https://hex-rays.com/blog/igors-tip-of-the-week-34-dummy-names/

  • In IDA’s disassembly, you may have often observed names that may look strange and cryptic on first sight: sub_73906D75, loc_40721B, off_40A27C and more. In IDA’s terminology, they’re called dummy names. They are used when a name is required by the assembly syntax but there is nothing suitable available

  • https://www.hex-rays.com/products/ida/support/idadoc/609.shtml

    • IDA Help: Names Representation

    • Dummy names are automatically generated by IDA. They are used to denote subroutines, program locations and data. Dummy names have various prefixes depending on the item type and value

---

And a few more I was looking at recently as well (that is sort of basically smart-rename:

• https://binary.ninja/2023/09/15/3.5-expanded-universe.html#automatic-variable-naming

  • Automatic Variable Naming One easy way to improve decompilation output is to come up with better default names for variables. There’s a lot of possible defaults you could choose and a number of different strategies are seen throughout different reverse engineering tools. Prior to 3.5, Binary Ninja left variables named based on their origin. Stack variables were var_OFFSET, register-based variables were reg_COUNTER, and global data variables were (data_). While this scheme isn’t changing, we’re being much more intelligent about situations where additional information is available.

    For example, if a variable is passed to a function and a variable name is available, we can now make a much better guess for the variable name. This is most obvious in binaries with type libraries.

  • This isn’t the only style of default names. Binary Ninja also will name loop counters with simpler names like i, or j, k, etc (in the case of nested loops)

• Vector35/binaryninja-api#2558

Originally posted by @0xdevalias in pionxzh/wakaru#34 (comment)

Originally posted by @0xdevalias in j4k0xb/webcrack#21 (comment)

On pionxzh/wakaru

Issue 34: support un-mangle identifiers

I just finished up writing some thoughts/references for variable renaming on the webcrack repo, that could also be a useful idea for here. (see quotes below)

When I was exploring PoC ideas for my own project previously, I was looking to generate a file similar to the 'module map' that this project is using; but instead of just for the names of modules, I wanted to be able to use it to provide a 'variable name map'. Though because the specific variables used in webpack/etc can change between builds, my thought was that first 'normalising' them to a 'known format' based on their context would make sense to do first.

That could then be later enhanced/expanded by being able to pre-process these 'variable name mappings' for various open source projects in a way that could then be applied 'automagically' without the end user needing to first create them.

It could also be enhanced by similar techniques such as what the humanify project does, by using LLMs/similar to generate suggested variable name mappings based on the code.

My personal ideal end goal for a feature like that would then allow me to use it within an IDE-like environment, where I can rename variables 'as I explore', knowing that the mappings/etc will be kept up to date.

---

When I was exploring this concept in my own deobfuscation PoC project, I was exploring to make the variable names unique + have them add sort of semantic information about their source/scope.

Eg. if it was an arg to a function, it might be arg_1. Or potentially if the function is foo, it might end up as foo_arg_1

It looks like most of the PoC code I was playing with was local/in a pretty messy/hacky state, but I did find a link in it to an online REPL I was playing around with some of it in. Not sure how outdated that code is, but it might be useful:

• https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples

There were a number of different AST parsers I was playing around with, but I think that this babel code may have been the latest (not sure which one):

• https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples#babel_v1_3.js
• https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples#babel_v1_3_clean.js

Within those files, I believe the functions getNameFromPath, getPrefix (and older commented out functions getTypePrefix, getPrefix

---

Edit: Came across this in another issue here:

I published my decompiler that I used in the above example. I think it might be a good reference for adding this feature. https://github.com/e9x/krunker-decompiler

Originally posted by @e9x in j4k0xb/webcrack#10 (comment)

And looking at it's libRenameVars code seems to be taking a vaguely similar approach to how I was looking at doing things in my original PoC that I described above:

• https://github.com/e9x/krunker-decompiler/blob/master/src/libRenameVars.ts
  • getVarPrefix will set a prefix based on the type (eg. func, arg, Class, imported, var)
  • getName generates a new variable name that does not conflict with existing names or reserved keywords
  • generateName generates a new name for a variable considering its scope, type, and the context in which it is used (e.g., whether it's a class, a function variable, etc.). It employs various AST manipulations to ensure the generated name is appropriate and does not conflict with existing names.

A more generalised summary/overview (via ChatGPT):

Certainly, the code implements a sophisticated algorithm for renaming variables in a JavaScript program, adhering to several high-level rules and strategies:

1. Type-Specific Prefixing:

   • The getVarPrefix function assigns specific prefixes to variable names based on their type (e.g., "func" for function names, "arg" for parameters). This approach helps in identifying the role of a variable just by its name.

2. Avoiding Reserved Keywords:

   • The script includes a comprehensive list of reserved JavaScript keywords. If a variable's name matches a reserved keyword, it is prefixed with an underscore to prevent syntax errors.

3. Unique Naming with Context Consideration:

   • The generateName function ensures that each variable gets a unique name that doesn't conflict with other variables in its scope. It also considers the context in which a variable is used. For example, if a variable is part of a class, it may receive a name that reflects this context, using pascalCase or camelCase as appropriate.

4. Handling Special Cases:

   • The script contains logic to handle special cases, such as variables that are function expressions (isFuncVar) or class instances (isClass). This affects the naming convention applied to these variables.

5. Randomness with Mersenne Twister:

   • A Mersenne Twister is used to generate random elements for variable names, ensuring that the names are not only unique within the scope of the program but also less predictable.

6. AST-Based Renaming:

   • The script analyzes the Abstract Syntax Tree (AST) of the program to understand the structure and scope of variables. This analysis guides the renaming process, ensuring that the new names are consistent with the variable's usage and position in the code.

7. Scope Analysis with ESLint Scope:

   • By leveraging eslint-scope, the script can accurately determine the scope of each variable. This is crucial in avoiding name collisions and ensuring that the renaming respects lexical scoping rules in JavaScript.

8. Consideration for Exported and Assigned Variables:

   • The script pays special attention to variables that are exported or assigned in specific ways (e.g., through Object.defineProperty). It ensures that these variables receive names that are appropriate for their roles.

In summary, the script uses a combination of type-based naming conventions, context consideration, randomness, AST analysis, and scope analysis to systematically rename variables in a JavaScript program. This approach aims to enhance readability, avoid conflicts, and maintain the logical structure of the program.

Originally posted by @0xdevalias in j4k0xb/webcrack#21 (comment)

---

And for an even cooler/more extreme version of improving variable naming; I just came across this blog post / project from @jehna that makes use of webcrack + ChatGPT for variable renaming:

• https://thejunkland.com/blog/using-llms-to-reverse-javascript-minification.html

  • Using LLMs to reverse JavaScript variable name minification This blog introduces a novel way to reverse minified Javascript using large language models (LLMs) like ChatGPT and llama2 while keeping the code semantically intact. The code is open source and available at Github project Humanify.

• https://github.com/jehna/humanify

  • Un-minify Javascript code using ChatGPT

  • This tool uses large language modeles (like ChatGPT & llama2) and other tools to un-minify Javascript code. Note that LLMs don't perform any structural changes – they only provide hints to rename variables and functions. The heavy lifting is done by Babel on AST level to ensure code stays 1-1 equivalent.

Originally posted by @0xdevalias in j4k0xb/webcrack#21 (comment)

Originally posted by @0xdevalias in pionxzh/wakaru#34 (comment)

For now, we have smart-rename that can guess the variable name based on the context. I would like to expand it to cover some other generic cases.

Linking to my smart-rename related issues to keep the contextual link here:

• pionxzh/wakaru#40
• pionxzh/wakaru#48
• pionxzh/wakaru#49

Originally posted by @0xdevalias in pionxzh/wakaru#34 (comment)

---

Another link from my reference notes that I forgot to include earlier; my thoughts on how to rename otherwise unknown variables are based on similar concepts that are used in reverse engineering tools such as IDA:

• https://hex-rays.com/blog/igors-tip-of-the-week-34-dummy-names/

  • In IDA’s disassembly, you may have often observed names that may look strange and cryptic on first sight: sub_73906D75, loc_40721B, off_40A27C and more. In IDA’s terminology, they’re called dummy names. They are used when a name is required by the assembly syntax but there is nothing suitable available

  • https://www.hex-rays.com/products/ida/support/idadoc/609.shtml

    • IDA Help: Names Representation

    • Dummy names are automatically generated by IDA. They are used to denote subroutines, program locations and data. Dummy names have various prefixes depending on the item type and value

Originally posted by @0xdevalias in j4k0xb/webcrack#21 (comment)

---

And a few more I was looking at recently as well (that is sort of basically smart-rename:

• https://binary.ninja/2023/09/15/3.5-expanded-universe.html#automatic-variable-naming

  • Automatic Variable Naming One easy way to improve decompilation output is to come up with better default names for variables. There’s a lot of possible defaults you could choose and a number of different strategies are seen throughout different reverse engineering tools. Prior to 3.5, Binary Ninja left variables named based on their origin. Stack variables were var_OFFSET, register-based variables were reg_COUNTER, and global data variables were (data_). While this scheme isn’t changing, we’re being much more intelligent about situations where additional information is available.

    For example, if a variable is passed to a function and a variable name is available, we can now make a much better guess for the variable name. This is most obvious in binaries with type libraries.

  • This isn’t the only style of default names. Binary Ninja also will name loop counters with simpler names like i, or j, k, etc (in the case of nested loops)

• Vector35/binaryninja-api#2558

Originally posted by @0xdevalias in pionxzh/wakaru#34 (comment)

---

Was looking closer at the sourcemap spec today, and the names field jumped out at me as potentially useful:

• https://tc39.es/source-map-spec/#names

  • names: a list of symbol names used by the mappings entry

• https://tc39.es/source-map-spec/#mappings

  • mappings: a string with the encoded mapping data (see 4.1 Mappings Structure)

• https://tc39.es/source-map-spec/#mappings-structure

  • The mappings data is broken down as follows:

    • each group representing a line in the generated file is separated by a semicolon (;)
    • each segment is separated by a comma (,)
    • each segment is made up of 1, 4, or 5 variable length fields.

  • It then goes on to describe the segment's in greater detail, but the specific part I was thinking could be relevant here would be this:

    • If present, the zero-based index into the names list associated with this segment. This field is a base 64 VLQ relative to the previous occurrence of this field unless this is the first occurrence of this field, in which case the whole value is represented.

Obviously if there is a full sourcemap for the webapp, then wakaru isn't really needed anyway.. but what I was thinking of here is that in combination with module detection (see - pionxzh/wakaru#41), if there are sourcemapss available for that original module, then we could potentially extract the original function/variable/etc names from the names field of the sourcemap, and use them in a sort of 'smart-rename with sourcemap' type way.

---

Another sourcemap related idea I had (which probably deserves it's own issue) is that it would be cool to be able to 'retroactively generate a sourcemap) for a webapp, based on the unminified output from wakaru; such that we could than take that sourcemap, and apply it to the original minified web app source for debugging the live app.

Edit: Created a new issue to track this:

• pionxzh/wakaru#66

Originally posted by @0xdevalias in pionxzh/wakaru#34 (comment)

---

It isn't very meaningful to support such a feature when you can access all the source code.

@pionxzh I was specifically talking about it in terms of bundled modules (eg. React, etc), and not the unique web app code of the app itself.

Originally posted by @0xdevalias in pionxzh/wakaru#34 (comment)

---

You mean like, for popular open-source projects, we can put some sourcemap in our project / read from the chunk, and then reverse map the minified variable and function name back to normal?

@pionxzh Similar to that, but probably not "put the sourcemap in our project" directly; but more process the sourcemaps from popular open-source projects and extract those details to an 'intermediary form'. That 'intermediary form' would be similar to the 'module map' file, as I described earlier in this thread:

When I was exploring PoC ideas for my own project previously, I was looking to generate a file similar to the 'module map' that this project is using; but instead of just for the names of modules, I wanted to be able to use it to provide a 'variable name map'. Though because the specific variables used in webpack/etc can change between builds, my thought was that first 'normalising' them to a 'known format' based on their context would make sense to do first.

That could then be later enhanced/expanded by being able to pre-process these 'variable name mappings' for various open source projects in a way that could then be applied 'automagically' without the end user needing to first create them.

It could also be enhanced by similar techniques such as what the humanify project does, by using LLMs/similar to generate suggested variable name mappings based on the code.

Originally posted by @0xdevalias in pionxzh/wakaru#34 (comment)

---

A configuration table/profile can be provided to allow users to manually write correspondences. wakaru can simply include the rules of the better known packages.

@StringKe nods, sounds like we are thinking about similar things here :)

---

Can you specify the content that you would expect to have? and the corresponding behavior

@pionxzh For me personally, I haven't deeply thought through all the use cases in depth, but at a high level I basically want to be able to take a web app that is going to be re-built multiple times, and be able to have a 'config file' similar to the 'module mapping' that wakaru has/had; but that also allows me to specify the variable/function names ('symbols') that are used within it.

The slightly more challenging part is that because the app will be re-built multiple times, the minified variables will change (sometimes every build), so we can't easily use those as the 'key' of the mapping. One idea I had for solving that is potentially by first renaming all of the variables based on a 'stable naming pattern' (eg. func_*, arg_*, const_*, etc; and then could just use a counter/similar based on the 'scope' it's being defined in) that would be generated based on the scope/type of the 'symbol', and would therefore be resilient to the minified variable names changing each build. Those 'stable intermediary names' could then potentially be used for the keys in the variable mapping.

Though then we also need to figure out what level of 'granularity' makes sense to generate those 'stable intermediary names' at; as having a 1:1 mapping of those 'stable name scopes' to JS scopes could potentially end up being really noisy in the mapping file. So maybe using a 'higher abstracted scope' would make more sense (eg. at the module level or similar)

My original hacky implementation of this in my own PoC code was using JS objects/JSON to map an explicit minified variable name to it's 'proper' name; but that broke because the minified names changed between builds. Even by implementing the 'stable naming pattern', if those 'stable names' included a 'counter' in them (eg. func_1, const_8, etc) we still probably wouldn't want to use those stable names directly as the key of an object, as if a new variable was added 'in between' in a later build, that would flow on to 'shifting' the 'counter' for every variable of a matching type afterwards, which would be a lot of effort to manually update in a mapping file. While I haven't thought too deeply about it, I think that by using an array in the mapping file, it should simplify things so that we only need to make a small change to 'fix the mappings' when a new variable is added that 'shifts' everything.

Even by using the array concept in the mappings file, there is still some manual pain/effort involved in trying to keep the mapping 'up to date' in newer builds. That's what lead me into some of the deeper/more esoteric ideas/thinking around 'fingerprinting' that I expand on below.

--

Another area I started looking into (but haven't deeply explored yet) for both figuring out how to map variable names to sections of code in a 'smart' way, and potentially also for module identification (see #41); is in the space of 'structural AST fingerprinting' or 'code similarity' algorithms and similar. (I realise that this is a rather deep/esoteric angle to be looking at this from, and that there are likely going to be far simpler/easier ways to implement the variable mapping/module identification in a 'good enough' way without going to this level of depth; but I'm curious to explore it regardless, to see if any good ideas come out of it)

I haven't gotten too far in my reading yet (got distracted on other things), but the high level of my idea was that maybe we could generate an 'AST fingerprint' that isn't impacted by the variable/function/etc names ('symbols') changing during minification; and then use that as the basis for the 'key' in the 'mappings file'; as that fingerprint could theoretically still identify a 'scope' (which might be a literal JS scope, or might be a higher level abstraction that we decide makes sense; the most abstract being probably at the bundled module level) even if the bundler decides to move some functions around to a different module/etc. Then obviously if we were able to generate those 'resilient fingerprints' to identify code even when it's been minified, that would make perfect sense to apply to module detection/etc (see #41) as well.

Some of the high level ideas / search terms that I was using to start my research in that area was things like:

• AST fingerprinting
• Source code similarity fingerprinting
• Control flow graphs
• Call flow graphs
• Program dependence graph
• etc

Here is a link dump of a bunch of the tabs I have open but haven't got around to reviewing in depth yet, RE: 'AST fingerprinting' / Code Similarity / etc:

Unsorted/Unreviewed Initial Link Dump RE: 'AST fingerprinting' / Code Similarity

• https://openreview.net/forum?id=BJxWx0NYPr

  • On the Effectiveness of Program Analysis in AI and Vice Versa

• https://dl.acm.org/doi/10.1145/3486860

  • A Unified Framework for Code Representation Learning

• https://inria.hal.science/hal-01648996/document

  • A Survey of Program Slicing Techniques

• https://hal.science/hal-00627811/document

  • From Program Slicing to Abstract Interpretation

• https://igm.univ-mlv.fr/~chilowi/research/syntax_tree_fingerprinting/syntax_tree_fingerprinting_ICPC09.pdf

  • Syntax Tree Fingerprinting for Source Code Similarity Detection

• https://ieeexplore.ieee.org/document/9960266

  • Exploring the Landscape of Deep Learning for Source Code Analysis

• https://en.wikipedia.org/wiki/Program_dependence_graph

  • Program Dependence Graph - Wikipedia

• https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=3437&context=theses

  • Enhancing Software Reliability Using Program Analysis Techniques

• https://dl.acm.org/doi/pdf/10.1145/1286821.1286826

  • Static Analysis of Executables to Detect Malicious Patterns

• https://patents.google.com/patent/US9459861B1/en

  • US Patent US9459861B1: Techniques for Code Analysis and Optimization

• https://www.unomaha.edu/college-of-information-science-and-technology/research-labs/_files/software-nsf.pdf

  • Software Engineering Research and Practice: NSF Workshop Report

• https://www.computer.org/csdl/journal/ts/2023/08/10125077/1Nc4Vd4vb7W

  • Advances in Automated Program Repair: IEEE Transactions

• https://stackoverflow.com/questions/7283702/assembly-level-function-fingerprint

  • Stack Overflow: Assembly-Level Function Fingerprinting

• https://en.wikipedia.org/wiki/Control-flow_graph

  • Control-Flow Graph - Wikipedia

• https://www.kapravelos.com/publications/fptechniques-www23.pdf

  • Novel Techniques for Malware and Vulnerability Analysis in Web Applications

• https://pubmed.ncbi.nlm.nih.gov/12197666/

  • PubMed: Program Analysis Techniques for Software Debugging

• https://stackoverflow.com/questions/15087195/data-flow-graph-construction

  • Stack Overflow: Data Flow Graph Construction in Programming

• https://www.cs.columbia.edu/~suman/secure_sw_devel/Basic_Program_Analysis_CF.pdf

  • Basic Program Analysis Techniques for Secure Software Development

• https://codeql.github.com/docs/writing-codeql-queries/about-data-flow-analysis/

  • CodeQL Guide: Understanding Data Flow Analysis for Secure Software Development

• https://codereview.stackexchange.com/questions/276387/call-flow-graph-from-python-abstract-syntax-tree

  • Code Review Stack Exchange: Call Flow Graph from Python AST

• https://clang.llvm.org/docs/DataFlowAnalysisIntro.html

  • LLVM Clang Documentation: Introduction to Data Flow Analysis

• https://www.researchgate.net/figure/AST-based-program-graph-with-additional-control-and-data-flow-edges_fig2_370980383

  • ResearchGate: AST-Based Program Graphs with Control and Data Flow Edges

--

Another idea I've had, but only lightly explored so far, is looking into how various projects like Terser, Webpack, etc choose their minified variable names in general; but also how they handle 'stable minified variables' between builds (which is something that I know at least Webpack has some concept of). My thought there is that by understanding how they implement their own 'stable minified variables between builds', that we might be able to leverage to either a) do similar, or b) be able to reverse engineer that in a way that might be able to be 'retroactively applied' on top of an existing minified project that didn't use 'stable minified variables', to 'stabilise' them.

Originally posted by @0xdevalias in pionxzh/wakaru#34 (comment)