Skip to content

Instantly share code, notes, and snippets.

@lovasoa
Last active December 5, 2024 14:01
Show Gist options
  • Save lovasoa/8691344 to your computer and use it in GitHub Desktop.
Save lovasoa/8691344 to your computer and use it in GitHub Desktop.
Walk through a directory recursively in node.js.
// ES6 version using asynchronous iterators, compatible with node v10.0+
const fs = require("fs");
const path = require("path");
async function* walk(dir) {
for await (const d of await fs.promises.opendir(dir)) {
const entry = path.join(dir, d.name);
if (d.isDirectory()) yield* walk(entry);
else if (d.isFile()) yield entry;
}
}
// Then, use it with a simple async for loop
async function main() {
for await (const p of walk('/tmp/'))
console.log(p)
}
// Callback-based version for old versions of Node
var fs = require("fs"),
path = require("path");
function walk(dir, callback) {
fs.readdir(dir, function(err, files) {
if (err) throw err;
files.forEach(function(file) {
var filepath = path.join(dir, file);
fs.stat(filepath, function(err,stats) {
if (stats.isDirectory()) {
walk(filepath, callback);
} else if (stats.isFile()) {
callback(filepath, stats);
}
});
});
});
}
if (exports) {
exports.walk = walk;
} else {
walk(".", manageFile);
}
@JustMaier
Copy link

Here's an es6 version

const fs = require('fs').promises;
const path = require('path');

async function walk(dir) {
    let files = await fs.readdir(dir);
    files = await Promise.all(files.map(async file => {
        const filePath = path.join(dir, file);
        const stats = await fs.stat(filePath);
        if (stats.isDirectory()) return walk(filePath);
        else if(stats.isFile()) return filePath;
    }));

    return files.reduce((all, folderContents) => all.concat(folderContents), []);
}

@tqwewe
Copy link

tqwewe commented May 11, 2020

Synchronous version:

const walkSync = (dir, callback) => {
  const files = fs.readdirSync(dir);
  files.forEach((file) => {
    var filepath = path.join(dir, file);
    const stats = fs.statSync(filepath);
    if (stats.isDirectory()) {
      walkSync(filepath, callback);
    } else if (stats.isFile()) {
      callback(filepath, stats);
    }
  });
};

@lovasoa
Copy link
Author

lovasoa commented May 11, 2020

I think the best way to do this in modern javascript is with an asynchronous iterator :

async function* walk(dir) {
    for await (const d of await fs.promises.opendir(dir)) {
        const entry = path.join(dir, d.name);
        if (d.isDirectory()) yield* await walk(entry);
        else if (d.isFile()) yield entry;
    }
}

The code is both clear, non-blocking, concise, easy to use, and avoids the explicit call to stats. It works in node v10.0+. You then iterate with just

for await (const p of walk('/tmp/'))
    console.log(p)

@npomfret
Copy link

npomfret commented Jun 6, 2020

@lovasoa I agree. But a useful feature (for me anyway) is to give the callback the ability to stop the process (when the file has been found, or it has gone too deep for example)

@lovasoa
Copy link
Author

lovasoa commented Jun 6, 2020

@npomfret : It is actually easier to stop the process early with the asynchronous iterator; just use break :

example :

  for await (const p of walk('/')) {
    if (p.endsWith("passwd")) {
      console.log(p)
      break;
    }
  }

@npomfret
Copy link

npomfret commented Jun 6, 2020

Ah yes, very good point.

@clubside
Copy link

I'm just getting started with node so I'm sure the answer is straight forward so forgive me...but if I enter the May 11 code first it won't accept the await part of for await (const p of walk('/tmp/')). If I remove that it will run then throw an exception on the walk part of that same line saying its not a function or its return value is no iterable. Too many concepts coming at me at once "p so I don't even know where to start in terms of figuring out these errors. Any help?

@lovasoa
Copy link
Author

lovasoa commented Jun 13, 2020

@clubside

if I enter the May 11 code first

I'm not sure I understand that...

it won't accept the await part of for await (const p of walk('/tmp/'))

I guess you are either using an old version version of node or you are trying to use for await outside of an async function.

Here are a few articles you can read to get started :

@clubside
Copy link

I'm using Node 12 according to node -v

It just looked like the two parts of the May 11 comment were a standalone I could just run but thanks for the links I'll see if I can work it out!

@jasonk
Copy link

jasonk commented Jun 17, 2020

You can easily turn it into a standalone, just wrap the top-level code in an async function:

async function* walk(dir) {
    for await (const d of await fs.promises.opendir(dir)) {
        const entry = path.join(dir, d.name);
        if (d.isDirectory()) yield* await walk(entry);
        else if (d.isFile()) yield entry;
    }
}
async function main() {
  for await (const p of walk('/tmp/'))
    console.log(p)
}
main()

Or do it all in one, like:

( async () => {
  for await (const p of walk('/tmp/'))
    console.log(p)
} )();

@cristea2017
Copy link

Here's an es6 version

const fs = require('fs').promises;
const path = require('path');

async function walk(dir) {
    let files = await fs.readdir(dir);
    files = await Promise.all(files.map(async file => {
        const filePath = path.join(dir, file);
        const stats = await fs.stat(filePath);
        if (stats.isDirectory()) return walk(filePath);
        else if(stats.isFile()) return filePath;
    }));

    return files.reduce((all, folderContents) => all.concat(folderContents), []);
}

I searched for this a lot of time , Thanks ;)

@maicss
Copy link

maicss commented Aug 25, 2021

Any idea how to control parallel number?

@n1kk
Copy link

n1kk commented Sep 9, 2021

Synchronous version:

const walkSync = (dir, callback) => {
  const files = fs.readdirSync(dir);
  files.forEach((file) => {
    var filepath = path.join(dir, file);
    const stats = fs.statSync(filepath);
    if (stats.isDirectory()) {
      walkSync(filepath, callback);
    } else if (stats.isFile()) {
      callback(filepath, stats);
    }
  });
};

@acidic9 just curious, is there a reason for a var on line 4 or it's just an oversight? The rest of places uses const, just seems like it could've been intentional :)

@TotallyInformation
Copy link

TotallyInformation commented Oct 11, 2023

For fun, here is the ES6/Promises version with an added simple file-type filter:

    async walk(dir, ftype) {
        let files = await fs.readdir(dir)
        files = await Promise.all(files.map(async file => {
            const filePath = path.join(dir, file)
            const stats = await fs.stat(filePath)
            if (stats.isDirectory()) {
                return this.walk(filePath, ftype)
            } else if (stats.isFile() && file.endsWith(ftype)) {
                return filePath
            }
        }))

        // Filter out undefined entries before concatenating
        return files.filter(Boolean).reduce((all, folderContents) => all.concat(folderContents), [])
    }

Oh, and a minor correction to how you use it. Fairly sure you should have the inner await not just the outer:

for await (const p of await walk(folder, '.html')) {
    console.log(p)
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment