Creating a FIFO-type file stream pipe between the browser and local applications

filestream.md

Today we are going to create a FIFO (First In First Out) type file stream between the browser and a local application.

The idea is to create a local file in the browser, when the file is created the local application writes data to the newly created file, in the browser we will read the data written to the file up to that point, then truncate the file to size 0, repeat, when the writing and reading of the file are complete we will remove the file from the local file system.

In this way we can configure our local application to watch for the creation of specific named files, each performing a discrete process, and stream data from the local application to the file, without using networking.

We'll be using Deno in this example for the local application. Use whatever local application you want. Node.js, QuickJS, C, C++, Rust, Bash, whatever.

Our JavaScript code looks something like this watcher.js

// deno run -A watcher.js

const file = 'output.txt';

while (true) {

  console.log(`Watching ${file}`);

  const watcher = Deno.watchFs('');

  try {
    for await (const {
        kind,
        paths: [path]
      }
      of watcher) {

      if (path.split('/').pop() === file && kind === 'access') {
        console.log(kind, path);
        break;
      }
    }
  } catch {}

  try {
    watcher.close();
  } catch {}

  let data = 'abcdefghijklmnopqrstuvwxyz';

  for (const char of data) {
    console.log(char);
    const handle = await Deno.open(file, {
      read: true,
      write: true
    });
    await new Blob([char.toUpperCase().repeat(441 * 4)], {
        type: 'application/octet-stream'
      })
      .stream().pipeTo(handle.writable);
    // Wait for the file to be read and truncated in the browser
    while (true) {
      const {
        size
      } = await Deno.stat(file);
      if (size === 0) {
        break;
      } 
    }

    await new Promise((resolve) => setTimeout(resolve, 60));
  }

  await Deno.remove(file);

  console.log('Done file streaming');
}

We are keeping the script running in the while loop so we can create, write and read to a file multiple times without stopping and starting the local script.

Synchronizing file creation, writing and reading the created file between the local application and the browser is not trivial. For now we use a 60 millisecond delay after the file has been written to by the local application, then after we get the file information where the file size is 0 before moving on to the next iteration of the loop which writes data to file. Technically the file will be removed then re-written by File System Access API when truncate(0) is called.

In the browser (Chromium 118) with --enable-features=FileSystemObserver flag set we do something like this in fso.js

var {
  readable,
  writable
} = new TransformStream();

var handle = await showSaveFilePicker({
  multiple: 'false',
  types: [{
    description: "Stream",
    accept: {
      "application/octet-stream": [".txt"],
    },
  }, ],
  excludeAcceptAllOption: true,
  startIn: 'documents',
  suggestedName: 'output.txt'
});

var status = await handle.requestPermission({
  mode: "readwrite"
});

var fileStream = async (changedHandle, type = '') => {
  try {
    var {
      size
    } = await changedHandle.getFile();
    if (size) {
      var stream = await changedHandle.createWritable();
      var file = await changedHandle.getFile();
      await file.stream().pipeTo(writable, {
        preventClose: true
      });
      await stream.truncate(0);
      await stream.close();
    }
  } catch (e) {
      console.log(e);
      try {
        fso.unobserve(changedHandle);
        await writable.close();
      } catch (err) {
        console.log(err);
      }
      finally {
        console.log(handle);
        await handle.remove();
      }
  }
}

readable.pipeThrough(new TextDecoderStream()).pipeTo(
  new WritableStream({
    async start() {
      try {
        return fileStream(handle);
      } catch (e) {
        console.log(e);
        throw e;
      }
    },
    write(value) {
      console.log(value.length);
    },
    close() {
      console.log('Stream closed');
    }
  })
).catch(console.error);


var fso = new FileSystemObserver(async ([{
  changedHandle,
  root,
  type
}], record) => {

  try {
    await fileStream(changedHandle, type);
  } catch (e) {
    console.log(e);
  }
});

fso.observe(handle);

Notice the first call to fileStream(handle) in the start() method of the WritableStream we are piping our data to, so we don't miss the data written to the file on the first iteration of the loop where we write data.

If all went as expected we should begin with no file named output.txt on the file system, create the file in the browser, upon creation the file watching application (notify(), inotify-tools, etc.) begins writing data, waiting for the data to be read then the file to be truncated in the browser, then cycles to the next iteration of writing data to the file, and when the process is complete the file output.txt will be removed from the filesystem. We try to make sure of the removal by redundantly using Deno.remove() in the local application and handle.remove() in the browser, where handle is an instance of a FileSystemFileHandle.