Concatenator

README.md

Concatenator

Concatenates chunked file to one (big) file on the server-side in asynchronous manner. The primary motivation is to be able to support Tus' Concatenation Extension.

Usage

• Download 'Concatenator.js' file to appropriate location.
• Require the downloaded file in the source file that you want to use Concatenator class.

// index.js
// ...
const Concatenator = require('./Concatenator');
// ...

• Instantiate a new Concatenator object with source files.

// index.js
// ...
const Concatenator = require('./Concatenator');
// Below two lines added
const chunkFilepaths = ['/path/to/upload/dir/x00', '/path/to/upload/dir/x01', '/path/to/upload/dir/x02'];
const cctr = new Concatenator(chunkFilepaths);
// ...

• Listen Concatenator instance events.

// index.js
// ...
const Concatenator = require('./Concatenator');

const chunkFilepaths = ['/path/to/upload/dir/x00', '/path/to/upload/dir/x01', '/path/to/upload/dir/x02'];
const cctr = new Concatenator(chunkFilepaths);

// Below lines added
// You should always listen for 'error' events, like every EventEmitter class.
cctr.on('error', (err) => {
  // Do something with the error
  console.error(err);
});
// We are going to listen 'ready' event once.
cctr.once('ready', () => {
  // All promises started in constructor were settled.
  // We can start the concatenation.
  cctr.start();
});
cctr.on('progress', (processed, total) => {
  const percentage = ((100 * processed) / total).toFixed(2);

  // Sample output of the log below might be;
  // %42.13 - 4213/10000 bytes
  console.log(`%${percentage} - ${processed}/${total} bytes`);
});
// We are going to listen 'done' event once.
cctr.once('done', (finalFile, bytesWritten) => {
  // Concatenation finished
  
  // Sample output of the log below might be;
  // 10000 bytes written to '/path/to/upload/dir/x00'
  console.log(`${bytesWritten} bytes written to '${finalFile}'`);

  // Optionally you can remove all the listeners for this instance.
  cctr.removeAllListeners();
  
  // From now on you are done with this Concatenator instance. You can
  // use `finalFile` parameter to access the final (concatenated) file.
});
// ...

Notes

• The instance should be 'ready' to start the concatenation. This is because the Concatenator tries to calculate chunks size (via first chunk) and last chunk size, and then depending on them the total size of the concatenated file. All of the file system methods are asynchronous.
• Normally the constructor takes one argument and concatenate rest of the chunks to first chunk. What this means is that unless a destination filepath is given to the constructor Concatenator will not create a file to concatenate all the chunks into that. Constructor accepts a destination filepath as 2nd argument. This destination file path tells the Concatenator to create a file and concatenate all of the chunks into that (destination) file. Of course destination file can be somewhere else in the file system. See examples.js.
• Also the constructor accepts an options object. Via options you can pass chunk size (in bytes) to ease the size calculation process for Concatenator, and pass the filename of final (concatenated) file's name. All of the options can be omitted. If options.chunkSize omitted, Concatenator tries to calculate the chunk size by first chunk's size. If options.filename omitted, Concatenator first try to decide the file name for the final file via destination filepath argument. If this is not the case, then the final file name will be the first chunk's file name.

Remarks

• Calling start on not ready instance will not start the concatenation. Instead emits an error saying 'The instance is not ready yet.'. You should listen 'ready' event to start the concatenation.
• First argument of the constructor must be array. The order of the array reflects the order of the chunks, hence the chunks will be concatenated to final file in this order.

License

MIT License

Copyright (c) 2017 Ozan Müyesseroğlu ozanmuyes@gmail.com

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Concatenator.js

const EventEmitter = require('events');
const fs = require('fs');
const path = require('path');

const DEFAULT_OPTIONS = {
  chunkSize: 0, // in bytes, may be omitted

  filename: '', // only renames the file, will not move - this should be done via `destFilepath`
  // `filename` has precedence over `destFilepath`, in other words it renames destination file, if defined
};

/**
 * Concatenates given chunks (files) into one in a non-blocking manner.
 * Order of `srcFilepaths` array matters, chunks will be concatenated in order of that array.
 * @extends {EventEmitter}
 */
class Concatenator extends EventEmitter {
  /**
   * Creates a new Concatenator instance.
   * @param  {string[]} srcFilepaths Source chunk (file) path(s), last chunk SHOULD be the last item
   * @param  {string} [destFilepath] Destination chunk (file) path
   * @param  {Object} [options]      Instance options that will be deep merged with DEFAULT_OPTIONS
   * @return {Concatenator}              Created Concatenator instance
   */
  constructor(srcFilepaths, destFilepath = '', options = {}) {
    super();

    if (typeof srcFilepaths !== 'object' || !Array.isArray(srcFilepaths)) {
      throw new TypeError('Source chunk filepaths parameter must be an array.');
    }

    if (srcFilepaths.length === 0) {
      throw new Error('Source chunk filepaths array was empty.');
    }

    this.srcFilepaths = srcFilepaths;

    // Since `destFilepath` is optional when omitted we need to swap parameters
    if (typeof destFilepath === 'object') {
      options = destFilepath;
      destFilepath = '';
    }


    this.options = Object.assign({}, DEFAULT_OPTIONS, options);

    // If destination filepath was not given...
    this.destFilepath = (!destFilepath)
      ? srcFilepaths[0] // ...get first item as destination
      : destFilepath;

    // Process other options here

    this._isInstReady = false;

    this.calculateSizes().then((sizes) => {
      this.options.chunkSize = sizes.chunkSize;
      this.lastChunkSize = sizes.lastChunkSize;
      this.totalSize = sizes.totalSize;

      this._isInstReady = true;

      this.emit('ready');
    }).catch((err) => {
      this.emit('error', err);
    });
  }

  /**
   * Get file size asynchronous.
   * @param  {string} filepath File path
   * @return {Promise}          Promise resolves to file size in bytes
   */
  getFileSize(filepath) {
    return new Promise((resolve, reject) => {
      fs.stat(filepath, (err, stats) => {
        if (err) {
          reject(err);
        } else {
          resolve(stats.size);
        }
      });
    });
  }

  /**
   * Calculates chunk size (if not given), last chunk size and total
   * size asynchronous.
   * @return {Promise}  Promise resolves to sizes object
   */
  calculateSizes() {
    return new Promise(async (resolve, reject) => {
      let chunkSize = -1;
      try {
        chunkSize = this.options.chunkSize || await this.getFileSize(this.srcFilepaths[0]);
      } catch (err) {
        reject(err);
      }

      let lastChunkSize = -1;
      try {
        lastChunkSize = await this.getFileSize(this.srcFilepaths[this.srcFilepaths.length - 1]);
      } catch (err) {
        reject(err);
      }

      const totalSize = (((this.srcFilepaths.length - 1 /*last chunk*/) * chunkSize) + lastChunkSize);

      resolve({
        chunkSize,
        lastChunkSize,
        totalSize,
      });
    });
  }

  /**
   * Appends given file to destination file
   * @param  {string} srcFilepath Source file path
   * @return {Promise}             Resolves when file appended to destination
   */
  appendToDestination(srcFilepath) {
    return new Promise((resolve, reject) => {
      // `reader` is an instance of 'fs.ReadStream' class (i.e. 'stream.Readable' class instance)
      const reader = fs.createReadStream(srcFilepath);

      reader.on('error', (err) => {
        reject(err);
      });

      reader.on('close', () => {
        // We are done with this chunk (file), so we can remove (unlink) it
        fs.unlink(srcFilepath, (err) => {
          if (err) {
            this.emit('error', err);
          }
        });

        resolve();
      });

      // By default Node.js closes stream.Writable when done with piping.
      // But this is not what we want, so we are saying to Node.js
      // to NOT close the writer stream after piping done.
      reader.pipe(this.writer, { end: false /*this is important*/ });
    });
  }

  async start() {
    if (!this._isInstReady) {
      this.emit('error', new Error('The instance is not ready yet.'));

      return;
    }

    // `writer` is an instance of 'fs.WriteStream' class (i.e. 'stream.Writable' class instance)
    this.writer = fs.createWriteStream(this.destFilepath, { flags: 'a' });

    let bytesWritten = 0;

    this.emit('progress', bytesWritten, this.totalSize);

    // We are iterating through source chunks (files), not using chunks count here
    for (let i = 0; i < this.srcFilepaths.length; i += 1) {
      if (this.srcFilepaths[i] === this.destFilepath) {
        // Skipping first chunk, since it is the destination file
        bytesWritten += this.options.chunkSize;

        this.emit('progress', bytesWritten, this.totalSize);

        continue;
      }

      try {
        await this.appendToDestination(this.srcFilepaths[i]);
      } catch (err) {
        this.emit('error', err);

        return;
      }

      bytesWritten += (i + 1 === this.srcFilepaths.length)
        ? this.lastChunkSize // last chunk's size
        : this.options.chunkSize;

      this.emit('progress', bytesWritten, this.totalSize);
    }

    this.writer.close();

    if (this.options.filename) {
      const destDirAndFilename = path.resolve(path.dirname(this.destFilepath), this.options.filename);

      fs.rename(this.destFilepath, destDirAndFilename, (err) => {
        if (err) {
          this.emit('error', err);
        }

        this.destFilepath = destDirAndFilename;
      });
    }

    this.emit('done', this.destFilepath, bytesWritten);
  }
}

module.exports = Concatenator;

examples.js

const Concatenator = require('./Concatenator');

const chunkFilepaths = [
  '/path/to/upload/dir/x00',
  '/path/to/upload/dir/x01',
  '/path/to/upload/dir/x02'
];

/**
 * Basic example
 * Concatenates given chunks to first chunk.
 */
const cctr1 = new Concatenator(chunkFilepaths);

cctr1.once('ready', () => {
  cctr1.start();
});
cctr1.once('done', (finalFile) => {
  // `finalFile` equals to '/path/to/upload/dir/x00'
});

/**
 * Example with destination file
 * Concatenates given chunks to destination file.
 * Before the concatenation creates the destination file, or if
 * the destination file exists append to it. This is probably
 * an unwanted behaviour, so being sure that the destination
 * file doesn't exist is your responsibility.
 */
const cctr2 = new Concatenator(chunkFilepaths, '/path/to/destination/file');

cctr2.once('ready', () => {
  cctr2.start();
});
cctr2.once('done', (finalFile) => {
  // `finalFile` equals to '/path/to/destination/file'
});

/**
 * Complex example
 * This example defines the destination file as well as `filename` option.
 * The rules above are valid for final file also final file will be renamed
 * as `filename`. But the final file's path are not going to be changed.
 */
const cctr3 = new Concatenator(chunkFilepaths, '/path/to/destination/file', { filaname: 'new_name' });

cctr3.once('ready', () => {
  cctr3.start();
});
cctr3.once('done', (finalFile) => {
  // `finalFile` equals to '/path/to/destination/new_name'
  // Notice the parent directory path, it is same as destination file's parent
});

index.js

const Concatenator = require('./Concatenator');

const chunkFilepaths = [
  '/path/to/upload/dir/x00',
  '/path/to/upload/dir/x01',
  '/path/to/upload/dir/x02'
];
const cctr = new Concatenator(chunkFilepaths);

// You should always listen for 'error' events, like every EventEmitter class.
cctr.on('error', (err) => {
  // Do something with the error
  console.error(err);
});
// We are going to listen 'ready' event once.
cctr.once('ready', () => {
  // All promises started in constructor were settled.
  // We can start the concatenation.
  cctr.start();
});
cctr.on('progress', (processed, total) => {
  const percentage = ((100 * processed) / total).toFixed(2);

  // Sample output of the log below might be;
  // %42.13 - 4213/10000 bytes
  console.log(`%${percentage} - ${processed}/${total} bytes`);
});
// We are going to listen 'done' event once.
cctr.once('done', (finalFile, bytesWritten) => {
  // Concatenation finished
  
  // Sample output of the log below might be;
  // 10000 bytes written to '/path/to/upload/dir/x00'
  console.log(`${bytesWritten} bytes written to '${finalFile}'`);

  // Optionally you can remove all the listeners for this instance.
  cctr.removeAllListeners();
  
  // From now on you are done with this Concatenator instance. You can
  // use `finalFile` parameter to access the final (concatenated) file.
});