pipeline.md

The pipeline is a sequence of configurable 'transformers'. They essentially take in a file and pass one or more files to the next stage.

For example the current pipeline might look like this:

• Index (finds changed files on the filesystem and passes them through
  • Encrypt
    • Cache
    • Upload
  • Database (stores item metadata in the database)

But if someone wants to, for example, add thumbnails for every image on the remote end, they can do this:

• Index (finds changed files on the filesystem and passes them through
  • Filter (only images)
    • Thumbnail (takes indexed files and creates thumbnails for them)
      • Encrypt, cache, upload
  • Encrypt
    • Cache
    • Upload
  • Database (stores item metadata in the database)

This transformation could be performed by simply adding a line in the XML.

Certain things might have to be handled differently on retrieval but in general the process should be performed in reverse.