Reading and Writing Event Streams to S3

README.md

• Request Rate and Performance Considerations
  AWS S3 Developer Guide (API Version 2006-03-01)

• How do I ingest a large number of small files from S3? My job looks like it's stalling.
  Databricks Cloud support forum thread

• What is the best way to ingest and analyze a large S3 dataset?
  Databricks Cloud support forum thread

• How can we get S3DistCp running on DBC?
  Databricks Cloud support forum thread

• How do I improve throughput of S3 writes in a Spark Streaming scenario?
  Databricks Cloud support forum thread

• Stall on loading many Parquet files on S3
  Databricks Cloud support forum thread

• Strategies for reading large numbers of files
  Apache Spark Users Mailing List

• Dealing with Hadoop's small files problem
  Snowplow Blog Post

• s3-streamlogger
  npm package

• Maximizing Amazon S3 Performance
  Slide deck from AWS re:Invent 2013 (STG304)

• The Bleeding Edge: Spark, Parquet and S3
  AppsFlyer tech blog post by Arnon Rotem-Gal-Oz

• Using S3DistCP to Merge Many Small S3 Files

• Hadoop and S3: 6 Tips for Top Performance
  Mortar Data blog post

• s4cmd
  Super S3 command line tool (python)

• The Open Guide to Amazon Web Services: S3

• AWS EMR S3DistCp
  AWS documentation

• fetch_and_combine.py
  Sample python script to aggregate Cloudfront logs on S3

• S3mper: Consistency in the Cloud
  Netflix tech blog

• AWS EMRFS
  AWS documentation

• Are We Consistent Yet?

Somewhat related, but more specific to Spark:

• https://wrightturn.wordpress.com/2015/07/22/getting-spark-data-from-aws-s3-using-boto-and-pyspark/
• http://tech.kinja.com/how-not-to-pull-from-s3-using-apache-spark-1704509219

https://github.com/apache/spark/blob/master/docs/cloud-integration.md

Important: Cloud Object Stores are Not Real Filesystems

While the stores appear to be filesystems, underneath they are still object stores, and the difference is significant