// Explains Dead Letter Queue + bad event handling + blocking
-
https://stackoverflow.com/questions/32501985/amazon-kinesis-aws-lambda-retries
-
https://aws.amazon.com/blogs/compute/amazon-kinesis-firehose-data-transformation-with-aws-lambda/
// You set the amount of events(batchsize) + guarantees of concurrency per shard+ plus key only active in 1 shard at the time
- https://github.com/shevchenkos/DynamoDbBackUp/blob/master/lambda/index.js
- https://github.com/shevchenkos/DynamoDbBackUp/blob/master/lib/backup.js#L15
Lambda Handle stream
Handle Record
- // Focused on backup
- // Has both replicate and backup functionality
- https://blog.mapbox.com/scaling-mapbox-infrastructure-with-dynamodb-streams-d53eabc5e972
- https://github.com/mapbox/dynamodb-replicator
- https://github.com/mapbox/dynamodb-replicator/blob/75b8fb362661d77dbdcc227ac6a71b77c3ab8c43/index.js
Lambda Handle Stream
- https://github.com/mapbox/dynamodb-replicator/blob/db3224a8f846be7dc8ac68f3856e7eb231464ff1/index.js#L126 Handle Record
- https://github.com/mapbox/dynamodb-replicator/blob/db3224a8f846be7dc8ac68f3856e7eb231464ff1/index.js#L179
- // Restore S3=> Dynamo using batchwrites
- // Also restore
- https://github.com/PageUpPeopleOrg/dynamo-incremental-restore/blob/master/lib/s3ToDynamo.js
- https://github.com/PageUpPeopleOrg/dynamo-incremental-restore
- //Use MD5 sum for storing S3 key - hotspots
- Multi-tenant flag for S3 location
- https://medium.com/pageup-tech/dynamodb-incremental-backups-bb9e0c3db8f2
Uses S3 versions to check up until certain point in time:
- // Restore via kinisis stream
- https://github.com/Stockflare/dynamodb-restore/blob/master/dynamodb-restore.rb#L81
- https://github.com/Stockflare/dynamodb-backup
- https://github.com/Stockflare/dynamodb-restore
- https://github.com/Stockflare/lambda-dynamodb-put
- https://github.com/jhurliman/node-rate-limiter
- https://github.com/chilts/oibackoff
- https://medium.com/@CodingJoe/dealing-with-dynamodb-write-capacity-limits-and-lambda-timeouts-f4e08d9f4b4f
- More of a replication script , not for backup
- //Deduplicate updates to the same key
- // http.globalAgent.maxSockets = 500;
- // Max inflight request checking for dynamo
- AWS DynamoDB Cross Region Replication using Lambda (Node.js) - http://www.mylinuxguru.com/?p=698
Python based Handling of Batches when writing:
- https://github.com/ReutersMedia/dynamodb-circular-repl
- https://github.com/ReutersMedia/dynamodb-circular-repl/blob/master/replicator.py#L89
- https://github.com/Signiant/dynamodb-replication
- https://github.com/Signiant/dynamodb-replication/blob/master/lambda/replicator/index.js#L64 - code retry/delay
- Nodejs/S3 Pipe/gzip/Metadata of tables https://github.com/CU-CloudCollab/aws-examples/blob/master/lambda/dynamodb-backup/backup.js
- Python/Using Dynamo->Lambda->Firehose->S3 https://github.com/awslabs/dynamodb-continuous-backup/blob/master/src/dynamo_continuous_backup.py#L173
- https://github.com/awslabs/lambda-streams-to-firehose
- Talks of Issues bout order of S3: http://www.abhayachauhan.com/?p=88
- Nice tips on using cloudtrail to automagically monitor creation of new dbs and start backup
- Real-time Back and Restore with Amazon DynamoDB and AWS Lambda - Ian Meyers - Serverless - https://www.youtube.com/watch?v=8xvXoz-amtk&list=PLnwBrRU5CSTmM4OsUzDaog2pqadQSxKBv&index=22
- Slides https://www.slideshare.net/AmazonWebServices/serverless-realtime-backup
- Indirectly useful - aws lambda replication + dealing with multi write issues
- AWS re:Invent 2016: Cross-Region Replication with Amazon DynamoDB Streams (DAT201) - YouTube - https://www.youtube.com/watch?v=NtaTC2Fq7Wo
- Multi write - dynamodb - timestamp, replicateme, writeId - https://youtu.be/NtaTC2Fq7Wo?t=32m13s
- The replication handler - https://youtu.be/NtaTC2Fq7Wo?t=28m3s
- keeps track of #items transferred / not lambda based / scans table
- https://github.com/VizvaStudio/dynamodb-util
- https://github.com/VizvaStudio/dynamodb-util/blob/master/lib/backup-helper.js#L51
- Make backups secure: https://medium.com/@SkyscannerCodevoyagers/backing-up-an-amazon-web-services-dynamodb-889d4787d291
- Medium - Detect Hotspots - https://medium.engineering/how-medium-detects-hotspots-in-dynamodb-using-elasticsearch-logstash-and-kibana-aaa3d6632cfd
- DynamoDB pitfall: limited throughout due to hot partitions | cloudonaut - https://cloudonaut.io/dynamodb-pitfall-limited-throughput-due-to-hot-partitions/
- One Year of DynamoDB at Timehop – Building Timehop – Medium - https://medium.com/building-timehop/one-year-of-dynamodb-at-timehop-f761d9fe5fa1
- How sharding works: https://medium.com/@jeeyoungk/how-sharding-works-b4dec46b3f6
- bchew/dynamodump: Simple backup and restore for Amazon DynamoDB using boto - https://github.com/bchew/dynamodump
- https://www.slideshare.net/mobile/AmazonWebServices/design-patterns-using-amazon-dynamodb
- DynamoDB and Cross-Region Replication : https://www.youtube.com/watch?v=Oa2ptw5LsOs
- AWS re:Invent 2016: Deep Dive on Amazon DynamoDB (DAT304) - YouTube - https://www.youtube.com/watch?v=bCW3lhsJKfw
- http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html#GuidelinesForTables.Partitions
- Demystifing DynamoDB Streams - https://www.infoq.com/presentations/dynamodb-streams
- awslabs/dynamodb-cross-region-library#38
- https://github.com/awslabs/dynamodb-cross-region-library
- Dynamodb replication: https://www.linkedin.com/pulse/all-your-replication-servers-belong-us-dave-north
- https://blog.sungardas.com/CTOLabs/2015/09/planning-for-failures-with-amazon-dynamodb-2/
- https://github.com/awslabs/dynamodb-import-export-tool
- Example using Dynamodb -> Lambda -> Kinesis - https://noise.getoto.net/2016/08/03/serverless-cross-account-stream-replication-using-aws-lambda-amazon-dynamodb-and-amazon-kinesis-firehose/
- AWS Announcement of Lambda & Dynamodb streams - https://aws.amazon.com/blogs/aws/dynamodb-update-triggers-streams-lambda-cross-region-replication-app/
- // Archiver via lambda but not dynamodb streams https://github.com/yegor256/dynamo-archive
- http://discourse.snowplowanalytics.com/t/real-time-reporting-using-aws-lambda-and-dynamodb-a-tutorial-to-compute-the-number-of-players-in-a-game-level-on-the-snowplow-event-stream-1-2/1008