Skip to content

Instantly share code, notes, and snippets.

View lvthillo's full-sized avatar

Lorenz Vanthillo lvthillo

View GitHub Profile
@lvthillo
lvthillo / ColorQueueSubscription.yaml
Created August 14, 2020 13:21
ColorQueueSubscription
ColorQueueSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: !GetAtt ColorQueue.Arn
Protocol: sqs
RawMessageDelivery: True
TopicArn: !Ref SNSTopic
@lvthillo
lvthillo / ColorQueueSubscription.yaml
Created August 14, 2020 13:21
ColorQueueSubscription
ColorQueueSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: !GetAtt ColorQueue.Arn
Protocol: sqs
RawMessageDelivery: True
TopicArn: !Ref SNSTopic
@lvthillo
lvthillo / QueuePolicy.yaml
Created August 14, 2020 13:19
The policy grants the SNS topic the SendMessage permission for the five queues
QueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref ColorQueue
- !Ref BlueYellowQueue
- !Ref RedQueue
- !Ref GreenHighQueue
- !Ref GreenLowQueue
PolicyDocument:
@lvthillo
lvthillo / load_data.py
Created March 18, 2020 16:37
Part of load_data function
import os
import json
import boto3
import logging
from botocore.exceptions import ClientError
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def get_ddb_connection():
@lvthillo
lvthillo / read.py
Last active December 17, 2019 19:56
textract = boto3.client('textract')
textract.start_document_analysis(
DocumentLocation={
'S3Object': {
'Bucket': bucket,
'Name': key
}
},
JobTag=key + '_Job',
FeatureTypes=[
@lvthillo
lvthillo / kibana.yml
Created June 21, 2019 17:54
Setup of Kibana
kibana:
image: docker.elastic.co/kibana/kibana:7.1.1
depends_on:
- logstash
ports:
- 5601:5601
networks:
- logging-network
@lvthillo
lvthillo / pipelines.yml
Created June 21, 2019 17:49
Logstash pipeline configuration
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html
- pipeline.id: main
path.config: "/usr/share/logstash/pipeline"
@lvthillo
lvthillo / logstash.conf
Created June 21, 2019 17:44
logstash.conf
input {
gelf {}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
}
}
@lvthillo
lvthillo / logstash.yml
Last active June 21, 2019 17:40
Setup of Logstash
logstash:
image: docker.elastic.co/logstash/logstash:7.1.1
depends_on:
- elasticsearch
ports:
- 12201:12201/udp
volumes:
- ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
networks:
- logging-network
@lvthillo
lvthillo / es.yml
Created June 20, 2019 20:36
Setup of ElasticSearch
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
environment:
- discovery.type=single-node
- xpack.security.enabled=false
networks:
- logging-network