Skip to content

Instantly share code, notes, and snippets.

@Nxtra
Nxtra / blog.md
Created Jan 28, 2021
athena blog
View blog.md

Serverless data pipelines: ETL workflow with Step Functions and Athena

This blog is Part 3 of a multi-part series around analysing Flanders’ traffic whilst leveraging the power of cloud components!
For part 1 see: https://medium.com/cloudway/real-time-data-processing-with-kinesis-data-analytics-ad52ad338c6d
For part 2 see: https://medium.com/cloubis/serverless-data-transform-with-kinesis-e468abd33409

What is our goal?

This blog aims to explore the use of the AWS Glue service in conjunction with the AWS Athena service to repartition raw streaming data events.
We previously landed these events on an Amazon S3 bucket partitioned according to the processing time on Kinesis.

@Nxtra
Nxtra / openapi-generator-maven-plugin-configuration.xml
Last active Jan 16, 2021
An example of how to configure the openapi-generator plugin for your maven projects
View openapi-generator-maven-plugin-configuration.xml
<plugin>
<groupId>org.openapitools</groupId>
<artifactId>openapi-generator-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<skipValidateSpec>false</skipValidateSpec>
@Nxtra
Nxtra / index.js
Last active Dec 6, 2020
A lambda function to resolve the editPostAndComments mutation.
View index.js
const AWS = require("aws-sdk");
const docClient = new AWS.DynamoDB.DocumentClient();
const POSTTABLE = process.env.POSTTABLE;
const COMMENTTABLE = process.env.COMMENTTABLE;
const resolvers = {
Mutation: {
deletePostAndComments: (event) => {
@Nxtra
Nxtra / template.json
Last active Nov 26, 2020
Amplify lambda access dynamodb api resources policy
View template.json
"AccessDynamoDBApiResourcesPolicy": {
"DependsOn": ["LambdaExecutionRole"],
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": "amplify-lambda-execution-policy",
"Roles": [
{
"Ref": "LambdaExecutionRole"
}
],
@Nxtra
Nxtra / schema.graphql
Last active Nov 25, 2020
Amplify schema with relations and custom mutation
View schema.graphql
type Blog @model {
id: ID!
name: String!
posts: [Post] @connection(keyName: "byBlog", fields: ["id"])
}
type Post @model @key(name: "byBlog", fields: ["blogID"]) {
id: ID!
title: String!
blogID: ID!
@Nxtra
Nxtra / template.yaml
Last active Nov 8, 2020
Setup a billing alarm for your AWS account with cloudformation
View template.yaml
AWSTemplateFormatVersion: "2010-09-09"
Description: Billing Alerts for your AWS Account
Parameters:
Email:
Type: String
Default: youremail@somerandomdomain.com
Description: The email address to receive alerts per email
Phone:
@Nxtra
Nxtra / example.test.js
Created Nov 6, 2020
Call custom signIn command before cypress test suite
View example.test.js
describe("Example test", () => {
before(() => {
cy.signIn();
});
after(() => {
cy.clearLocalStorageSnapshot();
cy.clearLocalStorage();
});
@Nxtra
Nxtra / commands.js
Created Nov 6, 2020
Cypress custom Amplify command.
View commands.js
Cypress.Commands.add("signIn", () => {
cy.then(() => Auth.signIn(username, password)).then((cognitoUser) => {
const idToken = cognitoUser.signInUserSession.idToken.jwtToken;
const accessToken = cognitoUser.signInUserSession.accessToken.jwtToken;
const makeKey = (name) => `CognitoIdentityServiceProvider.${cognitoUser.pool.clientId}.${cognitoUser.username}.${name}`;
cy.setLocalStorage(makeKey("accessToken"), accessToken);
cy.setLocalStorage(makeKey("idToken"), idToken);
cy.setLocalStorage(
@Nxtra
Nxtra / commands.js
Created Oct 24, 2020
Create a Cypress command to signIn with AWS Amplify before executing your tests! Repository: https://github.com/Nxtra/cypress-amplify-auth-test
View commands.js
const Auth = require("aws-amplify").Auth;
import "cypress-localstorage-commands";
const username = Cypress.env("username");
const password = Cypress.env("password");
const userPoolId = Cypress.env("userPoolId");
const clientId = Cypress.env("clientId");
const awsconfig = {
View event.JSON
{
"beschrijvende_id": "H101L20",
"unieke_id": "3159",
"lve_nr": "18",
"tijd_waarneming": "2020-04-06T17:51:00+01:00",
"tijd_laatst_gewijzigd": "2020-04-06T17:52:20+01:00",
"actueel_publicatie": "1",
"beschikbaar": "1",
"defect": "0",
"geldig": "0",