Skip to content

Instantly share code, notes, and snippets.

@evilUrge
Created January 27, 2022 14:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save evilUrge/1be714572a9c477027237bb8a9a45f33 to your computer and use it in GitHub Desktop.
Save evilUrge/1be714572a9c477027237bb8a9a45f33 to your computer and use it in GitHub Desktop.
Create a parquet columns and Glue table columns from a common object
import { Schema } as GlueSchema from '@aws-cdk/aws-glue';
/**
* Create a parquet columns and Glue table columns from a common object
* @param attributes { name: type }
* @returns {
* parquetSchema: { name: { type: 'type' }, ...},
* glueSchema: [
* { name: 'name', type: 'UTF8' },
* ...
* ]
* }
*/
export function generateCommonColumns(attributes: Record<string, 'string'|'int'|'bigInt'|'boolean'|'date'>) {
const mapper = {
parquet: {
string: { type: 'UTF8', optional: true },
int: { type: 'INT32', optional: true },
bigInt: { type: 'INT64', optional: true },
boolean: { type: 'BOOLEAN', optional: true },
date: { type: 'TIMESTAMP_MILLIS', optional: true },
},
glueColumns: {
string: GlueSchema.STRING,
int: GlueSchema.INTEGER,
bigInt: GlueSchema.BIG_INT,
boolean:GlueSchema.BOOLEAN,
date: GlueSchema.TIMESTAMP
},
}
const
parquetColumns = {},
glueColumns:any = [];
Object.keys(attributes).forEach((attrName) => {
if (mapper.glueColumns.hasOwnProperty(attributes[attrName])) {
Object.assign(parquetColumns,{ [attrName]: mapper.parquet[attributes[attrName]] });
glueColumns.push({ "name": attrName, type: mapper.glueColumns[attributes[attrName]] })
} else throw new Error(`${attrName} is not a valid attribute type`);
});
return {parquetColumns,glueColumns}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment