Skip to content

Instantly share code, notes, and snippets.

View decompress_lzo.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@arammaliachi
arammaliachi / count_fields.py
Created May 29, 2020
Count columns of a dynamic frame, AWS Glue Pyspark
View count_fields.py
from awsglue.dynamicframe import DynamicFrame
from pyspark.context import SparkContext
from awsglue.context import GlueContext
glueContext = GlueContext(SparkContext.getOrCreate())
datasource0 = glueContext.create_dynamic_frame.from_catalog(database='my database', table_name='my_table')
#Assuming <datasource0> is an object of the class <class 'awsglue.dynamicframe.DynamicFrame'>
columns_count = len(datasource0.schema().__dict__['fields']))
print('fields: {}'.format(columns_count))