Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Count columns of a dynamic frame, AWS Glue Pyspark
from awsglue.dynamicframe import DynamicFrame
from pyspark.context import SparkContext
from awsglue.context import GlueContext
glueContext = GlueContext(SparkContext.getOrCreate())
datasource0 = glueContext.create_dynamic_frame.from_catalog(database='my database', table_name='my_table')
#Assuming <datasource0> is an object of the class <class 'awsglue.dynamicframe.DynamicFrame'>
columns_count = len(datasource0.schema().__dict__['fields']))
print('fields: {}'.format(columns_count))
#stdout:
#fields: 18203
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment