Skip to content

Instantly share code, notes, and snippets.

@arammaliachi

arammaliachi/count_fields.py

Created May 29, 2020 19:45
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save arammaliachi/68abd8fc6317bcab5ba7f1e8cd85a6a3 to your computer and use it in GitHub Desktop.
Save arammaliachi/68abd8fc6317bcab5ba7f1e8cd85a6a3 to your computer and use it in GitHub Desktop.
Download ZIP
Count columns of a dynamic frame, AWS Glue Pyspark
Raw
count_fields.py
from awsglue.dynamicframe import DynamicFrame
from pyspark.context import SparkContext
from awsglue.context import GlueContext
glueContext = GlueContext(SparkContext.getOrCreate())
datasource0 = glueContext.create_dynamic_frame.from_catalog(database='my database', table_name='my_table')
#Assuming <datasource0> is an object of the class <class 'awsglue.dynamicframe.DynamicFrame'>
columns_count = len(datasource0.schema().__dict__['fields']))
print('fields: {}'.format(columns_count))
#stdout:
#fields: 18203
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment