César Aarón Fernández Niño cesar1091
🏠
cesar1091
/ Vangrant file configuration
Created
October 1, 2022 13:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #-*-mode: ruby-*- | |
| #vi: set ft=ruby: | |
| Vagrant.configure("2") do|config| | |
| config.vm.box = "itversity/centos7spark" | |
| config.vm.network "forwarded_port", guest: 8888, host: 8888 | |
| config.vm.network "forwarded_port", guest: 4040, host: 4040 | |
| config.vm.provider "virtualbox" do|vb| | |
| vb.cpus = "2" |
cesar1091
/ Copiamos los ficheros
Created
October 1, 2022 13:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| hdfs dfs -put /data/retail_db /public | |
| hdfs dfs -ls /public/retail_db |
cesar1091
/ pyspark init
Created
October 1, 2022 13:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| pyspark --packages org.apache.spark:spark-avro_2.11:2.4.4 |
cesar1091
/ validate_ejercicio1
Last active
October 1, 2022 14:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| validate = spark.read.format("avro").load("/user/vagrant/lab1/pregunta1/orders_avro") | |
| validate.printSchema() | |
| validate.count() | |
| validate.show() |
cesar1091
/ hdfs consulta
Created
October 1, 2022 14:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| hdfs dfs -ls /user/vagrant/lab1/pregunta1/orders_avro |
cesar1091
/ hdfs consulta 2
Created
October 1, 2022 14:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| hdfs dfs -ls /user/vagrant/lab1/pregunta2/customer_parquet |
cesar1091
/ hdfs consulta 3
Created
October 1, 2022 15:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| hdfs dfs -ls /user/vagrant/lab1/pregunta3/resultado |
cesar1091
/ hdfs consulta 4
Created
October 1, 2022 15:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| hdfs dfs -ls /user/vagrant/lab1/pregunta4/resultado |
cesar1091
/ Ejercicio6.py
Created
October 1, 2022 22:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| customer = spark.read.format("parquet").load("/user/vagrant/lab1/pregunta2/customer") | |
| customer.createOrReplaceTempView("customer") | |
| val result = spark.sql("select customer_id, concat(substring(customer_fname,1,3),' ', customer_lname) as name, customer_street from customer") | |
| result.rdd.map(lambda x: "\t".join(map(str,x))).saveAsTextFile("/user/vagrant/lab1/pregunta6/resultado","org.apache.hadoop.io.compress.BZip2Codec") |
cesar1091
/ Ejercicio5.py
Last active
October 1, 2022 22:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ProductSchema = StructType([StructField("product_id", IntegerType(), True), | |
| StructField("product_category_id", IntegerType(), True), | |
| StructField("product_name", StringType(), True), | |
| StructField("product_description", StringType(), True), | |
| StructField("product_price", FloatType(), True), | |
| StructField("product_image", StringType(), True)]) | |
| product = spark.read.format("csv").option("inferSchema","true").schema(ProductSchema).load("/public/retail_db/products/part-00000") | |
| product.createOrReplaceTempView("product") | |
| result =spark.sql("select product_id, max(product_price) as max_price from product group by product_id") |
OlderNewer