devender-yadav

Array

• Two Sum
• Best Time to Buy and Sell Stock
• Contains Duplicate
• Product of Array Except Self
• Maximum Subarray
• Maximum Product Subarray
• Find Minimum in Rotated Sorted Array

devender-yadav

df = spark.range(1, 7, 2)
df.show()

rdd = df.rdd
rdd = rdd.cache()
print("Storage Level - {}".format(rdd.getStorageLevel()))

print("Is Checkpointed - {}".format(rdd.isCheckpointed()))
print("Checkpoint File - {}".format(rdd.getCheckpointFile()))

devender-yadav

import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.mllib.linalg._
import org.apache.spark.{SparkConf, SparkContext}

// To use the latest sparse SVD implementation, please build your spark-assembly after this
// change: https://github.com/apache/spark/pull/1378

// Input tsv with 3 fields: rowIndex(Long), columnIndex(Long), weight(Double), indices start with 0
// Assume the number of rows is larger than the number of columns, and the number of columns is
// smaller than Int.MaxValue

devender-yadav

Project Structure

-myproj
  -proj
  -xx.py
  -dir1
    -xx.py
  -dir2
    -xx.py

-tests

devender-yadav

Static code analysis for python application

Pylint

How to install

pip install pylint

How to run

devender-yadav

Use local debian mirror - https://www.debian.org/mirror/sponsors.html

RUN echo \
   'deb http://mirror.cse.iitk.ac.in/debian/ stretch main\n \
    deb http://security.debian.org/debian-security stretch/updates main\n \
    deb http://mirror.cse.iitk.ac.in/debian/ stretch-updates main\n' \
    > /etc/apt/sources.list

devender-yadav

Download instantclient from Oracle website

Go to - https://www.oracle.com/technetwork/topics/intel-macsoft-096467.html

Download instantclient-basic and instantclient-sdk appropriate version.

For Example,

• instantclient-basic-macos.x64-12.1.0.2.0.zip
• instantclient-sdk-macos.x64-12.1.0.2.0.zip

devender-yadav

import sys
!{sys.executable} -m pip install PyAthena

from pyathena import connect
import pandas as pd
conn = connect(s3_staging_dir='s3://dev-sample-data123/temp',
               region_name='ap-south-1')

df = pd.read_sql("SELECT * FROM rank_table limit 10;", conn)

devender-yadav

import feedparser
  
NewsFeed = feedparser.parse("https://timesofindia.indiatimes.com/rssfeedstopstories.cms")

print ('Number of RSS posts :', len(NewsFeed.entries))

entry = NewsFeed.entries[1]
print('Post Title :',entry.title)

devender-yadav

Let us assume, initally mongo is using /data/db1 dbpath and then /data/db2

Start mongo /data/db1

sudo mongod --dbpath=/data/db1

Export data from mongo /data/db1

mongoexport --db test_db --collection collection1 --out tb_collection1_db1.json