Skip to content

Instantly share code, notes, and snippets.

@zsxwing
Last active April 3, 2017 04:45
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save zsxwing/5924191 to your computer and use it in GitHub Desktop.
Save zsxwing/5924191 to your computer and use it in GitHub Desktop.
Pig and Hive source codes overview. Small hint: use "explain" to find the operation keywords, and search them in the source codes to find the corresponding class.
Pig:
org.apache.pig.newplan.logical.relational.LogicalPlan
org.apache.pig.backend.hadoop.executionengine.physicalLayer.plans.PhysicalPlan
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.plans.MROperPlan
org.apache.pig.parser.QueryParserDriver.parse(String)
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(LogicalPlan,Properties)
org.apache.pig.PigServer.launchPlan(PhysicalPlan,String)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(PhysicalPlan,PigContext)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(MROperPlan,MapReduceOper,Configuration,PigContext)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(MROperPlan,String)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(PhysicalPlan,String,PigContext)
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.constructLROutput(List<Result>,List<Result>,Tuple)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce.Map.collect(Context,Tuple)
org.apache.pig.impl.io.PigNullableWritable
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POJoinPackage.getNext(Tuple)
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin
Hive:
org.apache.hadoop.hive.cli.CliDriver
org.apache.hadoop.hive.ql.Driver
org.apache.hadoop.hive.ql.Driver.run(String)
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(String,Context)
org.apache.hadoop.hive.ql.parse.HiveLexer
org.apache.hadoop.hive.ql.parse.HiveParser
ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
org.apache.hadoop.hive.ql.parse.ASTNode
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer
org.apache.hadoop.hive.ql.QueryPlan
org.apache.hadoop.hive.ql.Driver.compile(String,boolean)
org.apache.hadoop.hive.ql.exec.Task<TextendsSerializable>
org.apache.hadoop.hive.ql.exec.CopyTask
org.apache.hadoop.hive.ql.exec.MoveTask
org.apache.hadoop.hive.ql.exec.TaskRunner
org.apache.hadoop.hive.ql.Driver.execute()
org.apache.hadoop.hive.ql.exec.ExecDriver
org.apache.hadoop.hive.ql.exec.ExecMapper
org.apache.hadoop.hive.ql.exec.ExecReducer
org.apache.hadoop.hive.ql.plan.MapredWork
org.apache.hadoop.hive.ql.exec.MapOperator
org.apache.hadoop.hive.ql.plan.MapredWork.getReducer()
org.apache.hadoop.hive.ql.exec.Operator<TextendsOperatorDesc>
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator
org.apache.hadoop.hive.ql.exec.FileSinkOperator
org.apache.hadoop.hive.ql.exec.CommonJoinOperator<TextendsJoinDesc>
org.apache.hadoop.hive.ql.exec.JoinOperator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment