Skip to content

Instantly share code, notes, and snippets.

@haozhun

haozhun/0NOTE

Last active April 4, 2018 22:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save haozhun/2e4b7c52bf3c56c03ed06e1ad895e198 to your computer and use it in GitHub Desktop.
Save haozhun/2e4b7c52bf3c56c03ed06e1ad895e198 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
0NOTE
server zone +05:45
client1 (insert/read) zone -07:00
client2 (read) zone -04:00
ORC RCBinary RCText Text
Insert 06:00:00 06:00:00 06:00:00 06:00:00
Read 1 (same zone as insert) 18:45:00 06:00:00 18:45:00 18:45:00
Read 2 (different zone) 18:45:00 09:00:00 18:45:00 18:45:00
Raw
raw result
[root@hadoop-master /]# export _JAVA_OPTIONS=-Duser.timezone=America/Los_Angeles
[root@hadoop-master /]# hive
Picked up _JAVA_OPTIONS: -Duser.timezone=America/Los_Angeles
WARNING: Use "yarn jar" to launch YARN applications.
Picked up _JAVA_OPTIONS: -Duser.timezone=America/Los_Angeles
Logging initialized using configuration in jar:file:/usr/hdp/2.6.3.0-235/hive/lib/hive-common-1.2.1000.2.6.3.0-235.jar!/hive-log4j.properties
hive> CREATE TABLE t_textfile
> STORED AS TEXTFILE
> AS
> SELECT timestamp '2018-04-04 06:00:00' ct;
Query ID = root_20180404152235_5ee90845-c6cf-4f40-80dd-76153421a4e2
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1522878372036_0005, Tracking URL = http://hadoop-master:8088/proxy/application_1522878372036_0005/
Kill Command = /usr/hdp/2.6.3.0-235/hadoop/bin/hadoop job -kill job_1522878372036_0005
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2018-04-04 15:22:45,411 Stage-1 map = 0%, reduce = 0%
2018-04-04 15:22:50,669 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.8 sec
MapReduce Total cumulative CPU time: 1 seconds 800 msec
Ended Job = job_1522878372036_0005
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://hadoop-master:9000/user/hive/warehouse/.hive-staging_hive_2018-04-04_15-22-35_107_6694062079913526934-1/-ext-10001
Moving data to directory hdfs://hadoop-master:9000/user/hive/warehouse/t_textfile
Table default.t_textfile stats: [numFiles=1, numRows=1, totalSize=20, rawDataSize=19]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Cumulative CPU: 1.8 sec HDFS Read: 3692 HDFS Write: 94 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 800 msec
OK
Time taken: 17.324 seconds
hive> CREATE TABLE t_rctext
> ROW FORMAT SERDE "org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe"
> STORED AS RCFile
> AS
> SELECT timestamp '2018-04-04 06:00:00' ct;
Query ID = root_20180404152253_4498d074-0693-4883-8f4f-a2693bc55b5e
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1522878372036_0006, Tracking URL = http://hadoop-master:8088/proxy/application_1522878372036_0006/
Kill Command = /usr/hdp/2.6.3.0-235/hadoop/bin/hadoop job -kill job_1522878372036_0006
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2018-04-04 15:23:00,276 Stage-1 map = 0%, reduce = 0%
2018-04-04 15:23:06,506 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.77 sec
MapReduce Total cumulative CPU time: 1 seconds 770 msec
Ended Job = job_1522878372036_0006
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://hadoop-master:9000/user/hive/warehouse/.hive-staging_hive_2018-04-04_15-22-53_088_2664436609596790944-1/-ext-10001
Moving data to directory hdfs://hadoop-master:9000/user/hive/warehouse/t_rctext
Table default.t_rctext stats: [numFiles=1, numRows=1, totalSize=92, rawDataSize=19]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Cumulative CPU: 1.77 sec HDFS Read: 3748 HDFS Write: 164 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 770 msec
OK
Time taken: 14.929 seconds
hive> CREATE TABLE t_rcbinary
> ROW FORMAT SERDE "org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe"
> STORED AS RCFile
> AS
> SELECT timestamp '2018-04-04 06:00:00' ct;
Query ID = root_20180404152308_c2b4c2ce-223f-432a-b2cd-953d3d2307bd
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1522878372036_0007, Tracking URL = http://hadoop-master:8088/proxy/application_1522878372036_0007/
Kill Command = /usr/hdp/2.6.3.0-235/hadoop/bin/hadoop job -kill job_1522878372036_0007
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2018-04-04 15:23:15,766 Stage-1 map = 0%, reduce = 0%
2018-04-04 15:23:20,974 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.65 sec
MapReduce Total cumulative CPU time: 1 seconds 650 msec
Ended Job = job_1522878372036_0007
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://hadoop-master:9000/user/hive/warehouse/.hive-staging_hive_2018-04-04_15-23-08_882_2288308614577995945-1/-ext-10001
Moving data to directory hdfs://hadoop-master:9000/user/hive/warehouse/t_rcbinary
Table default.t_rcbinary stats: [numFiles=1, numRows=1, totalSize=77, rawDataSize=4]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Cumulative CPU: 1.65 sec HDFS Read: 3760 HDFS Write: 151 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 650 msec
OK
Time taken: 13.581 seconds
hive> CREATE TABLE t_orc
> STORED AS ORC
> AS
> SELECT timestamp '2018-04-04 06:00:00' ct;
Query ID = root_20180404152326_d32139a6-c3fa-44f0-98a8-c79fc9ba3d00
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1522878372036_0008, Tracking URL = http://hadoop-master:8088/proxy/application_1522878372036_0008/
Kill Command = /usr/hdp/2.6.3.0-235/hadoop/bin/hadoop job -kill job_1522878372036_0008
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2018-04-04 15:23:34,670 Stage-1 map = 0%, reduce = 0%
2018-04-04 15:23:39,989 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.72 sec
MapReduce Total cumulative CPU time: 1 seconds 720 msec
Ended Job = job_1522878372036_0008
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://hadoop-master:9000/user/hive/warehouse/.hive-staging_hive_2018-04-04_15-23-26_684_8139297715596225192-1/-ext-10001
Moving data to directory hdfs://hadoop-master:9000/user/hive/warehouse/t_orc
Table default.t_orc stats: [numFiles=1, numRows=1, totalSize=232, rawDataSize=40]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Cumulative CPU: 1.72 sec HDFS Read: 3736 HDFS Write: 301 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 720 msec
OK
Time taken: 14.738 seconds
hive> select * from t_textfile;
OK
2018-04-04 18:45:00
Time taken: 0.106 seconds, Fetched: 1 row(s)
hive> select * from t_rctext;
OK
2018-04-04 18:45:00
Time taken: 0.087 seconds, Fetched: 1 row(s)
hive> select * from t_rcbinary;
OK
2018-04-04 06:00:00
Time taken: 0.081 seconds, Fetched: 1 row(s)
hive> select * from t_orc;
OK
2018-04-04 18:45:00
Time taken: 0.091 seconds, Fetched: 1 row(s)
hive> [root@hadoop-masteexport _JAVA_OPTIONS=-Duser.timezone=America/New_York
[root@hadoop-master /]# hive
Picked up _JAVA_OPTIONS: -Duser.timezone=America/New_York
WARNING: Use "yarn jar" to launch YARN applications.
Picked up _JAVA_OPTIONS: -Duser.timezone=America/New_York
Logging initialized using configuration in jar:file:/usr/hdp/2.6.3.0-235/hive/lib/hive-common-1.2.1000.2.6.3.0-235.jar!/hive-log4j.properties
hive> select * from t_textfile;
OK
2018-04-04 18:45:00
Time taken: 1.433 seconds, Fetched: 1 row(s)
hive> select * from t_rctext;
OK
2018-04-04 18:45:00
Time taken: 0.118 seconds, Fetched: 1 row(s)
hive> select * from t_rcbinary;
OK
2018-04-04 09:00:00
Time taken: 0.146 seconds, Fetched: 1 row(s)
hive> select * from t_orc;
OK
2018-04-04 18:45:00
Time taken: 0.117 seconds, Fetched: 1 row(s)
hive>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment