Skip to content

Instantly share code, notes, and snippets.

View qiaohaijun's full-sized avatar
🎯
Focusing

qiao hai-jun qiaohaijun

🎯
Focusing
  • Beijing,China
View GitHub Profile
viewfs://nsX/user/hive/warehouse/default.db/web/uigs/web_uigs_wapsearch/201404/20140408/2014040815/web_uigs_wapsearch.location_pointer.2014040815 -> hdfs://cloud1016128162.wd.sjs.ss.nop.sogou-op.org:8020/logdata/uigs/appsearch//201404/20140408/*2014{-,}04{-,}08{_15,15}*
middle of get splits , path is viewfs://nsX/user/hive/warehouse/default.db/web/uigs/web_uigs_wapsearch/201404/20140408/2014040815
add hdfs://cloud1016128162.wd.sjs.ss.nop.sogou-op.org:8020/logdata/uigs/appsearch/201404/20140408/*2014{-,}04{-,}08{_15,15}* to input path
14/12/02 19:33:12 ERROR thriftserver.SparkExecuteStatementOperation: Error executing query:
java.io.IOException: Failed on local exception: java.io.EOFException; Host Details : local host is: "rsync.cloud-dev1013412128.web.sjs.vm.sogou-op.org/10.134.12.128"; destination host is: "cloud1016128162.wd.sjs.ss.nop.sogou-op.org":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1413)
at org

需要修改一下/etc/yum.repos.d/epel-7.repo

gpgcheck=0
enabled=1

这两项修改一下

安装服务软件

@qiaohaijun
qiaohaijun / select applog.channel as channel, applog.type as type, applog.parent as parent, applog.content as content, applog.deviceid as deviceid, logdate from default.web_uigs_wapsearch lateral view explode(applogs) temp as applog where logdate >= '2014112800'
Created December 4, 2014 10:40
select applog.channel as channel, applog.type as type, applog.parent as parent, applog.content as content, applog.deviceid as deviceid, logdate from default.web_uigs_wapsearch lateral view explode(applogs) temp as applog where logdate >= '2014112800' and logdate <= '2014120123';
4/12/04 18:38:35 ERROR thriftserver.SparkExecuteStatementOperation: Error executing query:
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: viewfs://nsX/user/hive/warehouse/default.db/web/uigs/web_uigs_wapsearch/201312/20131201/2013120102
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251)
at com.hadoop.mapred.DeprecatedLzoTextInputFormat.listStatus(DeprecatedLzoTextInputFormat.java:72)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270)
at com.hadoop.mapred.DeprecatedLzoTextInputFormat.getSplits(DeprecatedLzoTextInputFormat.java:112)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
at scala.Option.getOrElse(Option.scala:120)
@qiaohaijun
qiaohaijun / protobuf3.spec
Created January 4, 2018 03:35 — forked from apeyser/protobuf3.spec
RPM spec file for building protobuf version 3.0.0-beta-1 into /opt in parallel to old, old versions of protobuf
#
%global srcvers 3.0.0-beta-1
%global srcname protobuf-%{srcvers}
%global _prefix /opt/%{srcname}
#
Summary: Protocol Buffers Google''s data interchange format
Name: protobuf3
warning: core file may not match specified executable file.
[New LWP 15712]
[New LWP 15799]
[New LWP 15713]
[New LWP 15801]
[New LWP 15802]
[New LWP 15711]
[New LWP 15800]
[New LWP 15691]
[Thread debugging using libthread_db enabled]
(gdb) bt
#0 0x00007fcec8f181d7 in raise () from /lib64/libc.so.6
#1 0x00007fcec8f198c8 in abort () from /lib64/libc.so.6
#2 0x00007fcec981c9d5 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6
#3 0x00007fcec981a946 in ?? () from /lib64/libstdc++.so.6
#4 0x00007fcec981a973 in std::terminate() () from /lib64/libstdc++.so.6
#5 0x00007fcec981ab93 in __cxa_throw () from /lib64/libstdc++.so.6
#6 0x00007fcec986fa17 in std::__throw_out_of_range(char const*) () from /lib64/libstdc++.so.6
#7 0x0000000000412fd9 in std::vector<int, std::allocator<int> >::_M_range_check (this=0x14e7940, __n=1) at /usr/include/c++/4.8.2/bits/stl_vector.h:794
#8 0x0000000000412c57 in std::vector<int, std::allocator<int> >::at (this=0x14e7940, __n=1) at /usr/include/c++/4.8.2/bits/stl_vector.h:830

凸优化以及非线性机器学习算法的胜利

非线性不可求解

  1. 这个宇宙是非线性的宇宙,我们人类在其中也不过是一个非线性的存在
  2. 时间局部性,空间局部性可以用线性来拟合。

世界本身是复杂的

  1. 混沌是正常的
  2. 蝴蝶效应是一个例子

这个很平常的代码其实本身还是有些技术含量的

语言律师

  1. 把c++标准倒背如流
  2. 人肉编译器

本末倒置

  1. 本是创造价值
  2. 末是成为人肉编译器
  3. 完成需求,实现价值才是

熟练掌握c++和完成需求之间的关系