Skip to content

Instantly share code, notes, and snippets.

View zsxwing's full-sized avatar
:octocat:

Shixiong Zhu zsxwing

:octocat:
  • Databricks, Inc.
  • San Francisco
View GitHub Profile
@zsxwing
zsxwing / gist:c06d42c98145d1b60efe04c7e3122bec
Created April 28, 2023 07:59
git diff rel/release-2.7.7 rel/release-2.8.1 -- hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
diff --git a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
index 30551b989f1..b119bc7af67 100644
--- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
+++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
@@ -50,10 +50,10 @@
<property>
<name>hadoop.http.filter.initializers</name>
<value>org.apache.hadoop.http.lib.StaticUserWebFilter</value>
- <description>A comma separated list of class names. Each class in the list
- must extend org.apache.hadoop.http.FilterInitializer. The corresponding
val version = getCurrentVersionFromConnector
val appId = getCurrentAppIdFromConnector
val txn = startTxn()
val versionInTxn = txn.txnVersion(appId)
if (version <= versionInTxn) {
// Skip the write that's done
return
}
// write files and commit
package rx.android.observables;
import rx.Observable;
import rx.Observable.OnSubscribe;
import rx.Subscriber;
import rx.android.subscriptions.AndroidSubscriptions;
import rx.functions.Action0;
import android.content.BroadcastReceiver;
import android.content.Context;
import android.content.Intent;
watermark = 1 hour
First batch (max event time = null):
2017-06-07 10:00:00.000
StateStore will store 2017-06-07 10:00:00.000
Second batch (max event time = 2017-06-07 10:00:00.000):
2017-06-07 11:00:00.000
@zsxwing
zsxwing / gist:5924191
Last active April 3, 2017 04:45
Pig and Hive source codes overview. Small hint: use "explain" to find the operation keywords, and search them in the source codes to find the corresponding class.
Pig:
org.apache.pig.newplan.logical.relational.LogicalPlan
org.apache.pig.backend.hadoop.executionengine.physicalLayer.plans.PhysicalPlan
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.plans.MROperPlan
org.apache.pig.parser.QueryParserDriver.parse(String)
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(LogicalPlan,Properties)
org.apache.pig.PigServer.launchPlan(PhysicalPlan,String)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(PhysicalPlan,PigContext)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(MROperPlan,MapReduceOper,Configuration,PigContext)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(MROperPlan,String)
Observable.just(1)
.doOnSubscribe(() -> System.out.println("before 1st doOnSubscribe: " + Thread.currentThread().getName()))
.subscribeOn(Schedulers.newThread())
.doOnSubscribe(() -> System.out.println("before 2nd doOnSubscribe: " + Thread.currentThread().getName()))
.subscribeOn(Schedulers.io())
.doOnSubscribe(() -> System.out.println("before 3rd doOnSubscribe: " + Thread.currentThread().getName()))
.subscribeOn(Schedulers.computation())
.doOnSubscribe(() -> System.out.println("before subscribe: " + Thread.currentThread().getName()))
.subscribe(new Subscriber<Integer>() {
@Override
@zsxwing
zsxwing / StreamingApp.scala
Last active February 29, 2016 20:51
StreamingApp.scala
package streaming.app
import java.util.UUID
import scala.util.Random
import scala.util.control.NonFatal
import org.apache.commons.io.IOUtils
import org.apache.hadoop.fs.{Path, FileSystem}
import org.apache.spark.sql.SQLContext
@zsxwing
zsxwing / async.scala
Last active January 2, 2016 13:39
Asnyc exmaple
import rx.lang.scala.Observable._
import rx.lang.scala.JavaConversions._
import rx.lang.scala.ImplicitFunctionConversions._
import rx.util.async.Async
import java.util.concurrent.Executors
import rx.schedulers.Schedulers
object Test extends App {
val executor = Executors.newFixedThreadPool(10)
@zsxwing
zsxwing / build.gradle
Created September 7, 2013 09:40
A template of build.gradle for scala
apply plugin: 'scala'
apply plugin: 'eclipse'
sourceCompatibility = JavaVersion.VERSION_1_6
targetCompatibility = JavaVersion.VERSION_1_6
eclipse {
classpath {
downloadSources = true
downloadJavadoc = false
@zsxwing
zsxwing / exit_when_error.sh
Created September 2, 2013 11:08
We can add -e at the end of "#!/bin/bash" or use "set -e" to let the script exit when some error happens
#!/bin/bash -e
set -e