Skip to content

Instantly share code, notes, and snippets.

Avatar

Reynold Xin rxin

View GitHub Profile
@rxin
rxin / gist:6be132f46b72c27d8f89
Created Nov 1, 2014
test.scala on constructor parameter shadowing
View gist:6be132f46b72c27d8f89
class LegalPerson(name: String) {
def aaaaaaaaaa = name
}
class DoomedPerson(name: String) extends LegalPerson(name) {
def curName = name
}
@rxin
rxin / dstat
Created Sep 15, 2014
High sys usage with Transparent Huge Pages (THP) enabled
View dstat
date/time | used buff cach free|usr sys idl wai hiq siq| read writ| recv send
15-09 04:55:05|52.6G 56.8M 176G 11.4G| 4 2 82 12 0 0| 527M 0 | 0 198B
15-09 04:55:06|52.6G 56.8M 176G 10.9G| 3 2 80 15 0 0| 542M 64k| 581B 36k
15-09 04:55:07|52.6G 56.8M 177G 10.4G| 3 1 82 13 0 0| 535M 0 | 0 0
15-09 04:55:08|52.6G 56.8M 177G 9.87G| 2 1 85 12 0 0| 520M 0 | 0 506B
15-09 04:55:09|52.6G 56.8M 178G 9558M| 2 1 84 12 0 0| 549M 0 | 260B 520B
15-09 04:55:10|52.6G 56.8M 179G 9009M| 3 1 82 14 0 0| 557M 0 | 104B 594B
15-09 04:55:11|52.6G 56.8M 179G 8463M| 3 2 83 13 0 0| 530M 72k| 104B 272B
15-09 04:55:12|52.6G 56.8M 180G 7940M| 3 2 81 15 0 0| 532M 0 | 200B 888B
15-09 04:55:13|52.6G 56.8M 180G 7417M| 3 2 82 12 0 0| 510M 0 | 0 198B
@rxin
rxin / gist:37554d9f3e2e93220884
Last active Aug 29, 2015
Scala @inline final val impact on accessors
View gist:37554d9f3e2e93220884
scala> class A {
| private[this] val abc = 1
|
| def print() = println(abc)
| }
# abc gets inlined by the compiler as a field
[[syntax trees at end of cleanup]] // <console>
package $line4 {
@rxin
rxin / snappy-framed-input-1k.dump
Created Jul 15, 2014
snappy framed vs non framed jmap
View snappy-framed-input-1k.dump
num #instances #bytes class name
----------------------------------------------
1: 6443 80317016 [B
2: 15102 1939472 <methodKlass>
3: 15102 1749784 <constMethodKlass>
4: 1029 1161976 <constantPoolKlass>
5: 1029 959320 <instanceKlassKlass>
6: 891 697312 <constantPoolCacheKlass>
7: 3680 245496 [C
@rxin
rxin / lzf-1k.dump
Created Jul 14, 2014
Compression codec buffer allocation
View lzf-1k.dump
Every 1000 com.ning.compress.lzf.LZFOutputStream instances allocate 198976424 bytes.
Every 1000 org.xerial.snappy.SnappyOutputStream instances allocate 67660104 bytes.
View microbenchmark.markdown

I was curious about the results reported here, which reports that Scala's mutable maps are slower than Java's: http://www.infoq.com/news/2011/11/yammer-scala

In my tests, Scala's OpenHashMap equals or beats java's HashMap:

Insertion 100k elements (String keys) time in ms:

  • scala HashMap: 92.75
  • scala OpenHashMap: 14.03125
  • java HashMap: 15.78125
View gist:8910734
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
@rxin
rxin / gist:6896688
Last active Dec 25, 2015
take async
View gist:6896688
def takeAsync(num: Int): FutureAction[Seq[T]] = {
val promise = new CancellablePromise[Seq[T]]
promise.run {
val buf = new ArrayBuffer[T](num)
val totalParts = self.partitions.length
var partsScanned = 0
while (buf.size < num && partsScanned < totalParts && !promise.cancelled) {
// The number of partitions to try in this iteration. It is ok for this number to be
@rxin
rxin / update.sh
Last active Dec 21, 2015
Update Spark/Shark on EC2 AMI
View update.sh
set -e
set -o pipefail
/root/spark/bin/stop-all.sh
rm -rf ~/.ivy2/local/org.spark*
rm -rf ~/.ivy2/cache/org.spark*
cd /root/spark
git checkout master
@rxin
rxin / InsertPerf.scala
Last active Dec 19, 2015
Scala collection insert performance (2.9.3)
View InsertPerf.scala
// 1001 381 384 384 383 384 407 404 409 407
object ArrayBufferBenchmark extends scala.testing.Benchmark {
def run = {
val len = 10 * 1000 * 1000
val a = new scala.collection.mutable.ArrayBuffer[Int](len)