Skip to content

Instantly share code, notes, and snippets.

View rxin's full-sized avatar

Reynold Xin rxin

View GitHub Profile
package spark.util
import java.io.{ByteArrayInputStream, ByteArrayOutputStream}
import scala.collection.mutable
import org.objectweb.asm.{ClassReader, MethodVisitor}
import org.objectweb.asm.commons.EmptyVisitor
import org.objectweb.asm.Opcodes._
def testWrite(path: String): Long = {
val startTime = System.currentTimeMillis()
val out = new java.io.FileWriter(path)
var i = 1
val bytes = " " * (1024 * 1024)
while (i < 1000) {
out.write(bytes)
i += 1
}
out.close
@rxin
rxin / ByteBufferPerf.scala
Last active May 14, 2018 21:27
Comparison of performance over various approaches to read Java ByteBuffer. The best way is to use Unsafe, which also enables reading multiple primitive data types from the same buffer.
/**
* To compile:
* scalac -optimize ByteBufferPerf.scala
*
* JAVA_OPTS="-Xmx2g" scala IntArrayPerf 10
* 49 62 48 45 48 45 48 50 47 45
*
* JAVA_OPTS="-Xmx2g" scala ByteBufferPerf 10
* 479 491 484 480 484 481 477 477 472 473
@rxin
rxin / ramdisk.sh
Last active December 13, 2023 02:40
ramdisk create/delete on Mac OS X.
#!/bin/bash
# From http://tech.serbinn.net/2010/shell-script-to-create-ramdisk-on-mac-os-x/
#
ARGS=2
E_BADARGS=99
if [ $# -ne $ARGS ] # correct number of arguments to the script;
then
@rxin
rxin / ampcamp-ecnu-2013-data.sh
Last active December 14, 2015 10:49
scripts to help setup ampcamp @ ECNU March 2013
################################################################################
# Step 1. Download wiki traffic log.
# from
# https://s3.amazonaws.com/ampcamp/ampcamp-ecnu-2013/wikistats/part-00095.gz
# to
# https://s3.amazonaws.com/ampcamp/ampcamp-ecnu-2013/wikistats/part-00168.gz
# Note that 095 and 168 are both 0 bytes. The sole purpose of their existence is
# to verify the downloads.
# NOTE THAT THE FOLLOWING SCRIPT STARTS wget AS BACKGROUND PROCESSES.