mlehman

/* Example using MultipleOutputs to write a Spark RDD to multiples files.
   Based on saveAsNewAPIHadoopFile implemented in org.apache.spark.rdd.PairRDDFunctions, org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil.

  val values = sc.parallelize(List(
    ("fruit/items", "apple"),
    ("vegetable/items", "broccoli"),
    ("fruit/items", "pear"),
    ("fruit/items", "peach"),
    ("vegetable/items", "celery"),
    ("vegetable/items", "spinach")

MLnick

package example

import org.apache.avro.Schema.Parser
import java.io.{DataInput, DataOutput, File}
import org.apache.avro.generic.GenericData.Record
import org.apache.avro.generic.{GenericRecord, GenericDatumWriter}
import org.apache.avro.file.DataFileWriter
import org.apache.spark.SparkContext
import org.apache.avro.mapreduce.AvroKeyInputFormat
import org.apache.avro.mapred.AvroKey

armon

Consul Consistency

As Kyle brought up, Consul at the moment has a single known case of a potential inconsistency (Could be unknown cases lurking). Currently Consul works by electing a leader, who "leases" the position for LeaderLeaseTimeout interval. At each interval, it checks that a quorum of nodes still believes it to be the leader. At the same time, if a follower does not hear from the leader within randomInterva(HeartbeatTimeout, 2 * HeartbeatTimeout), it will start a new election.

0xabad1dea

Abadidea's Index of Weird Machines in Video Games

A "weird machine" is when user-supplied input is able to create an arbitrary new program running within an existing program due to Turing-completeness being exposed. Sometimes such functionality was deliberately included but it is often the result of exploitation of memory corruption. You can learn more at the langsec site. There is a good argument for weird machines being inherently dangerous, but this index is just for fun.

It is broken into two categories: intentional gameplay features which may be used as weird machines, and exploit-based machines which can be triggered by ordinary player input (tool-assisted for speed and precision is acceptable). Games with the sole purpose of programming (such as Core Wars) are not eligible and plugin APIs don't count. If you know of more, feel free to add a comment to this gist.

Intentional Gameplay Mechanics

banksean

// Ported from Stefan Gustavson's java implementation
// http://staffwww.itn.liu.se/~stegu/simplexnoise/simplexnoise.pdf
// Read Stefan's excellent paper for details on how this code works.
//
// Sean McCullough banksean@gmail.com

/**
 * You can pass in a random number generator object if you like.
 * It is assumed to have a random() method.
 */