Skip to content

Instantly share code, notes, and snippets.

View iht's full-sized avatar
💭
Migrating the world to the cloud

Israel Herraiz iht

💭
Migrating the world to the cloud
View GitHub Profile
@iht
iht / impala_traffic.ipynb
Last active August 29, 2015 13:56
IMPALA Traffic departing from selected airport
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@iht
iht / Goats_and_cars.ipynb
Created July 5, 2015 10:05
Goats and cars - probability of winning using different strategies
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

ODSC Europe 2017 (London)

Algorithmic Trading

With Machine & Deep Learning

Workshop by Dr. Yves J. Hilpisch | The Python Quants GmbH

London, 12. October 2017

Keybase proof

I hereby claim:

  • I am iht on github.
  • I am iht (https://keybase.io/iht) on keybase.
  • I have a public key ASCRsaMgH_rLlw8kPafLYxwu0jd4x0xrAv7IKj7UF0CHYQo

To claim this, I am signing this object:

case class MyDemoType(
name: String,
version: String,
someProperty: Int
)
@iht
iht / read_kafka.scala
Created January 27, 2020 21:51
Read all unacknowledged messages in a Kafka topic with KafkaIO in Apache Beam
KafkaIO.read[String, String]
.withBootstrapServers(config.broker)
.withTopic(config.kafkaTopic)
.withKeyDeserializer(classOf[StringDeserializer])
.withValueDeserializer(classOf[StringDeserializer])
.withStartReadTime(Instant.ofEpochMilli(1))
.withoutMetadata
@iht
iht / deserialize.scala
Last active January 27, 2020 21:52
Sample deserialization function for the Kafka2Avro pipelines
/** Returns an object of type T from a base64 encoded string.
*
* Overwrite this function for your deserialization code. In this example, we
* assume that Kafka contains plain old Java objects, encoded as base64
* strings.
*
* @tparam T The type of the object to be recovered
* @param s String with a base64 encoded serialized object
*/
def string2Object[T](s: String): T = {
@iht
iht / serialize.scala
Last active January 27, 2020 21:53
Sample serialization function for the Kafka2Avro pipelines
/** Returns a string representation of an object.
*
* Overwrite this function with your serialization code. In this example, we
* serialize the object and transform that into a base64 encoded string,
* which will be later on written to Kafka.
*
* @tparam T The type of the object ot be serialized. The type must be Serializable
* @param obj The object to be serialized
*/
def object2String[T](obj: T): String = {
@iht
iht / read_kafka.scala
Last active January 27, 2020 22:04
Read all unacknowledged messages in a Kafka topic with KafkaIO in Apache Beam
KafkaIO.read[String, String]
.withBootstrapServers(config.broker)
.withTopic(config.kafkaTopic)
.withKeyDeserializer(classOf[StringDeserializer])
.withValueDeserializer(classOf[StringDeserializer])
.withStartReadTime(Instant.ofEpochMilli(1))
.withoutMetadata
@iht
iht / business_rules.py
Created May 9, 2020 19:02
Example of business rules calculation in a streaming pipeline with Apache Beam
"""Business rules applied to each group in each window."""""
import apache_beam as beam
from datetime import datetime
from dateutil import parser as dateparser
class BusinessRulesDoFn(beam.DoFn):
"""This DoFn applies some business rules to a group of messages in a window.