Fokko

MacBook-Pro-van-Fokko:java fokkodriesprong$ docker run -t -i python:3.6 pip install apache-airflow[gcp_api]
Unable to find image 'python:3.6' locally
3.6: Pulling from library/python
5ae19949497e: Pull complete 
ed3d96a2798e: Pull complete 
f12136850781: Pull complete 
1a9ad5d5550b: Pull complete 
6f18049a0455: Pull complete 
8daf83b35d32: Pull complete 
c099d5e9ac68: Pull complete 

Fokko

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0

Fokko

C02VF05JHV2T:flink-gcs-fs fdriesprong$ spark-shell
2019-04-17 08:38:30 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://c02vf05jhv2t.localdomain:4040
Spark context available as 'sc' (master = local[*], app id = local-1555483115980).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__

Fokko

from airflow import DAG
from datetime import datetime

from bigquery_get_data import BigQueryGetDataOperator

dag = DAG(
    dag_id='godatafest',
    schedule_interval='@daily',
    default_args={
        'owner': 'GoDataDriven',

Fokko

# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#

Fokko

# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#

Fokko

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.avro.tool.TestJsonToFromBinaryFragmentTools
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.099 sec - in org.apache.avro.tool.TestJsonToFromBinaryFragmentTools
Running org.apache.avro.tool.TestTetherTool
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.272 sec - in org.apache.avro.tool.TestTetherTool
Running org.apache.avro.tool.TestConcatTool
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.597 sec - in org.apache.avro.tool.TestConcatTool
Running org.apache.avro.tool.TestCreateRandomFileTool

Fokko

import requests as rq
import argparse
import random

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('Set Vincent\'s cluster on fire').getOrCreate()

def f(_):
    current_max = {'score': 0.0};

Fokko

package com.godatadriven

object Day3 extends App {

  import scala.annotation.tailrec

  case class Pos(x: Int, y: Int) {
    def +(other: Pos): Pos = Pos(x + other.x, y + other.y)
  }

Fokko

[INFO] Scanning for projects...
[WARNING] The project io.druid:druid-benchmarks:jar:0.11.1-SNAPSHOT uses prerequisites which is only intended for maven-plugin projects but not for non maven-plugin projects. For such purposes you should use the maven-enforcer-plugin. See https://maven.apache.org/enforcer/enforcer-rules/requireMavenVersion.html
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] io.druid:druid
[INFO] java-util
[INFO] druid-api
[INFO] druid-common
[INFO] druid-hll