This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MacBook-Pro-van-Fokko:java fokkodriesprong$ docker run -t -i python:3.6 pip install apache-airflow[gcp_api] | |
Unable to find image 'python:3.6' locally | |
3.6: Pulling from library/python | |
5ae19949497e: Pull complete | |
ed3d96a2798e: Pull complete | |
f12136850781: Pull complete | |
1a9ad5d5550b: Pull complete | |
6f18049a0455: Pull complete | |
8daf83b35d32: Pull complete | |
c099d5e9ac68: Pull complete |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/* | |
* Licensed to the Apache Software Foundation (ASF) under one | |
* or more contributor license agreements. See the NOTICE file | |
* distributed with this work for additional information | |
* regarding copyright ownership. The ASF licenses this file | |
* to you under the Apache License, Version 2.0 (the | |
* "License"); you may not use this file except in compliance | |
* with the License. You may obtain a copy of the License at | |
* | |
* http://www.apache.org/licenses/LICENSE-2.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
C02VF05JHV2T:flink-gcs-fs fdriesprong$ spark-shell | |
2019-04-17 08:38:30 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
Setting default log level to "WARN". | |
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). | |
Spark context Web UI available at http://c02vf05jhv2t.localdomain:4040 | |
Spark context available as 'sc' (master = local[*], app id = local-1555483115980). | |
Spark session available as 'spark'. | |
Welcome to | |
____ __ | |
/ __/__ ___ _____/ /__ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from airflow import DAG | |
from datetime import datetime | |
from bigquery_get_data import BigQueryGetDataOperator | |
dag = DAG( | |
dag_id='godatafest', | |
schedule_interval='@daily', | |
default_args={ | |
'owner': 'GoDataDriven', |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# -*- coding: utf-8 -*- | |
# | |
# Licensed to the Apache Software Foundation (ASF) under one | |
# or more contributor license agreements. See the NOTICE file | |
# distributed with this work for additional information | |
# regarding copyright ownership. The ASF licenses this file | |
# to you under the Apache License, Version 2.0 (the | |
# "License"); you may not use this file except in compliance | |
# with the License. You may obtain a copy of the License at | |
# |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# -*- coding: utf-8 -*- | |
# | |
# Licensed to the Apache Software Foundation (ASF) under one | |
# or more contributor license agreements. See the NOTICE file | |
# distributed with this work for additional information | |
# regarding copyright ownership. The ASF licenses this file | |
# to you under the Apache License, Version 2.0 (the | |
# "License"); you may not use this file except in compliance | |
# with the License. You may obtain a copy of the License at | |
# |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
------------------------------------------------------- | |
T E S T S | |
------------------------------------------------------- | |
Running org.apache.avro.tool.TestJsonToFromBinaryFragmentTools | |
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.099 sec - in org.apache.avro.tool.TestJsonToFromBinaryFragmentTools | |
Running org.apache.avro.tool.TestTetherTool | |
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.272 sec - in org.apache.avro.tool.TestTetherTool | |
Running org.apache.avro.tool.TestConcatTool | |
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.597 sec - in org.apache.avro.tool.TestConcatTool | |
Running org.apache.avro.tool.TestCreateRandomFileTool |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import requests as rq | |
import argparse | |
import random | |
from pyspark.sql import SparkSession | |
spark = SparkSession.builder.appName('Set Vincent\'s cluster on fire').getOrCreate() | |
def f(_): | |
current_max = {'score': 0.0}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.godatadriven | |
object Day3 extends App { | |
import scala.annotation.tailrec | |
case class Pos(x: Int, y: Int) { | |
def +(other: Pos): Pos = Pos(x + other.x, y + other.y) | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[INFO] Scanning for projects... | |
[WARNING] The project io.druid:druid-benchmarks:jar:0.11.1-SNAPSHOT uses prerequisites which is only intended for maven-plugin projects but not for non maven-plugin projects. For such purposes you should use the maven-enforcer-plugin. See https://maven.apache.org/enforcer/enforcer-rules/requireMavenVersion.html | |
[INFO] ------------------------------------------------------------------------ | |
[INFO] Reactor Build Order: | |
[INFO] | |
[INFO] io.druid:druid | |
[INFO] java-util | |
[INFO] druid-api | |
[INFO] druid-common | |
[INFO] druid-hll |