brew install minikube
brew install hyperkit
minikube config set driver hyperkit
minikube start --memory 8192 --cpus 5 --disk-size=80GB
brew install docker
# point docker to the minikube env
eval $(minikube -p minikube docker-env)
# validate setup
Robert Metzger rmetzger
rmetzger
/ gist:e556bfda8082bceeae6a32e7e0208bb6
Last active
January 16, 2022 11:36
Free Docker on Mac Alternative using Hyperkit from minikube
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package de.robertmetzger; | |
| import org.apache.flink.api.common.serialization.Encoder; | |
| import org.apache.flink.api.java.tuple.Tuple2; | |
| import org.apache.flink.core.fs.Path; | |
| import org.apache.flink.core.io.SimpleVersionedSerializer; | |
| import org.apache.flink.streaming.api.datastream.DataStream; | |
| import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; | |
| import org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner; | |
| import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FROM flink:1.12.2 | |
| RUN wget https://mirror.netcologne.de/apache.org/hadoop/common/hadoop-3.1.4/hadoop-3.1.4.tar.gz ; tar xf hadoop-3.1.4.tar.gz | |
| RUN cp -r /opt/flink/hadoop-3.1.4/share/hadoop/common/* /opt/flink/hadoop-3.1.4/share/hadoop/hdfs/* /opt/flink/lib/ | |
| RUN mkdir -p /opt/hadoop/conf ; cp -r hadoop-3.1.4/etc/hadoop/* /opt/hadoop/conf/ | |
| RUN rm /opt/flink/lib/lib/slf4j-log4j12-1.7.25.jar ; rm /opt/flink/lib/lib/commons-cli-1.2.jar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| =============== INITIAL REPORT =============================== | |
| 2020-05-14 10:12:42,660 INFO org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer [] - Started Kinesis producer instance for region 'eu-central-1' | |
| 2020-05-14 10:12:42,660 DEBUG org.apache.flink.streaming.api.operators.BackendRestorerProcedure [] - Creating operator state backend for StreamSource_cbc357ccb763df2852fee8c4fc7d55f2_(1/1) with empty state. | |
| 2020-05-14 10:12:42,823 INFO org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer [] - Closing producer | |
| 2020-05-14 10:12:42,823 INFO org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer [] - Flushing outstanding 2 records | |
| 2020-05-14 10:12:42,826 ERROR org.apache.flink.streaming.runtime.tasks.StreamTask [] - Error during disposal of stream operator. | |
| org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.producer.DaemonException: The child process has been shutdown and can no longer accept messages. | |
| at org.apache.flink.kinesis.shaded.com.a |
rmetzger
/ gist:ca5a0f30ebc4d1852180570889e07242
Created
February 12, 2020 12:10
Flink 1.11 AskTimeoutException
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| This excerpt is taken from https://dev.azure.com/rmetzger/Flink/_build/results?buildId=4940&view=results | |
| 2020-02-07T11:38:27.0968542Z [FAIL] Test script contains errors. | |
| 2020-02-07T11:38:27.0980571Z Checking for errors... | |
| 2020-02-07T11:38:27.1067102Z Found error in log files: | |
| 2020-02-07T11:38:27.1081555Z 2020-02-07 11:27:50,918 INFO org.apache.flink.client.cli.CliFrontend - -------------------------------------------------------------------------------- | |
| 2020-02-07T11:38:27.1082318Z 2020-02-07 11:27:50,919 INFO org.apache.flink.client.cli.CliFrontend - Starting Command Line Client (Version: 1.11-SNAPSHOT, Rev:e682fc5, Date:07.02.2020 @ 10:01:55 UTC) | |
| 2020-02-07T11:38:27.1082804Z 2020-02-07 11:27:50,919 INFO org.apache.flink.client.cli.CliFrontend - OS current user: agent1_azpcontainer | |
| 2020-02-07T11:38:27.1083188Z 2020-02-07 11:27:50,920 INFO org.apache.flink.client.cli.CliFrontend - Current Hadoop/Kerberos user: <no |
rmetzger
/ gist:cb4b84358d535fbcbfce4f48a0b547de
Created
August 18, 2019 07:12
Flink builds times / 2 cores / 7.5gb: 03:32 h
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [INFO] Reactor Summary: | |
| [INFO] | |
| [INFO] force-shading ...................................... SUCCESS [ 4.501 s] | |
| [INFO] flink .............................................. SUCCESS [ 36.734 s] | |
| [INFO] flink-annotations .................................. SUCCESS [ 3.541 s] | |
| [INFO] flink-shaded-curator ............................... SUCCESS [ 2.003 s] | |
| [INFO] flink-test-utils-parent ............................ SUCCESS [ 0.214 s] | |
| [INFO] flink-test-utils-junit ............................. SUCCESS [ 3.727 s] | |
| [INFO] flink-metrics ...................................... SUCCESS [ 0.188 s] | |
| [INFO] flink-metrics-core ................................. SUCCESS [ 3.481 s] |
rmetzger
/ gist:ffe7b972b721b1033b9c5fe19b42f1a7
Created
August 17, 2019 18:45
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [INFO] Reactor Summary: | |
| [INFO] | |
| [INFO] force-shading ...................................... SUCCESS [ 0.830 s] | |
| [INFO] flink .............................................. SUCCESS [ 4.011 s] | |
| [INFO] flink-annotations .................................. SUCCESS [ 0.860 s] | |
| [INFO] flink-shaded-curator ............................... SUCCESS [ 0.928 s] | |
| [INFO] flink-test-utils-parent ............................ SUCCESS [ 0.072 s] | |
| [INFO] flink-test-utils-junit ............................. SUCCESS [ 1.381 s] | |
| [INFO] flink-metrics ...................................... SUCCESS [ 0.083 s] | |
| [INFO] flink-metrics-core ................................. SUCCESS [ 1.735 s] |
rmetzger
/ gist:6e080eff53726d084814a2becb853b4f
Created
August 17, 2019 16:41
Flink builds times / 32 cores / 64gb: 01:25 h
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [INFO] Reactor Summary: | |
| [INFO] | |
| [INFO] force-shading ...................................... SUCCESS [ 3.847 s] | |
| [INFO] flink .............................................. SUCCESS [ 37.611 s] | |
| [INFO] flink-annotations .................................. SUCCESS [ 2.338 s] | |
| [INFO] flink-shaded-curator ............................... SUCCESS [ 1.090 s] | |
| [INFO] flink-test-utils-parent ............................ SUCCESS [ 0.066 s] | |
| [INFO] flink-test-utils-junit ............................. SUCCESS [ 2.259 s] | |
| [INFO] flink-metrics ...................................... SUCCESS [ 0.105 s] | |
| [INFO] flink-metrics-core ................................. SUCCESS [ 2.486 s] |
rmetzger
/ gist:939662235c797bc790a59a92ed2ff9f1
Created
April 26, 2019 14:28
Flink frontend-maven-plugin
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [INFO] Installing node version v10.9.0 | |
| [INFO] Downloading https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.gz to /root/.m2/repository/com/github/eirslett/node/10.9.0/node-10.9.0-linux-x64.tar.gz | |
| [INFO] No proxies configured | |
| [INFO] No proxy was configured, downloading directly | |
| [INFO] Unpacking /root/.m2/repository/com/github/eirslett/node/10.9.0/node-10.9.0-linux-x64.tar.gz into /flink/flink-runtime-web/web-dashboard/node/tmp | |
| [INFO] Copying node binary from /flink/flink-runtime-web/web-dashboard/node/tmp/node-v10.9.0-linux-x64/bin/node to /flink/flink-runtime-web/web-dashboard/node/node | |
| [INFO] Extracting NPM | |
| [INFO] Installed node locally. | |
| [INFO] | |
| [INFO] --- frontend-maven-plugin:1.6:npm (npm install) @ flink-runtime-web_2.11 --- |
rmetzger
/ gist:7c123a6d94aa31ad19cc8f9b57b893a4
Created
April 24, 2019 15:25
Flink UI deps for d1e5eae6c71d256a40592f0082f63577a120192a
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @angular/animations 7.2.10 : MIT License | |
| @angular/cdk 7.3.6 : MIT License | |
| @angular/common 7.2.10 : MIT License | |
| @angular/compiler 7.2.10 : MIT License | |
| @angular/core 7.2.10 : MIT License | |
| @angular/forms 7.2.10 : MIT License | |
| @angular/platform-browser 7.2.10 : MIT License | |
| @angular/platform-browser-dynamic 7.2.10 : MIT License | |
| @angular/router 7.2.10 : MIT License | |
| @ant-design/icons-angular 2.0.2 : MIT License |
NewerOlder