gaborgsomogyi

from itertools import product

def generate_paintings():
    colors = ['A', 'B', 'C', 'D']
    hexagons = ['H1', 'H2', 'H3', 'H4', 'H5', 'H6', 'H7']  # 1 in the middle, 6 around
    
    # Generate all possible combinations of colors for the hexagons
    all_paintings = product(colors, repeat=7)
    

gaborgsomogyi

Dependency management in python

Python is not able to handle 2 different versions from the same package.

Dependency handling issue

When Pip <20.3 finds an unresolvable dependency (resolves a dependecy to 2+ versions) then would not halt the installation process. It’d successfully continue the process by installing the first matching dependency in the list of conflicts.

gaborgsomogyi

Prerequisites:

• Install any container system like docker/Virtualbox/hyperkit
• Install minikube
• mkdir -p ${HOME}/share
• chmod 777 ${HOME}/share
• minikube start --mount-string="${HOME}/share:/share" --mount
• kubectl create clusterrolebinding flink-role-binding-default --clusterrole=edit --serviceaccount=default:default
• kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml

Clone the necessary repos:

gaborgsomogyi

    @SuppressWarnings("unchecked")
    public static void setEnv(String key, String value) {
        try {
            Map<String, String> env = System.getenv();
            Class<?> cl = env.getClass();
            Field field = cl.getDeclaredField("m");
            field.setAccessible(true);
            Map<String, String> writableEnv = (Map<String, String>) field.get(env);
 writableEnv.put(key, value);

gaborgsomogyi

Flink version 1.16.0

Note1: types are java compatible
Note2: all strings are in UTF format

• int magic: 0x4960672d (0x60 0x49 0x2d 0x67)
• int version: 1|2|3
• long checkpointId
• int numMasterStates
  numMasterStates times

gaborgsomogyi

Prerequisites:

• Deploy Flink k8s operator
• Install cmctl

Double check that the appropriate certificate is created

$ kubectl get cert       
NAME                          READY   SECRET                AGE
flink-operator-serving-cert   True    webhook-server-cert   91m

gaborgsomogyi

Versions

Flink version: 1.16-SNAPSHOT

Mock dependencies

Mockito version: 2.21.0
Powermock version: 2.0.9

• Both uses Bytebuddy.
• Powermock uses Mockito

gaborgsomogyi

• 0.10.0.1 => 2.0.0 (https://issues.apache.org/jira/browse/SPARK-18057):
  • Java7 support dropped
  • Test only: server.boundPort added ListenerName parameter
  • Test only: server.replicaManager.getPartition instead of (tp.topic, tp.partition) only TopicPartition is the parameter
  • Test only: server.getLogManager().logDirs moved to server.getLogManager().liveLogDirs
• 2.0.0 => 2.1.0 (https://issues.apache.org/jira/browse/SPARK-25954):
  • No breaking
  • Java11 support added
• 2.1.0 => 2.1.1 (https://issues.apache.org/jira/browse/SPARK-26916):
• No breaking

gaborgsomogyi

Modify krb5.conf

$ cat /etc/krb5.conf
[libdefaults]
	udp_preference_limit=1
	default_realm = VPC.CLOUDERA.COM
	forwardable = true

[realms]
	VPC.CLOUDERA.COM = {

gaborgsomogyi

When the following job has started then the job fails because in submit time we don't know on which node the job will run and the security.kerberos.spnego.principal=HTTP/csa-sec-1.vpc.cloudera.com@VPC.CLOUDERA.COM is host specific.

flink run \
-yD security.kerberos.login.keytab=/cdep/keytabs/systest.keytab \
-yD security.kerberos.login.principal=systest@VPC.CLOUDERA.COM \
-yD security.spnego.auth.enabled=true \
-yt /var/run/cloudera-scm-agent/process/1546340184-flink-FLINK_HISTORY_SERVER/flink.keytab \
-yD security.kerberos.spnego.keytab=flink.keytab \