gaborgsomogyi

from itertools import product

def generate_paintings():
    colors = ['A', 'B', 'C', 'D']
    hexagons = ['H1', 'H2', 'H3', 'H4', 'H5', 'H6', 'H7']  # 1 in the middle, 6 around
    
    # Generate all possible combinations of colors for the hexagons
    all_paintings = product(colors, repeat=7)
    

gaborgsomogyi

Dependency management in python

Python is not able to handle 2 different versions from the same package.

Dependency handling issue

When Pip <20.3 finds an unresolvable dependency (resolves a dependecy to 2+ versions) then would not halt the installation process. It’d successfully continue the process by installing the first matching dependency in the list of conflicts.

gaborgsomogyi

    @SuppressWarnings("unchecked")
    public static void setEnv(String key, String value) {
        try {
            Map<String, String> env = System.getenv();
            Class<?> cl = env.getClass();
            Field field = cl.getDeclaredField("m");
            field.setAccessible(true);
            Map<String, String> writableEnv = (Map<String, String>) field.get(env);
 writableEnv.put(key, value);

gaborgsomogyi

Flink version 1.16.0

Note1: types are java compatible
Note2: all strings are in UTF format

• int magic: 0x4960672d (0x60 0x49 0x2d 0x67)
• int version: 1|2|3
• long checkpointId
• int numMasterStates
  numMasterStates times

gaborgsomogyi

Prerequisites:

• Deploy Flink k8s operator
• Install cmctl

Double check that the appropriate certificate is created

$ kubectl get cert       
NAME                          READY   SECRET                AGE
flink-operator-serving-cert   True    webhook-server-cert   91m

gaborgsomogyi

Versions

Flink version: 1.16-SNAPSHOT

Mock dependencies

Mockito version: 2.21.0
Powermock version: 2.0.9

• Both uses Bytebuddy.
• Powermock uses Mockito

gaborgsomogyi

Prerequisites:

• Install any container system like docker/Virtualbox/hyperkit
• Install minikube
• mkdir -p ${HOME}/share
• chmod 777 ${HOME}/share
• minikube start --mount-string="${HOME}/share:/share" --mount
• kubectl create clusterrolebinding flink-role-binding-default --clusterrole=edit --serviceaccount=default:default
• kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml

Clone the necessary repos:

gaborgsomogyi

When the following job has started then the job fails because in submit time we don't know on which node the job will run and the security.kerberos.spnego.principal=HTTP/csa-sec-1.vpc.cloudera.com@VPC.CLOUDERA.COM is host specific.

flink run \
-yD security.kerberos.login.keytab=/cdep/keytabs/systest.keytab \
-yD security.kerberos.login.principal=systest@VPC.CLOUDERA.COM \
-yD security.spnego.auth.enabled=true \
-yt /var/run/cloudera-scm-agent/process/1546340184-flink-FLINK_HISTORY_SERVER/flink.keytab \
-yD security.kerberos.spnego.keytab=flink.keytab \

gaborgsomogyi

sun.net.www.protocol.http.NegotiateAuthentication caches per host (and no port information in the key) whether negotiation is supported or not (for instance SPNEGO). Worth to mention the cache is static (god knows why). This is so far so good but can lead to issues in unit tests. For example:

• Developer writes a positive and negative test case for an SPNEGO protected server
• Normally the order of test cases are not defined
• Negative test case executes where SPNEGO turned off and expect 401
• NegotiateAuthentication stores false in a map
• Negative test case pass
• Positive test case executes where SPNEGO turned on and expect 200
• NegotiateAuthentication gives base isSupported=false from cache

gaborgsomogyi

Modify krb5.conf

$ cat /etc/krb5.conf
[libdefaults]
	udp_preference_limit=1
	default_realm = VPC.CLOUDERA.COM
	forwardable = true

[realms]
	VPC.CLOUDERA.COM = {