Skip to content

Instantly share code, notes, and snippets.

View gaborgsomogyi's full-sized avatar

Gabor Somogyi gaborgsomogyi

27 followers · 10 following
View GitHub Profile
@gaborgsomogyi
gaborgsomogyi / Log exception chain.md
Created December 3, 2020 14:13 
    @tailrec
    def logExceptionChain(e: Throwable): Unit = {
      logError("Exception in chain: %s(%s)".format(e.getClass.getName, e.getMessage))
      if (e.getCause != null) {
        logExceptionChain(e.getCause)
      }
    }
    logExceptionChain(e)
@gaborgsomogyi
gaborgsomogyi / AdminClient used Spark API compatibility.md
Last active October 20, 2020 12:43

AdminClient

As a general rule, the AdminClient protocol is backward compatible (protocol negotiation happens in the background). The listed APIs used by Spark are working with metadata API and the latest AdminClient is compatible with it (please see the bullet points for details).

  • listTopics:
  • describeTopics:
    These APIs are working with metadata API, which is an integral part of Kafka almost from the beginning. The functionality has been added in Kafka 0.8.1 in KAFKA-1227, 17263. Since it exists there were no breaking changes introduced.
  • listOffsets:
    This API is working with metadata API, which is an integral part of Kafka almost from the beginning. The functionality has been added in Kafka 0.10.1 in KAFKA-4148, 1852 and [1897](https://github.com/apache/kaf
@gaborgsomogyi
gaborgsomogyi / Map usb into virtualbox.md
Last active October 18, 2020 09:12 
cd "c:\Program Files\Oracle\VirtualBox"
VBoxManage internalcommands createrawvmdk -filename "%USERPROFILE%"\.VirtualBox\usb1.vmdk -rawdisk \\.\PhysicalDrive1
@gaborgsomogyi
gaborgsomogyi / Spark Schema Registry support.md
Created October 14, 2020 11:44
  • CDP Private Cloud Base 7.1.5
  • Cloudera Runtime Update 7.2.3.0
@gaborgsomogyi
gaborgsomogyi / KafkaConsumer vs AdminClient needed ACLs.md
Last active September 29, 2020 14:28

KafkaConsumer

  • assign, subscribe, subscribePattern:
    • On topic resourse describe operation
  • seek, seekToBeginning, seekToEnd
    • On topic resourse describe operation
  • poll
    • On group resourse read operation
    • On topic resourse read operation
  • assignment: None
  • pause: None
@gaborgsomogyi
gaborgsomogyi / Spark standalone streaming.md
Last active September 23, 2020 10:52

Commands

Spark console

git co https://github.com/gaborgsomogyi/spark.git
cd spark
sbuild -d
sbin/start-master.sh
sbin/start-worker.sh spark://gsomogyi-MBP16:7077

Check that http://localhost:8080/ is open and the worker is available.

@gaborgsomogyi
gaborgsomogyi / Kafka auto topic creation.md
Last active September 2, 2020 09:28

Kafka auto topic creation is an interesting and somewhat really dangerous feature. I would like to write down my findings related this area.

Feature description

Broker and Consumer side KIP
Producer side KIP
AdminClient side: Not supported in 2.6

Behavior and danger

The feature is simple when a Producer or Consumer has an assignment with a broker and detects that a topic is deleted

@gaborgsomogyi
gaborgsomogyi / YARN AttemptId
Created May 8, 2020 12:10
Format:
appattempt_{clusterTimestamp}_{appId}_{attemptId}
e.g.: appattempt_1410901177871_0001_01.
attemptId starts from 1.
@gaborgsomogyi
gaborgsomogyi / DB2 useful commands.md
Last active March 23, 2020 17:28

Set environment variables

USERPROFILE=/database/config/db2inst1/sqllib/userprofile
echo "export DB2_KRB5_PRINCIPAL=db2/example.com@EXAMPLE.COM" >> $USERPROFILE
echo "export KRB5_KTNAME=/var/custom/db2.keytab" >> $USERPROFILE

Pass evironment variables (variables starting with DB2 passed automatically)

su - db2inst1 -c "db2set DB2ENVLIST=KRB5_KTNAME"

@gaborgsomogyi
gaborgsomogyi / Install SQPlus to apt based system.md
Last active March 9, 2020 10:04
  1. Download .rpm package here
    • curl -O https://download.oracle.com/otn_software/linux/instantclient/193000/oracle-instantclient19.3-basic-19.3.0.0.0-1.x86_64.rpm
    • curl -O https://download.oracle.com/otn_software/linux/instantclient/193000/oracle-instantclient19.3-devel-19.3.0.0.0-1.x86_64.rpm
    • curl -O https://download.oracle.com/otn_software/linux/instantclient/193000/oracle-instantclient19.3-sqlplus-19.3.0.0.0-1.x86_64.rpm
  2. apt-get -y install alien libaio1
  3. Convert the rpm files and install them
    • alien -i oracle-instantclient*-basic-*.rpm
    • alien -i oracle-instantclient*-devel-*.rpm
    • alien -i oracle-instantclient*-sqlplus-*.rpm
  4. ldconfig