AnanthaRajuC

/*
    Welcome to your first dbt model!
    Did you know that you can also configure models directly within SQL files?
    This will override configurations stated in dbt_project.yml

    Try changing "table" to "view" below
*/

{{ config(materialized='table') }}

AnanthaRajuC

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#

AnanthaRajuC

#!/usr/bin/env python
# coding: utf8

# Script to create superset users

import pandas as pd
from sqlalchemy import create_engine
import datetime
import pexpect
import os

AnanthaRajuC

==========================
How Software Companies Die
==========================
 - Orson Scott Card

The environment that nurtures creative programmers kills management and
marketing types - and vice versa.

Programming is the Great Game. It consumes you, body and soul. When
you're caught up in it, nothing else matters. When you emerge into

AnanthaRajuC

/*
JPA (Java Persistence API)

Transaction Management with an Entity-Mananger:
---
entityManager.getTransaction().begin();
entityManager.persist(<some-entity>);
entityManager.getTransaction().commit();
entityManager.clear();

AnanthaRajuC

#Multi Stage Docker Build

#STAGE 1 - Build the layered jar
#Use Maven base image 
FROM maven-3.8.3-openjdk-17 AS builder
COPY src /home/app/src
COPY pom.xml /home/app
#Build an uber jar
RUN mvn -f /home/app/pom.xml package
WORKDIR /home/app/target

AnanthaRajuC

# A Docker Compose must always start with the version tag.
# We use '3' because it's the last version.
version: "3.7"
# You should know that Docker Compose works with services.
# 1 service = 1 container.
# For example, a service, a server, a client, a database...
# We use the keyword 'services' to start to create services.
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest

AnanthaRajuC

/* APPENDIX 1-B: CLEANUP
			This makes this sql query re-run able */
            
DROP TABLE IF EXISTS JSON_TABLE;
DROP TABLE IF EXISTS SPLIT_TABLE;
DROP VIEW IF EXISTS SPLIT_VIEW;

/* APPENDIX 1-B: Prepare TABLE
			Let's say this is an example table */

AnanthaRajuC

Delete message after consuming it in KAFKA

In Kafka, the responsibility of what has been consumed is the responsibility of the consumer and this is also one of the main reasons why Kafka has such great horizontal scalability.

Using the high level consumer API will automatically do this for you by committing consumed offsets in Zookeeper (or a more recent configuration option is using by a special Kafka topic to keep track of consumed messages).

The simple consumer API make you deal with how and where to keep track of consumed messages yourself.

Purging of messages in Kafka is done automatically by either specifying a retention time for a topic or by defining a disk quota for it so for your case of one 5GB file, this file will be deleted after the retention period you define has passed, regardless of if it has been consumed or not.

AnanthaRajuC

docker run --tty --network postgres_debezium_cdc_default confluentinc/cp-kafkacat kafkacat -b kafka:9092 -C -s key=s -s value=avro -r http://schema-registry:8081 -t postgres.public.student