Skip to content

Instantly share code, notes, and snippets.

View sadikovi's full-sized avatar

Ivan sadikovi

86 followers · 22 following
  • New Zealand
View GitHub Profile
@sadikovi
sadikovi / db.scala
Last active August 5, 2021 15:24
Run SQL queries against a JDBC source in the notebook (for quick debugging, copy-paste the code, set url and props, and run queries)
object DB {
import org.apache.spark.sql._
import org.apache.spark.sql.types._
var url = "jdbc:sqlserver://..."
var props = new java.util.Properties()
var autoCommit = true
var spark = SparkSession.getActiveSession.get
def execute(conn: java.sql.Connection, query: String): DataFrame = {
@sadikovi
sadikovi / Tetris.java
Last active March 28, 2021 15:50
Tetris in JavaFX
import java.util.Random;
import java.util.concurrent.ConcurrentHashMap;
import javafx.animation.AnimationTimer;
import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.Group;
import javafx.scene.canvas.Canvas;
import javafx.scene.canvas.GraphicsContext;
import javafx.scene.paint.Color;
@sadikovi
sadikovi / google_chrome_proxy_start.sh
Last active February 20, 2021 07:29
Start Google Chrome with OWASP ZAP proxy
open -a "Google Chrome" --args --proxy-server=http://localhost:8080 --ignore-certificate-errors
@sadikovi
sadikovi / parquet_read.scala
Created November 4, 2018 09:46
Parquet MR read file and list all of the records
////////////////////////////////////////////////////////////////
// == Parquet read ==
////////////////////////////////////////////////////////////////
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.mapreduce._
import org.apache.hadoop.mapreduce.lib.input.FileSplit
import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
import org.apache.parquet.hadoop.ParquetInputSplit
import org.apache.parquet.hadoop.ParquetRecordReader
@sadikovi
sadikovi / CustomEncoder.scala
Created September 21, 2018 12:10
Code to create a custom encoder for a class with different fields, including Row
def clazz[T](cls: Class[T], encoders: Seq[(String, ExpressionEncoder[_])]): ExpressionEncoder[T] = {
encoders.foreach { case (_, enc) => enc.assertUnresolved() }
val schema = StructType(encoders.map {
case (fieldName, e) =>
val (dataType, nullable) = if (e.flat) {
e.schema.head.dataType -> e.schema.head.nullable
} else {
e.schema -> true
}
@sadikovi
sadikovi / Example.java
Last active September 14, 2018 08:23
Issue #158 example
final class Example {
void /* test */ func() {
String a = "a";
String b = "a" + b + "c()";
Buffer buf = "test" + "new Buffer() {};";
HashSet<String> test = new HashSet<String>();
}
public int get_int() {
@sadikovi
sadikovi / DefaultSource.scala
Last active June 18, 2018 12:53
Example of StreamSinkProvider for structured streaming with custom query execution
package org.apache.spark.sql.sadikovi
import java.io.{ObjectInputStream, ObjectOutputStream}
import java.util.UUID
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.mapreduce.{JobContext, TaskAttemptContext}
import org.apache.spark.internal.io._
@sadikovi
sadikovi / module.patch
Created May 31, 2018 07:05
Java 9 module grammar for language-java
diff --git a/grammars/java.cson b/grammars/java.cson
index cb9947a..399c914 100644
--- a/grammars/java.cson
+++ b/grammars/java.cson
@@ -109,6 +109,9 @@
{
'include': '#code'
}
+ {
+ 'include': '#module'
@sadikovi
sadikovi / stats.rs
Created April 26, 2018 08:36
Mutable Statistics Buffer (for collecting statistics during writes). For PR https://github.com/sunchao/parquet-rs/pull/94
// ----------------------------------------------------------------------
// Statistics updates
struct MutableStatisticsBuffer<T: DataType> {
typed: TypedStatistics<T>,
sort_order: SortOrder
}
impl<T: DataType> MutableStatisticsBuffer<T> {
pub fn new(column_order: ColumnOrder, is_min_max_deprecated: bool) -> Self {
@sadikovi
sadikovi / utc_date.rs
Last active March 10, 2024 07:47
Convert timestamp in seconds into datetime in UTC as Rust function
use std::fmt;
#[derive(Clone, Debug)]
pub struct DateTime {
/// Seconds after the minute - [0, 59]
pub sec: i32,
/// Minutes after the hour - [0, 59]
pub min: i32,
/// Hours after midnight - [0, 23]
pub hour: i32,