Skip to content

Instantly share code, notes, and snippets.

View atais's full-sized avatar

Michał Siatkowski atais

View GitHub Profile
@dusenberrymw
dusenberrymw / spark_tips_and_tricks.md
Last active February 8, 2023 05:11
Tips and tricks for Apache Spark.

Spark Tips & Tricks

Misc. Tips & Tricks

  • If values are integers in [0, 255], Parquet will automatically compress to use 1 byte unsigned integers, thus decreasing the size of saved DataFrame by a factor of 8.
  • Partition DataFrames to have evenly-distributed, ~128MB partition sizes (empirical finding). Always err on the higher side w.r.t. number of partitions.
  • Pay particular attention to the number of partitions when using flatMap, especially if the following operation will result in high memory usage. The flatMap op usually results in a DataFrame with a [much] larger number of rows, yet the number of partitions will remain the same. Thus, if a subsequent op causes a large expansion of memory usage (i.e. converting a DataFrame of indices to a DataFrame of large Vectors), the memory usage per partition may become too high. In this case, it is beneficial to repartition the output of flatMap to a number of partitions that will safely allow for appropriate partition memory sizes, based upon the
@AutomationD
AutomationD / Logstash.xml
Last active June 18, 2020 20:39
Logstash IntelliJ Idea Filetype
<filetype binary="false" description="Logstash Config" name="Logstash Config">
<highlighting>
<options>
<option name="LINE_COMMENT" value="#" />
<option name="COMMENT_START" value="" />
<option name="COMMENT_END" value="" />
<option name="HEX_PREFIX" value="" />
<option name="NUM_POSTFIXES" value="" />
<option name="HAS_BRACES" value="true" />
<option name="HAS_BRACKETS" value="true" />
@jaibeee
jaibeee / brew-perms.sh
Last active February 15, 2024 22:49
Configure homebrew permissions to allow multiple users on MAC OSX. Any user from the admin group will be able to manage the homebrew and cask installation on the machine.
#!/bin/sh
# Configure homebrew permissions to allow multiple users on MAC OSX.
# Any user from the admin group will be able to manage the homebrew and cask installation on the machine.
# allow admins to manage homebrew's local install directory
chgrp -R admin /usr/local
chmod -R g+w /usr/local
# allow admins to homebrew's local cache of formulae and source files
chgrp -R admin /Library/Caches/Homebrew
@cvrebert
cvrebert / css_regression_testing.md
Last active May 9, 2023 12:13
Survey of screenshot-based CSS testing tools

Currently considering https://github.com/webdriverio/webdrivercss


Core Goals:

  • Can test in up-to-date versions of all major browsers
  • Can test on up-to-date versions of all major OSes
  • Can test in IE9 (because Bootstrap v4 will support IE9+)
  • Don't want to have to setup/maintain our own cluster of VMs running all the necessary OSes (and all the versions of Windows)
  • Workflow for management of reference/baseline/norm screenshots
(function () {
var dateTimeController = function ($scope, $rootScope) {
$scope.vm = {
message: "Bootstrap DateTimePicker Directive",
dateTime: {}
};
$scope.$watch('change', function(){
console.log($scope.vm.dateTime);
});
@RadoBuransky
RadoBuransky / dist-play-app-initd
Last active March 24, 2020 20:26
Init.d shell script for Play framework distributed application. Provides start, stop, restart and status commands to control applications packaged using standard "play dist" packaging command.
#!/bin/bash
#
# =========================================================================
# Copyright 2014 Rado Buransky, Dominion Marine Media
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
@kevin-smets
kevin-smets / iterm2-solarized.md
Last active April 18, 2024 13:34
iTerm2 + Oh My Zsh + Solarized color scheme + Source Code Pro Powerline + Font Awesome + [Powerlevel10k] - (macOS)

Default

Default

Powerlevel10k

Powerlevel10k

#!/usr/bin/ruby
# Create display override file to force Mac OS X to use RGB mode for Display
# see http://embdev.net/topic/284710
require 'base64'
data=`ioreg -l -d0 -w 0 -r -c AppleDisplay`
edids=data.scan(/IODisplayEDID.*?<([a-z0-9]+)>/i).flatten
vendorids=data.scan(/DisplayVendorID.*?([0-9]+)/i).flatten
@miglen
miglen / Apache Tomcat 8 Start stop script init.d script
Last active November 10, 2022 20:03 — forked from valotas/tomcat.sh
Apache Tomcat init script (or startup/controll script). Works fine for version 7/8. Read the comments for release history. Feel free to modify, copy and give suggestions. (c) GNU General Public License
#!/bin/bash
#
# description: Apache Tomcat init script
# processname: tomcat
# chkconfig: 234 20 80
#
#
# Copyright (C) 2014 Miglen Evlogiev
#
# This program is free software: you can redistribute it and/or modify it under