Skip to content

Instantly share code, notes, and snippets.

@romiras
romiras / !resque_async_drainer.rb
Last active May 28, 2022
Asynchronously drain Resque queue with concurrent workers in Ruby
View !resque_async_drainer.rb
require_relative './logging'
require_relative './async_consumer'
def drain(data_store, queue_name, output_filename)
json_store = File.open(output_filename, 'a')
executor = lambda do |class_name, args|
Logging.logger.info "class_name: #{class_name}, args: #{args}"
json_store.write("#{ { class: class_name, args: args }.to_json }\n")
@romiras
romiras / async_queue_consumer.rb
Created May 27, 2022
Demo async queue consumer with concurrent N workers in Ruby
View async_queue_consumer.rb
require 'concurrent'
require 'benchmark'
require 'logger'
module Logging # Credits: https://stackoverflow.com/a/6768164/10118318
# This is the magical bit that gets mixed into your classes
def logger
Logging.logger
end
@romiras
romiras / concurrent_workers_batch_test.rb
Last active Jan 25, 2022
Concurrently classify items by type and batch them with multiple workers
View concurrent_workers_batch_test.rb
# Concurrently classify items by type and batch them with multiple workers
require 'logger'
NUM_WORKERS = 10
NUM_ITEMS = 12
NUM_ITERATIONS = 5000
Item = Struct.new(:id, :type)
@romiras
romiras / es_bulk_upload.sh
Last active Dec 8, 2021
Script for uploading 30 K dataset into ES
View es_bulk_upload.sh
split --verbose -l1000 dataset-bulk-30k.ndjson bulk.
for f in bulk.??; do echo $f; curl -i -X POST localhost:9200/_bulk -H "Content-Type: application/x-ndjson" --data-binary @$f; done
@romiras
romiras / es_ndjson_split.rb
Last active Nov 11, 2021
A script for transforming .jsonl file into file ready for bulk import to ES.
View es_ndjson_split.rb
# Reads .jsonl file and transforms it into file ready for bulk import to ES.
require 'json'
File.open(ARGV[1], 'wb') {|f|
ARGF.each_line { |line|
a = JSON.parse(line)
out = {
index: {
"_id": a["_id"],
@romiras
romiras / concurrent_workers_demo.rb
Last active Nov 4, 2021
Workers with Concurrent::Future in Ruby
View concurrent_workers_demo.rb
require 'concurrent'
require 'benchmark'
max = 20 # number of tasks to process
n_workers = 4 # number of concurrent workers
results = [] #
lambda = -> (iter, i) { results << ('%04d' % i); d = rand(0.005)+0.001; puts("Iteration #{iter}. sleep %.3f" % d); sleep(d) }
workers = Array.new(n_workers, lambda)
iter = 0
@romiras
romiras / Gemfile
Created May 8, 2019
EventMachine async URL fetcher
View Gemfile
source "https://rubygems.org"
gem 'eventmachine'
gem 'em-http-request'
@romiras
romiras / dictionary-gnu-sort-bd.txt
Last active Jan 1, 2020 — forked from klauspost/dictionary-sorted.txt
Brotli dictionary - printed escaped - sorted with "sort -bd" (with dictionary order, ignoring blanks), a tool from GNU coreutils
View dictionary-gnu-sort-bd.txt
"<!--"
"><!--"
"||[];"
"--><!--"
"--></"
"----"
"!--<"
"//--></"
"//-->"
"...</"
@romiras
romiras / Readme.md
Last active Jun 9, 2020
EBK backup extraction scripts
View Readme.md

How to use

Assumed you have files with extension .ebk stored by Kies.

Prerequisites

Install Ruby 2.3 or later

Example for running in Bash terminal. Linux is not mandatory for running Ruby program.

@romiras
romiras / rezip.py
Created Jan 14, 2017
ReZip - tool for recompression Zip files, used to efficiently store in SCM
View rezip.py
#!/usr/bin/env python3
"""Read zip format file from stdin and write new zip to stdout.
With the --store option the output will be an uncompressed zip.
Uncompressed files are stored more efficiently in VCS.
Example:
python rezip.py --store < file.docx > file.flat.docx
Based on https://github.com/costerwi/rezip