Skip to content

Instantly share code, notes, and snippets.

View abitofhelp's full-sized avatar

Michael Gardner abitofhelp

View GitHub Profile
@abitofhelp
abitofhelp / rxgenerator.kt
Created July 1, 2019 09:13
Reactive streaming from Azure Blob to rxjava generator/parser. This snippet shows how to stream content from Azure Blob Storage using their Java SDK v11 and reactive streams. The blob's stream is fed into a rxjava/rxkotlin generator, which parses sample data from the blob stream. Subscribers use the Flowable to work with each sample parsed from …
/**
* Method generator parses sample data from a json stream.
* @param sampleJsonUrl String is the URL to the Azure Blob Storage sample data.
* @return Flowable<Sample> is a cold, synchronous, stateful and backpressure-aware
* generator of features.
*/
fun generator(sampleJsonUrl: String) =
blobStorage.downloadBlob(sampleJsonUrl)
.map { bbuf: ByteBuffer -> JsonFactory().createParser(bbuf.array()) }
.map { jParser ->
@abitofhelp
abitofhelp / Dockerfile-DeployAspNet2IISWithPsWebDeploy
Created December 19, 2018 07:39
Dockerfile to deploy an AspNet app to IIS in a container using PSWebDeploy
# The general framwork of my script is based on fragments from all over the Internet, including
# the following website. Since there is no clear owner of this information and it was contributed
# by many people, I am assuming that an MIT license is appropriate.
# https://fluentbytes.com/deploying-asp-net-4-5-to-docker-on-windows/
# docker run -p 8088:80 --restart always --name aspnetdocker -dit abitofhelp/aspnetdocker
# Now we can browse to the website. Be aware that you can only reach the container from the outside,
# so if you would browse to localhost, which results in the #127.0.0.0 you will not see any results.
# You need to address your machine on its actual hostname or outside IP address.
@abitofhelp
abitofhelp / go-stream-file-between-goroutines-with-pipe.go
Created September 15, 2018 03:04
This gist implements a streaming process where a chunk of data is read from a file in a goroutine, the data is passed through a channel to another goroutine, which writes the data to a Linux pipe.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main implements streaming of a file through a FIFO pipe, from which data is written to a new file.
// Using a FIFO pipe avoids having to allocate buffers within the for loop, which eliminates a data race condition
// on reading data from the file and sending it to the channel. However, using a pipe is slower than a
// channel. A quick test with a 3.9GB binary file required 5.3s with a channel, and 12.6s with a pipe.
// I've created a companion gist,"go-stream-file-between-goroutines-with-channel" for comparison.
@abitofhelp
abitofhelp / cs-job-scheduler.cs
Created August 14, 2018 22:14
This gist implements a job scheduler that makes it easier to have long running jobs go to sleep and wake up on a schedule.
using System;
using System.Diagnostics;
using System.Globalization;
using System.Threading;
using NLog;
using NodaTime;
using NodaTime.Text;
// Copyright 2009 The Noda Time Authors. All rights reserved.
// Use of this source code is governed by the Apache License 2.0,
@abitofhelp
abitofhelp / cs-fast-file-scanner.cs
Last active August 14, 2018 22:02
This gist is for a very high-performance file system scanner that supports international characters and paths longer than .NET's limitation.
/*
* Copyright (c) 2008-2015 Peter Palotas, Jeffrey Jangli, Alexandr Normuradov
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software (Alphaleonis) and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
@abitofhelp
abitofhelp / cs-stringextensions.cs
Last active August 14, 2018 22:01
This gist implements methods that extend the abilities of string objects.
////////////////////////////////////////////////////////////////////////////////////////////////////
// file: Extensions\StringExtensions.cs
//
// summary: Implements the string extensions class
////////////////////////////////////////////////////////////////////////////////////////////////////
using System;
using System.Linq;
using System.Security.Cryptography;
@abitofhelp
abitofhelp / go-refcntbuffer.go
Last active September 15, 2018 01:15
This gist implements a streaming process where a chunk of data is read from a file in a goroutine, the data is passed through a channel to another goroutine, which writes the data to a file. It uses a reference counting pool of buffers.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package refcnt implements a reference count based []byte buffer.
package refcnt
import (
"errors"
@abitofhelp
abitofhelp / go-stream-file-between-goroutines-with-channel
Last active July 31, 2018 01:16
This gist implements a streaming process where a chunk of data is read from a file in a goroutine, the data is passed through a channel to another goroutine, which writes the data to a file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main implements streaming of a file through a buffered channel where it is written to a new file.
// A data race condition arose in the for loop that reads data from the input file and sending it to the channel.
// I moved the allocation of the buffer to the top of the loop, to resolve the data race issue.
// I've created a companion gist,"go-stream-file-between-goroutines-with-pipe" that resolves the race issue using a
// FIFO pipe. It worked, but the price was that it was slower than using a channel.
@abitofhelp
abitofhelp / go-directed-acyclic-graph-etl.go
Last active July 28, 2018 20:53
This gist implements a directed acyclic graph to implement an extract, transform, and load ("ETL") process for files.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main implements an ETL process where a file is read, and its contents streamed to a fan-out consisting
// of two, parallel goroutines: One that calculates the MD5 checksum and passes it along to the fan-in stage; The
// other saves the content stream to a new file and passes the file's metadata to the fan-in stage. The fan-in stage
// is the sink for the ETL process. It merges the metadata from the fan-out operations into a JSON object, which is
// persisted to a local file.
@abitofhelp
abitofhelp / go-many-to-many-buffered-channel.go
Last active July 28, 2018 02:21
This gist shows the use of a buffered channel with multiple senders and receivers.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main is the entry point for the application
// and is responsible for configuring the environment.
package main
import (