Skip to content

Instantly share code, notes, and snippets.

View abitofhelp's full-sized avatar

Michael Gardner abitofhelp

3 followers · 0 following
View GitHub Profile
@abitofhelp
abitofhelp / go-read-file-chunks.go
Last active July 28, 2018 05:42
This gist shows how to read from a file using chunks and calculating the MD5 hash on-the-fly.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main is the entry point for the application
// and is responsible for configuring the environment.
package main
import (
@abitofhelp
abitofhelp / go-producer-consumer.go
Last active July 26, 2018 19:51
This gist shows how to implement a producer-consumer pattern in go.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main is the entry point for the application
// and is responsible for configuring the environment.
package main
import (
@abitofhelp
abitofhelp / go-one-to-many-buffered-channel.go
Last active July 28, 2018 01:33
This gist shows the use of a buffered channel with a single sender and multiple receivers.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main is the entry point for the application
// and is responsible for configuring the environment.
package main
import (
@abitofhelp
abitofhelp / go-many-to-many-buffered-channel.go
Last active July 28, 2018 02:21
This gist shows the use of a buffered channel with multiple senders and receivers.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main is the entry point for the application
// and is responsible for configuring the environment.
package main
import (
@abitofhelp
abitofhelp / go-directed-acyclic-graph-etl.go
Last active July 28, 2018 20:53
This gist implements a directed acyclic graph to implement an extract, transform, and load ("ETL") process for files.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main implements an ETL process where a file is read, and its contents streamed to a fan-out consisting
// of two, parallel goroutines: One that calculates the MD5 checksum and passes it along to the fan-in stage; The
// other saves the content stream to a new file and passes the file's metadata to the fan-in stage. The fan-in stage
// is the sink for the ETL process. It merges the metadata from the fan-out operations into a JSON object, which is
// persisted to a local file.
@abitofhelp
abitofhelp / go-stream-file-between-goroutines-with-channel
Last active July 31, 2018 01:16
This gist implements a streaming process where a chunk of data is read from a file in a goroutine, the data is passed through a channel to another goroutine, which writes the data to a file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a MIT license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package main implements streaming of a file through a buffered channel where it is written to a new file.
// A data race condition arose in the for loop that reads data from the input file and sending it to the channel.
// I moved the allocation of the buffer to the top of the loop, to resolve the data race issue.
// I've created a companion gist,"go-stream-file-between-goroutines-with-pipe" that resolves the race issue using a
// FIFO pipe. It worked, but the price was that it was slower than using a channel.
@abitofhelp
abitofhelp / go-refcntbuffer.go
Last active September 15, 2018 01:15
This gist implements a streaming process where a chunk of data is read from a file in a goroutine, the data is passed through a channel to another goroutine, which writes the data to a file. It uses a reference counting pool of buffers.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Copyright (c) 2018 A Bit of Help, Inc. - All Rights Reserved, Worldwide.
// Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Package refcnt implements a reference count based []byte buffer.
package refcnt
import (
"errors"
@abitofhelp
abitofhelp / cs-stringextensions.cs
Last active August 14, 2018 22:01
This gist implements methods that extend the abilities of string objects.
////////////////////////////////////////////////////////////////////////////////////////////////////
// file: Extensions\StringExtensions.cs
//
// summary: Implements the string extensions class
////////////////////////////////////////////////////////////////////////////////////////////////////
using System;
using System.Linq;
using System.Security.Cryptography;
@abitofhelp
abitofhelp / cs-fast-file-scanner.cs
Last active August 14, 2018 22:02
This gist is for a very high-performance file system scanner that supports international characters and paths longer than .NET's limitation.
/*
* Copyright (c) 2008-2015 Peter Palotas, Jeffrey Jangli, Alexandr Normuradov
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software (Alphaleonis) and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
@abitofhelp
abitofhelp / cs-job-scheduler.cs
Created August 14, 2018 22:14
This gist implements a job scheduler that makes it easier to have long running jobs go to sleep and wake up on a schedule.
using System;
using System.Diagnostics;
using System.Globalization;
using System.Threading;
using NLog;
using NodaTime;
using NodaTime.Text;
// Copyright 2009 The Noda Time Authors. All rights reserved.
// Use of this source code is governed by the Apache License 2.0,