Skip to content

Instantly share code, notes, and snippets.

@dinvlad
dinvlad / PubSubServiceExcerpt.java
Last active September 18, 2018 14:11
Workaround for listening on PubSub subscriptions even when StreamingPull stops silently
// ... other dependencies ...
import com.google.api.core.ApiService;
import com.google.common.util.concurrent.MoreExecutors;
import com.google.cloud.pubsub.v1.MessageReceiver;
import com.google.cloud.pubsub.v1.Subscriber;
import com.google.pubsub.v1.Subscription;
import org.springframework.stereotype.Service;
@dinvlad
dinvlad / gist:83c078481649a35e16d9112d2743dab9
Last active July 28, 2019 07:40 — forked from mackstann/gist:4229933
Gigantic recursive directory entry counter
// http://www.olark.com/spw/2011/08/you-can-list-a-directory-with-8-million-files-but-not-with-ls/
#define _GNU_SOURCE
#include <dirent.h> /* Defines DT_* constants */
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/syscall.h>
@dinvlad
dinvlad / terraform-init.sh
Last active May 10, 2020 01:29
Script to configure encrypted GCS backend and GCP project-specific .tfvars for Terraform
#!/usr/bin/env bash
set -euo pipefail
# define GCP project and GCS bucket backend for Terraform state
GCP_PROJECT="$1"
BACKEND_BUCKET="${2:-${GCP_PROJECT}-terraform}"
# create versioned bucket if it doesn't exist, skip otherwise
if gsutil mb -p "${GCP_PROJECT}" "gs://${BACKEND_BUCKET}" ; then
@dinvlad
dinvlad / app.py
Created July 17, 2020 18:30
Validate interactive request from Slack
import hashlib
import hmac
import os
from time import time
from flask import Request, abort
SLACK_SIGNING_SECRET = os.environ['SLACK_SIGNING_SECRET']
@dinvlad
dinvlad / cloudrun.jinja
Created September 6, 2019 15:51
Create Cloud Run Service through Deployment Manager
{% set PROJECT = env['project'] %}
{% set REGION = properties['region'] | default('us-east1') %}
{% set PARENT = 'projects/' + PROJECT + '/locations/' + REGION %}
{% set PREFIX = env['deployment'] %}
{% set CLOUDRUN_TYPE_PROVIDER = PREFIX + '-provider' %}
{% set CLOUDRUN_SERVICE_TYPE = PROJECT + '/' + CLOUDRUN_TYPE_PROVIDER + ':projects.locations.services' %}
@dinvlad
dinvlad / filesystem_vs_git_local.txt
Last active August 10, 2022 15:52
Truffelhog not detecting AWS secrets, with or without -max-depth 1 option
$ trufflehog filesystem --directory . --trace
DEBU[0000] running version dev
DEBU[0000] running version dev
DEBU[0000] running with up to 32 workers
DEBU[0000] loaded 2 decoders
DEBU[0000] loaded 694 detectors total, 694 with verification enabled. 0 with verification disabled
🐷🔑🐷 TruffleHog. Unearth your secrets. 🐷🔑🐷
Found unverified result 🐷🔑❓
Detector Type: AWS
@dinvlad
dinvlad / gist:7d2f505c56bffb9e743c2ce8f4b2a746
Created August 22, 2022 21:43
Trufflehog hangs indefinitely on scanning a Jupyter notebook
$ trufflehog git https://github.com/dinvlad/pyro-cov.git \
--branch bkotzen-CEPI --max-depth 2 \
--since-commit 9cd031577750f88648a0cee5f8d6ed781f717c6b --trace
DEBU[0000] running version dev
DEBU[0000] running version dev
DEBU[0000] running with up to 1 workers
DEBU[0000] loaded 2 decoders
DEBU[0000] loaded 694 detectors total, 694 with verification enabled. 0 with verification disabled
DEBU[0000] Cloning remote Git repo without authentication
@dinvlad
dinvlad / get-aws-token
Created October 25, 2016 23:32
Get session token for virtual MFA-enabled actions through AWS CLI
#!/bin/bash
# Thanks to MattJ at:
# http://www.brassmill.net/2015/10/using-the-aws-cli-with-roles-security-token-service-and-mfa/
#
# User must have 'iam:GetUser' permission on themselves that doesn't require MFA
[ -z $1 ] && echo "Please enter your MFA code" && exit 1
user=$(aws iam get-user \
@dinvlad
dinvlad / trace.txt
Created June 28, 2023 19:54
Trufflehog update redirect error
$ trufflehog filesystem . --debug --trace
2023/06/28 15:53:38 [updater parent] run
2023/06/28 15:53:38 [updater parent] checking for updates...
2023/06/28 15:53:39 [updater parent] failed to get latest version: Post "https://oss.trufflehog.org/updates": stopped after 10 redirects
2023/06/28 15:53:39 [updater parent] starting /usr/local/bin/trufflehog
2023/06/28 15:53:40 [updater child#1] run
2023/06/28 15:53:40 [updater child#1] start program
2023-06-28T15:53:40-04:00 info-2 trufflehog trufflehog 3.40.0
2023-06-28T15:53:40-04:00 info-2 trufflehog engine started {"workers": 16}
2023-06-28T15:53:40-04:00 info-0 trufflehog loaded decoders {"count": 3}
@dinvlad
dinvlad / retries.ts
Last active February 2, 2024 16:18
Retries with exponential backoff and jitter for idempotent background Google Cloud Functions
// one can also use Google Cloud Firestore library,
// with a slight change in semantics
import { firestore } from 'firebase-admin';
import { EventContext, runWith } from 'firebase-functions';
import { promisify } from 'util';
const eventCollection = 'function-events';
enum EventStatus {
RUNNING = 'running',