Skip to content

Instantly share code, notes, and snippets.

View camallen's full-sized avatar

Campbell Allen camallen

21 followers · 4 following
View GitHub Profile
@camallen
camallen / ebmp_known_bird_taxa.json
Last active July 29, 2022 09:34
Allowed birds in ebmp system
[
{ "common_name": "blue tit", "genus": "cyanistes", "species": "caeruleus", "bto_species_code": "bluti" },
{ "common_name": "great tit", "genus": "parus", "species": "major", "bto_species_code": "greti" },
{ "common_name": "coal tit", "genus": "periparus", "species": "ater", "bto_species_code": "coati" },
{ "common_name": "marsh tit", "genus": "poecile", "species": "palustris", "bto_species_code": "marti" },
{ "common_name": "nuthatch", "genus": "sitta", "species": "europaeus", "bto_species_code": "nutha" },
{ "common_name": "great spotted woodpecker", "genus": "dendrocopos", "species": "major", "bto_species_code": "grswo" },
{ "common_name": "goldcrest", "genus": "regulus", "species": "regulus", "bto_species_code": "goldc" },
{ "common_name": "goldfinch", "genus": "carduelis", "species": "carduelis", "bto_species_code": "goldf" },
{ "common_name": "blackbird", "genus": "turdus", "species": "merula", "bto_species_code": "blabi" },
@camallen
camallen / Dockerfile
Last active May 25, 2021 16:50
rails app multi stage docker builds
# Start from a small, trusted base image with the version pinned down
FROM ruby:2.5-slim-stretch AS base
# Install system dependencies required both at runtime and build time
# The image uses Postgres but you can swap it with mariadb-dev (for MySQL) or sqlite-dev
RUN apt-get update && \
apt-get install --no-install-recommends -y \
build-essential \
# git is required for installing gems from git repos
git \
@camallen
camallen / sidekiq_jobs_cleanup.rb
Created January 29, 2021 13:55
cleanup failed and retryable sidekiq jobs
# NOTE: sidekiq 6+ introduces new `.scan` method on Sets, use this when the version matches
# https://github.com/mperham/sidekiq/wiki/API#scan
# e.g. rs = Sidekiq::RetrySet.new, rs.scan('string-to-look-for')
# -----------------------------------------------------------------------
# remove any old jobs that won't finish and are in the DEAD set / Morgue
rs = Sidekiq::RetrySet.new
jobs_to_remove_from_retry = rs.select do |job|
# 'workflow_attached_image'
@camallen
camallen / search_index_update_subject_mime_types_gz_storage.py
Last active October 10, 2020 13:44
Update blob content types - use a search index to generate a list of az storage cli cmds
from azure.core.credentials import AzureKeyCredential
from azure.search.documents import SearchClient
from urllib.parse import urlparse
from progress.bar import Bar
from sys import platform
import re
import subprocess
import mimetypes
import sys
import os
@camallen
camallen / unretire_workflow_subjects.rb
Created September 25, 2020 13:07
Unretire subects for a workflow
workflow_ids = []
subject_set_ids = []
workflow_ids.map do |wf_id|
subject_set_ids.map do |set_id|
set_subject_ids = SetMemberSubject.where(subject_set_id: set_id).pluck(:subject_id)
# UNCOMMMENT ONE OF THESE, I DO A SANITY CHECK COUNT FIRST, THEN UNRETIRE
# run a count for sanity check
# SubjectWorkflowStatus.where.not(retired_at: nil).where(workflow_id: wf_id, subject_id: set_subject_ids).count
# RUN THE UNRETIRE
SubjectWorkflowStatus.where.not(retired_at: nil).where(workflow_id: wf_id, subject_id: set_subject_ids).update_all(retired_at: nil, retirement_reason: nil)
@camallen
camallen / unlock_and_reload_designator_workflow.sh
Last active April 16, 2020 16:18
Unlock and reload a designator workflow
creds="$basicUsername:$basicPassword"
host="https://designator.zooniverse.org/api/workflows"
if [ -n "$1" ]; then
curl -v --user $creds -X POST "${host}/${1}/unlock"
curl -v --user $creds -X POST "${host}/${1}/reload"
fi
@camallen
camallen / zooniverse-public-stream-debug.html
Created March 31, 2020 14:39
<!DOCTYPE html>
<head>
<meta charset="utf-8"/>
<title>Zooniverse Public Stream Debug</title>
<script src="https://js.pusher.com/5.1/pusher.min.js"></script>
</head>
<body>
<h1>Zooniverse Public Stream Debug</h1>
<p>
See the console log statements for the json event payloads.
@camallen
camallen / code_used_to_break_and_fix_workflow_and_translation_strings_data.rb
Created October 22, 2019 09:56
Broken survey task labels on AmazonCam Tambopata project (invalid translation strings data)
# 1. The underlying cause of the issue
# ~17:00-17:30 on Friday 18th October, 2019
# the commands I used to work on the translation fix issue
# fix the workflow strings for an older survey task (missing descriptions)
# https://www.zooniverse.org/lab/3040/workflows/2485
workflow = Workflow.find 2485
tasks = workflow.tasks
task_string_extractor = TasksVisitors::ExtractStrings.new
@camallen
camallen / one_task_workflows_with_single_questions.sql
Created September 20, 2019 13:01
Query to find all the 1 task workflows with single question only
select id, tasks --(kv.value -> 'type')
from
workflows w,
jsonb_each(w.tasks) kv
WHERE kv.value -> 'type' = '"single"'
AND id IN (
select id FROM
(SELECT id, (SELECT COUNT(*) FROM jsonb_object_keys(tasks)) as keys FROM workflows)as keys_tuple
WHERE keys = 1
)
@camallen
camallen / aggregation_caesar_via_docker.sh
Last active August 6, 2019 15:01
Aggregation caesar for offline use via docker
# use docker to run the aggregation code container with the current dir as /data
docker run --rm -it --name aggregation_caesar -v ${PWD}:/data zooniverse/aggregation-for-caesar:latest bash
# based on the examples in the documentation at https://aggregation-caesar.zooniverse.org/Scripts.html
# configure the extractors/reducers from the workflows export
panoptes_aggregation config /data/workflows.csv $WORKFLOW_ID
# extract the data for the different tasks
# note the extractor config from the step above