Skip to content

Instantly share code, notes, and snippets.

View MilesDowe's full-sized avatar

Miles Dowe MilesDowe

View GitHub Profile
@MilesDowe
MilesDowe / fill_missing_col.py
Created April 20, 2023 19:53
Simple way to update a column of a DF using values from another, similar DF.
bad = bad.set_index('KeyCol').MissingCol.dropna()
good.fillna(good.drop('MissingCol', 1).join(bad, on='KeyCol'))
@MilesDowe
MilesDowe / athena_presto-string-join-groupby-example.sql
Created July 12, 2022 20:27
Just a simple example of how to use `array_join/distinct/agg` to get a string list of results grouped by a value (in Athena/PrestoDB)
select
customer_id
array_join(array_distinct(array_agg(purchase_id)),'|') as current_purchases_list
from transactions
group by customer_id;
@MilesDowe
MilesDowe / vimwiki_org.md
Last active February 23, 2022 16:02
Vimwiki project board organization ideas

Using vimwiki or wiki.vim, I'm trying to setup some organization boards for projects. Here are some notes on the subject.

Board structure

Below is an example of how I structure individual project boards:

# Ad hoc tasks #

[backlog](./backlog.md) :: Tasks available for grooming
@MilesDowe
MilesDowe / blender_vid_editing_notes.md
Created November 26, 2021 04:05
Just some notes on editing videos in Blender.

Multiple audio tracks in a video

Blender doesn't recognize multiple audio tracks and will only provide the first (as far as I can tell).

What I instead do is:

  1. Rip the other audio track(s) into a separate audio file using ffmpeg.
  2. In Blender, click "add" and then "sound".
  3. Select the freshly ripped audio track.
@MilesDowe
MilesDowe / aws_codecommit_migration_example.md
Created November 22, 2021 18:57
Example of migrating CodeCommit repos from one account to another
  1. Add AWS creds (access + secret keys) for both the "original" and "new" account locations to your credentials file
  • In this example, we save them as profiles acct_orig and acct_new, respectively.
  1. Add CodeCommit power user permissions to an existing user in AWS console, for both accounts.
  2. For each of the users you added CodeCommit permissions to, also generate a Git username & password for that user in AWS IAM console. Keep it somewhere safe.
  3. On the machine you've saved the AWS credentials to, perform the following script
# for each repo in the acct_orig account/region
repo="<your repo name>"
aws codecommit create-repository --repository-name "${repo}" --profile acct_new

Installed from scoop, when using gpg, would error when starting gpg-agent --daemon, claiming "could not bind socket to" and a file under ~/scoop/apps/gpg/current/gnupg/S.gpg-agent.

To resolve, manually add the location and file, with contents below:

%Assuan%
socket=/the/location/of/this/file/youre/writing/S.gpg-agent

Then run gpg-agent --daemon and it's resolved.

Run pipenv run pip install -r <(pipenv lock -r) --target python/ to use pipenv to save deps locally under ./python for Lambda Layers usage.

Zip your python directory and import it as a Layer.

@MilesDowe
MilesDowe / pip_and_pipenv_corp_firewall_issues.md
Created October 15, 2021 19:00
How I fixed issues with WSS and the corporate firewall.

Had system update, company added Symantec WSS to OS. But caused issues in VM since didn't have cert.

Found cert near WSS executable and added to VM. Added to some apps, but had issues with pip and pipenv.

Solution was to fix pip with:

pip config set global.cert /path/to/wss-ssl-intercept-ca.crt

Had EC2 servers suddenly be really slow to log in. Was able to resolve by:

  1. Editing /etc/ssh/sshd_config to set UseDNS no.
    • This seemingly doesn't provide any added security, AFAIK.
  2. Restarting the service with sudo /sbin/service sshd restart.

If you deleted some file from your Jupyter UI, you should be able to retrieve it from this location:

~/.local/share/Trash/files/