Skip to content

Instantly share code, notes, and snippets.

@bebosudo

bebosudo/TIL.md

Last active November 13, 2023 14:39
Show Gist options
  • Save bebosudo/32147c6eba38b17f828eeaf00b45acef to your computer and use it in GitHub Desktop.
Save bebosudo/32147c6eba38b17f828eeaf00b45acef to your computer and use it in GitHub Desktop.
Download ZIP
Things I Learned/Today I Learned
Raw
TIL.md

Python

  • Extra requirement packages can be defined for a python package, using square brackets: pip install "project[extra]", see https://stackoverflow.com/a/46775606/
  • dask is a really cool framework for distributed tasks! For example, one may set an object in shared memory using .scatter: https://docs.dask.org/en/latest/futures.html#move-data
  • To uninstall a package installed with pip -e git which gives "No files were found to uninstall", search for a LIBRARYNAME.egg-link or LIBRARYNAME.egg-info file, and rename/delete it; a list of dirs where pip searches when doing a freeze can be found with python -c 'import sys; print(sys.path)'
  • Path configuration files can change the python path: https://stackoverflow.com/questions/60338280/
  • To test whether a port can be used, bind to the port on all the interfaces; see snippet in the bash section below
  • Convert 123MB into bytes
units = {"KB": 2**10, "MB": 2**20, "GB": 2**30, "TB": 2**40, "PB": 2**50}

def human_to_machine(filesize):
    human_unit = filesize[-2:].upper()
    if human_unit in units:
        return int(filesize[:-2]) * units[human_unit]
    else:  # if the last two digits are not a unit, it's already in bytes
        return int(filesize)

Or bytes to human:

IEC_UNITS = {2**0: 'Bi', 2**10: 'Ki', 2**20: 'Mi', 2**30: 'Gi', 2**40: 'Ti', 2**50: 'Pi',
             2**60: 'Ei', 2**70: 'Zi', 2**80: 'Yi'}

def b2h(nbytes):
    """Convert from bytes to human-readable powers of 2 amounts (ISO/IEC 80000)"""
    if nbytes < 0:
        raise ValueError("Number of bytes can't be lower than zero")

    for unit, unit_name in IEC_UNITS.items():
        if nbytes / unit < 1024:  # This means nbytes "fits" in this unit
            return nbytes / unit, unit_name
  • Reboot/crash-safe locking mechanism for single Linux box:
def get_lock():
    """Use a socket as a clean lock on the application. stackoverflow.com/a/7758075"""

    # Hold reference to our socket somewhere or it gets garbage collected at function exit.
    get_lock._lock_socket = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
    # Any placeholder here is fine.
    process_name = "my application name"

    try:
        # The null byte (\0) prefix means the socket is created in the abstract namespace instead
        # of being created on the file system itself; Linux only feature.
        get_lock._lock_socket.bind("\0" + process_name)
    except socket.error:
        # Here handle case of another instance of this application running
  • Basic logging config
import logging

_INFO = "\033[92m"     # GREEN
_WARNING = "\033[93m"  # YELLOW
_ERROR = "\033[91m"    # RED
_RESET = "\033[0m"     # RESET

logging.basicConfig(format="{asctime} [{levelname}] {message}", style="{", level=logging.INFO)
logging.addLevelName(logging.INFO, _INFO + logging.getLevelName(logging.INFO) + _RESET)
logging.addLevelName(logging.WARNING, _WARNING + logging.getLevelName(logging.WARNING) + _RESET)
logging.addLevelName(logging.ERROR, _ERROR + logging.getLevelName(logging.ERROR) + _RESET)

def _quit(*vargs, **kwargs):
    logging.error(*vargs, **kwargs)
    raise SystemExit

logging.quit = _quit
  • Extract MX records and select highest priority using host (from bind-utils package on RPM):
import subprocess as sp

def mx_to_smtp(domain):
    """Extract SMTP server with highest priority from MX record of argument `domain`."""
    try:
        # host is part of bind-utils on RHEL, but it should be almost universal.
       	cmd = ("host", "-t", "MX", domain)
        proc = sp.run(cmd, stdout=sp.PIPE, stderr=sp.PIPE, check=True)
    except sp.CalledProcessError as exc:
        logging.quit("Error while extracting SMTP hosts from domain '%s', error %s - %s",
                     domain, exc, exc.stderr.decode("utf-8").strip().replace("\n", "|"))

    prior_host = [line.split()[-2:] for line in proc.stdout.decode("utf-8").strip().split("\n")]
    _, hostname = sorted(prior_host, key=lambda pair: int(pair[0]))[0]
    return hostname.strip(".")
  • Send email function
import smtplib
from email.message import EmailMessage

def send_email(body, subject, recipients, email_from, smtp_server, smtp_port):
    msg = EmailMessage()
    msg.set_content(body)

    msg["Subject"] = subject
    msg["From"] = email_from
    msg["To"] = recipients

    logging.info("Opening connection to smpt server '%s:%s'", smtp_server, smtp_port)
    with smtplib.SMTP(smtp_server, port=smtp_port) as smtp:
        logging.info("Sending email to '%s'", recipients)
        # The output of .send_message is a dict with an entry for each recipient that failed.
        return smtp.send_message(msg)

Bash

  • Test if a port is available by binding to it on all the interfaces (0.0.0.0); Python 2.7,3+ needed; thanks to @afrighetto:
# If no return is present, the last command exitcode is returned.
function is_port_avail() {{
  python - <<'EOF' $1
import sys, errno, socket
from contextlib import closing

with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
  s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
  try:
    s.bind(("0.0.0.0", int(sys.argv[1])))
    sys.exit(0)  # 0 is True==available in Unix, 1 is False==already taken
  except socket.error as e:
    sys.exit(1)
EOF
}}

is_port_avail 1234 && run_server 1234
  • print all text to screen, while saving only certain text to file; useful if debug is too verbose to be stored, but still useful to be seen in the window of a screen session (|& == 2>&1 |):
$ echo -e "INFO: ciao\nDEBUG:buh\nDEBUG:asfasd\nINFO: asdfaasdfad" |& tee /dev/tty | grep -v DEBUG > /info.log
INFO: ciao
DEBUG:buh
DEBUG:asfasd
INFO: asdfaasdfad

$ cat /tmp/info.log 
INFO: ciao
INFO: asdfaasdfad
  • spawn a single ssh agent and recycle agent process details between sessions; add to your ~/.bashrc:
# Spawn an ssh agent if not running, and save details to reuse it.
ssh-add -l &>/dev/null
if [ "$?" == 2 ]; then
    # Could not open a connection to your authentication agent.

    # Load stored agent connection info.
    test -r ~/.ssh-agent && \
        eval "$(<~/.ssh-agent)" >/dev/null

    ssh-add -l &>/dev/null
    if [ "$?" == 2 ]; then
        # Start agent and store agent connection info.
        (umask 066; ssh-agent > ~/.ssh-agent)
        eval "$(<~/.ssh-agent)" >/dev/null
    fi
fi

kubernetes

  • Generic intros to k8s:

  • Pod: 1+ containers. 99% of times it's a single container; there may be cases where multiple containers may need to go hand-in-hand inside a pod, to form a single micro-service.

  • Service: expose a pod/container to the cluster or the external world, using things like ClusterIP, NodePort, LoadBalancer, etc; all these services need to specify certain parameters:

    • port is the one exposed to the rest of the cluster;
    • targetPort is the one that the application inside the container listens to (could it be also set using ConfigMaps?);
    • nodePort opens a port on the physical server the container is running on (if NodePort service is defined, but no nodePort param, random # in range: 30000-32767);

  • Resources: "requests" set the minimum amount of resources to allocate, while "limits" specify the maximum amount of resources that the pod/container can reach during runtime. It's important to set requests on containers, otherwise k8s will use the limit as a request, and it'll try to allocate the pod on a node using the limit; so always set a minimum amount in requests, to improve pod allocation scheduling

  • Resources: setting a low amount of memory on a container may cause it to be OOM-Killed (see it with kubectl describe pod _pod_name_), but setting a low amount of cpu share will just slow down the pod execution, and won't cause any error (because CPU is a compressible resource). See: https://medium.com/@betz.mark/understanding-resource-limits-in-kubernetes-cpu-time-9eff74d3161b

  • labels: .metadata.labels are labels related to the object being defined (the deployment, the service, etc); .spec.selector.matchLabels or .spec.selector need to point to the objects we want to reference, and have to be identical to .spec.template.metadata.labels; https://medium.com/@zwhitchcox/matchlabels-labels-and-selectors-explained-in-detail-for-beginners-d421bdd05362

  • Storage: a persistent volume (pv) is the base "unit" of storage; it can by manually created, but the best thing to do is letting k8s manage them, through persistent volume claims (pvc). PVC are pieces of software that connect PV to pods; a pod does not directly ask for a specific PV, but let the PVC claim it from a storage class object (can a storage class be set as default?)

  • kubectl diff -f depl.yaml reports the difference between the current status of the system and the one requested in depl.yaml

  • Make sure to wait for all components to be deleted before reinstalling a helm chart, or the new installation may find leftover components that are being deleted and not recreate them: helm uninstall this; sleep 1m; helm install --debug --wait this

  • Run pod on-the-fly: kubectl -n mysql-shell run temp-mysql-shell --image=mysql:5 --command -- sh -c 'sleep infinity'

  • Create secret from cmdline: read -sp "Registry password: " HUB_PASSWORD; kubectl -n namespacehere create secret docker-registry reg-creds --docker-server=hub.example.org --docker-username="myuser@example.org" --docker-password=${HUB_PASSWORD}

  • To add a new TLS certificate to k8s behind a AWS Load Balancer, go to EC2 > Load Balancer > select the one connected to the k8s cluster > click on the https:443 listener > in the new window, choose the Certificates tab > Add a new one > Select the new one. Alternatively, you can set the new one as a new default, if it covers also domains previously covered by the older certificate.

  • Environment variables need to be expanded with $(variable) rather than ${variable} when passed as arguments to an entrypoint, see docs.

Kafka

Rust

Apache httpd

  • Location directives need to be in reverse order of relevance, first /, then /app: https://serverfault.com/a/391488
  • Multiple rewrite conditions can be set up following the scheme RewriteCond X \n RewriteCond Y \n RewriteRule Z and then again [Cond]+ Rule
  • To proxy websockets along with http(s):
<VirtualHost *:80>
    RewriteEngine On
    # Maybe not useful, as many app directly open wss://
    RewriteCond %{HTTP:UPGRADE} =websocket [NC]
    RewriteCond %{HTTP:CONNECTION} Upgrade [NC]
    RewriteRule ^/?(.*) wss://%{SERVER_NAME}/$1 [P]

    # Redirect to HTTPS
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [END,NE,R=permanent]
</VirtualHost>

<VirtualHost *:443>
    RewriteEngine On
    RewriteCond %{HTTP:UPGRADE} =websocket [NC]
    RewriteCond %{HTTP:CONNECTION} Upgrade [NC]
    RewriteRule ^/?(.*) ws://backend:5678/$1 [P]

    # Always redirect to www. version
    RewriteCond %{HTTP_HOST} !^www\. [NC]
    RewriteRule ^/?(.*) https://www.%{SERVER_NAME}/$1 [END,NE,R=permanent]
    
    # From least specific (/) to more specific (/app)
    <Location />
        # default site config/proxypass
    </Location>
    
    <Location /app>
        ProxyPass http://backend:5678/app
        ProxyPassReverse http://backend:5678/app
    </Location>
</VirtualHost>

SLURM

FFmpeg

  • Download a m3u8 playlist, using only FFmpeg: ffmpeg -i "https://example.com/playlist.m3u8" -codec copy output.mp4

GIMP

  • To crop a person out of a picture to a 512x512 px image with 5 px white border (e.g. Telegram stickers):
    • Tools > Selection Tools > Intelligent Scissors [I]
    • Connect all the points around the image until you close the loop; click inside it to convert it to a selection
    • Layer > Transparency > Add Alpha Channel
    • Select > Invert [Ctrl+I]
    • Edit > Clear [Delete]
    • Image > Crop to content
    • Image > Scale Image > 502 px on the larger of the two axes
    • Image > Canvas size > 512 px to both axes > Center
    • Select > None [Ctrl+Shift+A]
    • Layer > Transparency > Alpha to Selection
    • Layer > New Layer [Ctrl+Shift+N] > Transparency
    • Move the empty layer below the one with the cropped image
    • Select > Border > 5 px
    • Edit > Fill with FG Color (should be white)
    • File > Export as > Name the file with .png to preserve the transparency

Ansible

  • 'NoneType' object is not iterable on a simple ping of a server (ansible 2.9); delete the cache file in cache.d/${SERVER}
  • Transform nested dicts in set_facts (no with_items/loop?) using jinja2. E.g. given the following in conf.yaml:
users:
  user1:
    name: Ajeje Brazov
  user2:
    name: Antani Coso
  user3:
    name: Tizio Caio

to obtain the following:

set_fact:
  sublist_users:
  - nick: user1
    name: Ajeje Brazov
  - nick: user3
    name: Tizio Caio

use this jinja snippet:

set_fact:
  sublist_users: |
    {% set list_users = [] -%}
    {% for _user in ['user1', 'user3', ] -%}
      {% set _ = list_users.append({'nick': _user, 'name': users[_user].name}) -%}
    {%- endfor %}
    {{ list_users }}
  • Load a file/template into a ansible variable, differentiating by distro, with fallback to generic file/template:
- name: If variable_to_load_into is undefined, load it from file
  set_fact:
    variable_to_load_into: '{{ lookup("template", item) }}'
  with_first_found:
    - '../templates/{{ "progr-config-" ~ progr_release ~ "-" ~ ansible_os_family ~ ".cfg.j2" }}'
    - '../templates/{{ "progr-config-" ~ progr_release ~ ".cfg.j2" }}'
    - '../templates/progr-config.cfg.j2'
  when: variable_to_load_into is not defined

if you want to avoid template parsing change the set_fact with variable_to_load_into: '{{ lookup("file", item) }}'. This tries to load ../templates/progr-config-123.456-RedHat.cfg.j2 first, fallbacking to ../templates/progr-config-123.456.cfg.j2 and finally to ../templates/progr-config.cfg.j2.

  • To "take track" of a long-running shell command, add a |& tee /dev/shm/job_$(date -u +"%Y-%m-%dT%H-%M").log at the end of the shell command.

Systemd💩

Leaflet

  • To clear all the layers added to a map except for the "basic" layer (e.g. the OSM satellite tiles), use:
while (Object.keys(mymap._layers).length > 1) { mymap.removeLayer(mymap._layers[Object.keys(mymap._layers)[1]]); }

Unison

Install unison 2.51.2 on Ubuntu 16.04:

$ wget https://github.com/ocaml/opam/releases/download/2.1.0-alpha/opam-2.1.0-alpha-x86_64-linux
$ chmod +x opam-2.1.0-alpha-x86_64-linux
$ mv ./opam-2.1.0-alpha-x86_64-linux ~/bin/  # Move the opam binary into one of the $PATH dirs
$ ln -s opam-2.1.0-alpha-x86_64-linux ~/bin/opam

$ mkdir temp_bubblewrap && cd temp_bubblewrap
$ wget http://security.ubuntu.com/ubuntu/pool/main/b/bubblewrap/bubblewrap_0.2.1-1ubuntu0.1_amd64.deb
$ ar x bubblewrap_0.2.1-1ubuntu0.1_amd64.deb
$ tar xf data.tar.xz ./usr/bin/bwrap
$ mv usr/bin/bwrap ~/bin/  # Move the bwrap binary into one of the $PATH dirs

$ opam switch create 4.08.1
$ eval $(opam env --switch=4.08.1)

$ wget https://github.com/bcpierce00/unison/archive/v2.51.2.tar.gz
$ mv {v,unison-}2.51.2.tar.gz
$ tar -xf unison-2.51.2.tar.gz
$ cd unison-2.51.2/

$ wget https://git.archlinux.org/svntogit/packages.git/plain/trunk/4.08-compatibility.patch?h=packages/unison
$ mv 4.08-compatibility.patch\?h\=packages%2Funison unison-ocaml-4.08.patch
$ patch -p1 <unison-ocaml-4.08.patch
$ make

# mv src/unison /usr/bin/unison-2.51-ocaml-4.08.1
# ln -s unison-2.51 /usr/bin/unison-2.51.2

Cray

  • To show all compiler and linker options: cc -v -craype-verbose

SSL

  • Show chain of certificates in file:
$ openssl crl2pkcs7 -nocrl -certfile certificate_file.pem | openssl pkcs7 -print_certs -noout
subject=/C=YourCountryHere/ST=YourStateHere/O=Your Organization Name Here/OU=Self-Enrolled/CN=common.name.here.example.com
issuer=/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA

subject=/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA
issuer=/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority

subject=/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services

Make sure to order certificates in the order requested by your application. Nginx requires the most local certificate to be specified first, then the intermediate certificates following. The issuer of a certificate should match with the subject of the line below. You should drop the root certificate (the one where the subject matches the issuer) to reduce trust chain issues.

  • Check modulus of certificate and private key for them to match (only the first certificate is used, so ordering matters)
$ openssl rsa -modulus -noout -in website.key |openssl md5
(stdin)= 1234567890

$ openssl x509 -modulus -noout -in website.cert |openssl md5
(stdin)= 1234567890

GitHub Action

  • Add a debug block to print the content of the github context variable 
      - name: Debug GH context
    run: echo "$GITHUB_CONTEXT"
    env:
      GITHUB_CONTEXT: ${{ toJson(github) }}

Podman

  • To make podman look for unqualified (without hostname) images in Docker Hub, set unqualified-search-registries = ["docker.io"] in /etc/containers/registries.conf.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment