Skip to content

Instantly share code, notes, and snippets.

View dingp's full-sized avatar

Pengfei Ding dingp

  • NERSC/LBNL
View GitHub Profile
@dingp
dingp / Containerfile
Last active August 29, 2025 16:13
Container recipe for cxi+libfabric+mpich
FROM docker.io/nvidia/cuda:12.8.1-devel-ubuntu24.04
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install -y \
build-essential \
ca-certificates \
pkg-config \
automake \
autoconf \
@dingp
dingp / Containerfile-iperf
Created August 23, 2025 07:59
Container recipe for building N10 benchmark apps
FROM awlavely/gccopenmpi:2.0
RUN mkdir -p /n10-benchmark
RUN zypper --non-interactive addrepo https://download.opensuse.org/repositories/network:utilities/15.6/network:utilities.repo
RUN zypper --gpg-auto-import-keys refresh
RUN zypper --non-interactive update
@dingp
dingp / slurm4-podman-hpc.md
Last active August 8, 2025 06:36
Using slurm in podman-hpc containers

Using the slurm4 module with podman-hpc on Perlmutter

This guide shows how to run a podman-hpc container with the slurm4 module, submit a SLURM job from inside the container, and check the job from the host.

  • Note: The module flag matches the YAML filename; use --slurm4 for slurm4.yaml.
  • Requirement: The job script and output paths must be identical inside and outside the container. Bind-mount host paths to the exact same in-container paths, for example -v /path:/path.

Experimental status and limitations

  • The slurm4 module is experimental and may be removed at any time.
  • Not all SLURM commands/features will work inside the container with this module.
@dingp
dingp / create_default_users.py
Created August 4, 2025 04:32
scripts to inteact with MongoDB to recreate users/roles, generate random credentials and find databases updated in a year/month.
#!/usr/bin/env python3
"""
Script to create default users for MongoDB databases that have no users.
Creates two users per database: <db_name>_ro (read) and <db_name>_admin (dbOwner)
Can work with a JSON file or a specific database name.
"""
import json
import sys
import secrets
@dingp
dingp / send_user_emails.py
Last active August 1, 2025 01:26
script to send many user emails by taking in a csv file exported from service now query
#!/usr/bin/env python3
"""
Script to parse incident.csv file downloaded from ServiceNow query,
extract usernames from caller_id column,
and send email notifications.
"""
import argparse
import csv
import re
@dingp
dingp / build_run_push_mpi4py_matrix.sh
Last active July 18, 2025 06:40
Build docker images of different mpi4py version with different mpich versions
#!/bin/bash
build_and_migrate() {
local mpich=$1
local mpi4py=$2
podman-hpc build --build-arg=mpich=${mpich} --build-arg=mpi4py=${mpi4py} -t ghcr.io/dingp/ubuntu:22.04-mpich-${mpich}-mpi4py-${mpi4py} -f mpi4py-mpich.Dockerfile .
podman-hpc migrate ghcr.io/dingp/ubuntu:22.04-mpich-${mpich}-mpi4py-${mpi4py}
}
@dingp
dingp / slurm_job_report.py
Created June 19, 2025 21:29
Script to check job success rate for the past 24 hours
#!/usr/bin/env python3
import subprocess
import datetime
import sys
import os
from collections import defaultdict, Counter
def run_sacct_command():
"""Run sacct command to get jobs finished in the past 24 hours"""
@dingp
dingp / compare_rpms.py
Last active June 16, 2025 16:07
compare RPMs installed on two hosts
import argparse
import os
import subprocess
import time
def parse_rpm_list(file_path):
with open(file_path, 'r') as file:
rpms = {}
for line in file:
line = line.strip()
#!/bin/bash
GH_ACCT=$1
cd /data/chart_storage
# Loop over all subdirectories
for dir in *; do
# Remove trailing slash to get the subdirectory name
subdir_name="${dir%/}"
#!/bin/bash
declare -a distros=(
"alma"
"debian"
"ubuntu"
"fedora"
)
declare -a ubuntu_versions=(