malteos

Connect via SSH to a Slurm compute job that runs as Enroot container

Being able to SSH directly into a compute job has the advantage of using all remote development tools such as using your IDE's debugger also for GPU jobs (VSCode, PyCharm, ...).

• Slurm: Scheduling system that many HPC clusters use
• Enroot: Container system like Docker for NVIDIA GPUs

General problem:

malteos

#!/bin/bash
#SBATCH --job-name=oxw-bloom-1b7-twc-german
#SBATCH --ntasks-per-node=1          # crucial - only 1 task per dist per node!
#SBATCH --nodes=4
#SBATCH --gres=gpu:4       # ---> does not matter on JUWELS
#SBATCH --cpus-per-task=48           # number of cores per tasks
#SBATCH --hint=nomultithread         # we get physical cores not logical
#SBATCH --time=0-12:00:00
#SBATCH --output=%j.%x.out
#SBATCH --partition=booster

malteos

AF	Afghanistan
EG	Ägypten
AL	Albanien
DZ	Algerien
AD	Andorra
AO	Angola
AI	Anguilla
AQ	Antarktis
AG	Antigua und Barbuda
GQ	Äquatorial Guinea

malteos

# Copyright 2022 EleutherAI and The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,

malteos

import argparse

import os
import torch

from transformers.models.auto import AutoModelForCausalLM

LAYER_FILE_PREFIX = 'layer_'
MODEL_FILE_PREFIX = 'model_'
EMBEDDING_LAYER_INDEX = 1

malteos

# Obtain Lets-Encrypt SSL Certificate via Docker DNS challenge
# adjust:
# - domains (-d foo.me)
mkdir letsencrypt_etc letsencrypt_var
docker run -it --rm --name certbot \
    -v "./letsencrypt_etc:/etc/letsencrypt" \
    -v "./letsencrypt_var:/var/lib/letsencrypt" \
    certbot/certbot certonly -d foo.me -d *.foo.me --manual --preferred-challenges dns

malteos

[AutoDJ]

[Master]

[VinylControl]

[PreviewDeck1]

[Channel1]
play y

malteos

Returns only 'Main Page'

curl -XPOST localhost:9200/wiki_content/_search?pretty -d '
{
  "_source": [
    "title"
  ],
  "query": {
    "bool": {
      "should": [

malteos

Testing Queries for CirrusSearch extension

As produced by SimpleKeywordFeature.doApply() (boosting has no effect)

curl -XPOST localhost:9200/wiki_content/_search?pretty -d '
{
  "_source": [
    "title"
  ],
  "query": {

malteos

def auc(y_true, y_val, plot=False):
    """
    Computes the AUC (area under the receiver operator curve).
    For example, y val could be the output of a learning algorithm (binary logistic regression, ...).
    :param y_true: true labels in {-1,+1}
    :param y_val: predicted value where lower values tend to correspond to label -1 and higher values to label +1
    :param plot: whether to plot the ROC curve or not
    :return: returns the AUC
    """
    sum_cond_pos = np.count_nonzero(y_true == 1)