Kilian Lieret klieret

## hf_model_downloader.md

      
              2 files
            
          
              0 forks
            
          
              0 comments
            
          
              5 stars
            
          
                muhark
                / hf_model_downloader.md
            
            
              Last active
              June 20, 2024 05:37
            
              
                Offline HuggingFace Models on HPC
              
          
    Downloading HuggingFace Models

This gist shares a little workflow and script for a task that most people using university HPCs for NLP research will need to do: downloading and storing HuggingFace models for use on compute nodes.
What this workflow is for:

Context: you want to use HuggingFace models on Della (or other HPC clusters).
Problem 1: you cannot call AutoModel.from_pretrained('model/name') at run time because compute nodes are not connected to the internet.
Problem 2: running AutoModel.from_pretrained() on the head node is impractical because the model is too large to be loaded.
Problem 3: you do not want to save the model weights to the default ~/.cache/ because you only get 10GB of storage on /home


## AnkiCardSimilarity.py
import os
import re
import sqlite3
import zipfile

import numpy as np
import pandas as pd
import spacy
from ankipandas import Collection
from sklearn.feature_extraction.text import TfidfVectorizer

## install_mosh.md

      
              1 file
            
          
              0 forks
            
          
              0 comments
            
          
              4 stars
            
          
                gengwg
                / install_mosh.md
            
            
              Last active
              March 28, 2021 18:37
            
              
                install mosh on raspberry pi
              
          
    On Server

sudo apt install mosh
sudo ufw allow from 192.168.1.0/24 to any port 60000:61000 proto udp
On Client


## README.md

      
              1 file
            
          
              1 fork
            
          
              5 comments
            
          
              21 stars
            
          
                ravron
                / README.md
            
            
              Last active
              March 18, 2024 18:16
            
              
                Options to prevent accidental Yubikey OTP triggering
              
          
    Tired of spamming Yubikey one-time password (OTP) codes into Slack? Here are two options to help prevent that. You can do either or both. Both require ykman, the Yubikey CLI configuration manager. Get it with Homebrew:
brew install ykman

If you…

…don't have Homebrew? Install it, then run the command above
…don't feel comfortable running terminal commands? Skip to the bottom
…aren't sure what the big deal is? See "What's the big deal with Yubikey codes?" below


## toggle_keybindings.tmux.conf
bind -T root F12  \
  set prefix None \;\
  set key-table off \;\
  set status-style "fg=$color_status_text,bg=$color_window_off_status_bg" \;\
  set window-status-current-format "#[fg=$color_window_off_status_bg,bg=$color_window_off_status_current_bg]$separator_powerline_right#[default] #I:#W# #[fg=$color_window_off_status_current_bg,bg=$color_window_off_status_bg]$separator_powerline_right#[default]" \;\
  set window-status-current-style "fg=$color_dark,bold,bg=$color_window_off_status_current_bg" \;\
  if -F '#{pane_in_mode}' 'send-keys -X cancel' \;\
  refresh-client -S \;\

bind -T off F12 \

## limits.py
import scipy.special
import scipy.stats

def binomial_limit(n, k, sigma=1):
    """
    Calculates the upper and lower limit for the probability p of a binomial distribution
    if an experiment yielded k successes for n trials.
    The confidence level for the limits is given in sigmas of the gaussian distribution.
    In contrast to other methods (see https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval),
    this method is exact in the sense that there is no approximation involved for the binomial distribution.

## getting-started-with-airflow.md

      
              1 file
            
          
              3 forks
            
          
              3 comments
            
          
              22 stars
            
          
                rosiehoyem
                / getting-started-with-airflow.md
            
            
              Last active
              January 14, 2023 18:09
            
              
                Getting Started With Airflow
              
          
    Install Airflow

1. Install Airflow

Follow the installation instructions on the Airflow website.
Update Airflow Configurations

To configure Airflow to use Postgres rather than the default Sqlite3, go to airflow.cfg and update this configuration to LocalExecutor:
# The executor class that airflow should use. Choices include

  
## redirecting-github-pages.md

      
              1 file
            
          
              6 forks
            
          
              23 comments
            
          
              110 stars
            
          
                domenic
                / redirecting-github-pages.md
            
            
              Created
              February 10, 2017 19:28
            
              
                Redirecting GitHub pages after a repository move
              
          
    Redirecting GitHub Pages after a repository move

The problem

You have a repository, call it alice/repo. You would like to transfer it to the user bob, so it will become bob/repo.
However, you make heavy use of the GitHub Pages feature, so that people are often accessing https://alice.github.io/repo/. GitHub will helpfully redirect all of your repository stuff hosted on github.com after the move, but will not redirect the GitHub Pages hosted on github.io.
The solution


## for-loop-sorted-collection
{% comment %}
*
*  This loop loops through a collection called `collection_name`
*  and sorts it by the front matter variable `date` and than filters
*  the collection with `reverse` in reverse order
*
*  To make it work you first have to assign the data to a new string
*  called `sorted`.
*
{% endcomment %}

## client.html
<html>
    <body>
        <h1>I feel lonely</h1>

        <script type="text/javascript" src="//code.jquery.com/jquery-2.1.3.min.js"></script>
        <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/socket.io/0.9.16/socket.io.min.js"></script>
        <script type="text/javascript" charset="utf-8">
            var socket = io.connect('http://' + document.domain + ':' + location.port);
            socket.on('connect', function() {
                socket.emit('connected');
	import os
	import re
	import sqlite3
	import zipfile

	import numpy as np
	import pandas as pd
	import spacy
	from ankipandas import Collection
	from sklearn.feature_extraction.text import TfidfVectorizer
	bind -T root F12 \
	set prefix None \;\
	set key-table off \;\
	set status-style "fg=$color_status_text,bg=$color_window_off_status_bg" \;\
	set window-status-current-format "#[fg=$color_window_off_status_bg,bg=$color_window_off_status_current_bg]$separator_powerline_right#[default] #I:#W# #[fg=$color_window_off_status_current_bg,bg=$color_window_off_status_bg]$separator_powerline_right#[default]" \;\
	set window-status-current-style "fg=$color_dark,bold,bg=$color_window_off_status_current_bg" \;\
	if -F '#{pane_in_mode}' 'send-keys -X cancel' \;\
	refresh-client -S \;\

	bind -T off F12 \
	import scipy.special
	import scipy.stats

	def binomial_limit(n, k, sigma=1):
	"""
	Calculates the upper and lower limit for the probability p of a binomial distribution
	if an experiment yielded k successes for n trials.
	The confidence level for the limits is given in sigmas of the gaussian distribution.
	In contrast to other methods (see https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval),
	this method is exact in the sense that there is no approximation involved for the binomial distribution.
	{% comment %}
	*
	* This loop loops through a collection called `collection_name`
	* and sorts it by the front matter variable `date` and than filters
	* the collection with `reverse` in reverse order
	*
	* To make it work you first have to assign the data to a new string
	* called `sorted`.
	*
	{% endcomment %}
	<html>
	<body>
	<h1>I feel lonely</h1>

	<script type="text/javascript" src="//code.jquery.com/jquery-2.1.3.min.js"></script>
	<script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/socket.io/0.9.16/socket.io.min.js"></script>
	<script type="text/javascript" charset="utf-8">
	var socket = io.connect('http://' + document.domain + ':' + location.port);
	socket.on('connect', function() {
	socket.emit('connected');