Skip to content

Instantly share code, notes, and snippets.

@infinex
infinex / dag.py
Last active March 6, 2022 13:25
dagster to s3
from dagster import Out, fs_io_manager, job, op, DynamicOutput,DynamicOut
from dagster_aws.s3 import s3_pickle_io_manager, s3_resource
@op(out=DynamicOut(io_manager_key="s3_io"))
def files_in_directory(context):
for f in ["1","2","3"]:
yield DynamicOutput(
value=f,
# create a mapping key from the file name
@infinex
infinex / convert.py
Created June 4, 2021 08:02
translation fine tuning
import json
from pathlib import Path
from datasets import load_dataset
import random
ds = load_dataset("id_panl_bppt")
tr_data = []
val_data = []
@infinex
infinex / Makefile
Created June 4, 2021 05:40
makfile
TATOEBA_RELEASED_BT = https://object.pouta.csc.fi/Tatoeba-MT-bt/released-data.txt
tatoeba-all-bt:
for b in ${shell wget -qq -O - ${TATOEBA_RELEASED_BT} | grep -v '.txt' | cut -f1 -d'/' | sort -u}; do \
s=`echo $$b | cut -f1 -d'-'`; \
t=`echo $$b | cut -f2 -d'-'`; \
echo "${MAKE} -C bt-tatoeba SRC=$$s TRG=$$t fetch-bt"; \
echo "${MAKE} MODELTYPE=transformer-align HPC_CORES=2 HPC_MEM=32g tatoeba-$${t}2$${s}-train-bt.submitcpu"; \
done
@infinex
infinex / git
Last active April 23, 2021 09:28
git
# undo git add
git rm --cached -r dir
# apply patches
git diff master Branch1 > ../patchfile
git checkout Branch2
git apply ../patchfile
@infinex
infinex / MONO T5
Last active March 23, 2021 15:31
ranking
import torch
from transformers import AutoTokenizer, T5ForConditionalGeneration
passages = [['7744105', 'For Earth-centered it was Geocentric Theory proposed by greeks under the guidance of Ptolemy and Sun-centered was Heliocentric theory proposed by Nicolas Copernicus in 16th century A.D. In short, Your Answers are: 1st blank - Geo-Centric Theory. 2nd blank - Heliocentric Theory.'], ['2593796', 'Copernicus proposed a heliocentric model of the solar system â\x80\x93 a model where everything orbited around the Sun. Today, with advancements in science and technology, the geocentric model seems preposterous.he geocentric model, also known as the Ptolemaic system, is a theory that was developed by philosophers in Ancient Greece and was named after the philosopher Claudius Ptolemy who lived circa 90 to 168 A.D. It was developed to explain how the planets, the Sun, and even the stars orbit around the Earth.'], ['6217200', 'The geocentric model, also known as the Ptolemaic system, is a theory that was developed by
@infinex
infinex / fastapi_ap_scheduler.py
Last active February 10, 2023 10:24
python utils
from datetime import datetime, time
# FastAPI and Pydantic Related Libraries
from time import sleep
from apscheduler.events import EVENT_JOB_ERROR
from fastapi import FastAPI
from pydantic import BaseModel, Field
from typing import List
@infinex
infinex / bigsur.txt
Last active December 10, 2020 12:49
virtual machine qem setup big sur archlinux
https://www.funkyspacemonkey.com/how-to-install-macos-big-sur-with-opencore-on-linux
git clone https://github.com/kholia/OSX-KVM.git macOS-Big-Sur
./fetch-macOS.py --big-sur
3. Install xar. Arch and derivates users can install xar from the AUR. Now we need to extract SharedSupport.dmg from InstallAssistant.pkg.
If you’ve installed xar from AUR, run: xar -xf InstallAssistant.pkg
4. Install darling-dmg. If you’re running Arch or derivates, you can install the package from the AUR. We will need it to mount SharedSupport.dmg
If you’ve installed the package from the AUR, run: darling-dmg SharedSupport.dmg temp
5. Now that SharedSupport.dmg is mounted we need to extract BaseSystem.dmg from it. The file we need is located inside a .zip file, so we’ll use 7zip to extract it. To do that, run:
import tensorflow as tf
import math
from functools import partial
import numpy as np
def get_shape_list(tensor, expected_rank=None, name=None):
"""Returns a list of the shape of tensor, preferring static dimensions.
Args:
tensor: A tf.Tensor object to find the shape of.
expected_rank: (optional) int. The expected rank of `tensor`. If this is
@infinex
infinex / cloudSettings
Last active October 25, 2020 13:35
Visual Studio Code Settings Sync Gist
{"lastUpload":"2020-10-25T13:35:37.349Z","extensionVersion":"v3.4.3"}
@infinex
infinex / .vimrc
Last active October 12, 2020 10:37
vim
# ideavim
set visualbell
set noshowmatch
set relativenumber
set nohlsearch
set hidden
set noerrorbells
set tabstop=4 softtabstop=4
set shiftwidth=4