Skip to content

Instantly share code, notes, and snippets.

View odashi's full-sized avatar
🏠
Working from home

Yusuke Oda odashi

🏠
Working from home
View GitHub Profile
@odashi
odashi / download_c4_en.bash
Last active January 14, 2024 00:22
Downloads AllenAI mC4 dataset
#!/bin/bash
url_prefix=https://huggingface.co/datasets/allenai/c4/resolve/1ddc917116b730e1859edef32896ec5c16be51d0/en
function download() {
dataset_name=$1; shift
splits=$1; shift
last_split=$(perl -e "print ${splits} - 1")
mkdir -p ${dataset_name}
for i in $(seq 0 ${last_split}); do
@odashi
odashi / gemini_adc.py
Last active December 20, 2023 05:59
Example to call Gemini API with ADC
# You may need to install the following libraries:
# * google-auth
# * requests
import json
from typing import Any
import google.auth
import google.auth.credentials
import google.auth.transport.requests
@odashi
odashi / auth.tsx
Created December 9, 2023 14:48
Firebase Authentication provider with next.js
"use client";
import { getApp, getApps, initializeApp } from "firebase/app";
import {
getAuth,
GoogleAuthProvider,
signInWithPopup,
User,
} from "firebase/auth";
import {
@odashi
odashi / BUILD
Last active November 17, 2022 06:27
Bazel rule to generate Python structs from OpenAPI documents
load("@pip_deps//:requirements.bzl", "requirement")
py_binary(
name = "datamodel_codegen",
srcs = ["datamodel_codegen_main.py"],
main = "datamodel_codegen_main.py",
visibility = ["//visibility:public"],
deps = [
requirement("datamodel-code-generator"),
],
@odashi
odashi / class_fastapi_app.py
Last active April 13, 2023 15:14
FastAPI server defined on a class
# Sometimes you want to define your FastAPI app within a class for some reason, e.g.,
# you wanted to initialize the servers at the point you intended, not the point of import.
#
# In this case, @app.method decorator doesn't work as-is with instance methods due to the
# difference of its signature.
#
# You need to manually call the decorator as a usual function after obtaining self (e.g.,
# in the __init__ method like below) rather than using the decorator syntax.
class MyApp:
@odashi
odashi / nai_prompt_generator.py
Last active October 26, 2022 14:57
Colab UI to generate NovelAI prompts.
!pip install ipywidgets &> /dev/null
import ipywidgets as wgt
controls = wgt.VBox([])
add_button = wgt.Button(description="Add", icon="plus")
pos_prompt_area = wgt.Textarea(placeholder="Positive prompts appear here.")
neg_prompt_area = wgt.Textarea(placeholder="Negative prompts appear here.")
ui = wgt.VBox([pos_prompt_area, neg_prompt_area, add_button, controls])
def generate_prompt(change):
@odashi
odashi / should_mute_on_twitter.txt
Last active December 26, 2022 13:02
should_mute.txt
あまり大きな声で言えないのですが
あまり知られてないのですが
ありえない
あんまり言いたくないけど
いい加減にしてほしい
いい歳してお恥ずかしいのですが
悪用厳禁なのですが
安心してください
言いたくない過去を言うと
言わせてください
@odashi
odashi / delete_old_tweets.py
Last active July 26, 2023 20:35
Cloud Functions script to delete old tweets.
import datetime
import logging
import os
from google.cloud import logging as cloud_logging
import tweepy
THRESHOLD_DAYS = 3
BATCH_SIZE = 25
@odashi
odashi / print-xml.py
Last active December 1, 2021 07:14
Pretty-prints XML file.
#!/usr/bin/env python3
import argparse
from xml.dom import minidom
def main():
p = argparse.ArgumentParser(description='Pretty-print an XML file.')
p.add_argument('file', type=str, help='File path to print.')
args = p.parse_args()
data = open(args.file).read()
@odashi
odashi / kytea_word_segmenter.cc
Last active January 26, 2021 07:11
Simple KyTea word segmenter-only pipeline.
// Wrapper class of KyTea word segmenter.
// Author: odashi
// Date: 2021-01-26
// License: MIT
#include <memory>
#include <string>
#include "kytea/kytea.h"
#include "kytea/string-util.h"