名無しＫさん koke2c95

## a_cah_to_parquet_pyspark.md

      
              4 files
            
          
              1 fork
            
          
              0 comments
            
          
              1 star
            
          
                rom1504
                / a_cah_to_parquet_pyspark.md
            
            
              Last active
              August 10, 2021 19:46
            
              
                cah_stats_spark.py
              
          
    This is a script to convert the csv of crawling at home to parquet
See the result at https://colab.research.google.com/drive/14Hc_fUUOrG9260VzD_XsTxWX7f5cptyL?usp=sharing

  
## j6b_train_hf_ds.py
#  So now you want to finetune that GPT-J-6B on a 3090/TITAN GPU ... okay
#  More exploratory coding. It uses the Huggingface model port, deepspeed and reads all text/md files from a target directory
#  It is a fragment of a larger system with remote editing, but that's another story
#  This is the raw, training tester. Items to look out for:
#  - uses DeepSpeed and has a DS config
#  - to save space uses SGD instead of ADAM
#  - uses gradient checkpointing
#  - freezes 25% of the layers to fit

# Assumes you can already run https://gist.github.com/kinoc/2d636a68876cd3de7b6e9c9452b61089

## LITD.user.js
// ==UserScript==
// @name         Look Into The Deep
// @namespace    LITD
// @version      10.0
// @description  ...
// @author       made by fredgy-kun
// @match        https://*.donmai.us/uploads/*
// @match        https://*.donmai.us/posts/*
// @connect      deepdanbooru.donmai.us
// @grant        GM.xmlHttpRequest

## 複数の役職に分割できる可能性のあるアニメスタッフの役職の表記一覧.txt
2Dデザイン・特殊効果
2Dワークス・特殊効果
2D・モニターデザイン
2D・特殊効果
3Dディレクター・モデリング
3Dディレクター・モデルデザイン
3Dメカニックデザイン・3D監修
3Dヴィジュアル・ディレクター
3D・CG制作・モニターグラフィックス
CAD・メカニックデザイン

## imagenet_class_index.json
[
    {
        "num": "n01440764",
        "en": "tench",
        "ja": "テンチ"
    },
    {
        "num": "n01443537",
        "en": "goldfish",
        "ja": "金魚"

## imagenet1000_clsidx_to_labels.txt
{0: 'tench, Tinca tinca',
 1: 'goldfish, Carassius auratus',
 2: 'great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias',
 3: 'tiger shark, Galeocerdo cuvieri',
 4: 'hammerhead, hammerhead shark',
 5: 'electric ray, crampfish, numbfish, torpedo',
 6: 'stingray',
 7: 'cock',
 8: 'hen',
 9: 'ostrich, Struthio camelus',
	# So now you want to finetune that GPT-J-6B on a 3090/TITAN GPU ... okay
	# More exploratory coding. It uses the Huggingface model port, deepspeed and reads all text/md files from a target directory
	# It is a fragment of a larger system with remote editing, but that's another story
	# This is the raw, training tester. Items to look out for:
	# - uses DeepSpeed and has a DS config
	# - to save space uses SGD instead of ADAM
	# - uses gradient checkpointing
	# - freezes 25% of the layers to fit

	# Assumes you can already run https://gist.github.com/kinoc/2d636a68876cd3de7b6e9c9452b61089
	// ==UserScript==
	// @name Look Into The Deep
	// @namespace LITD
	// @version 10.0
	// @description ...
	// @author made by fredgy-kun
	// @match https://.donmai.us/uploads/
	// @match https://.donmai.us/posts/
	// @connect deepdanbooru.donmai.us
	// @grant GM.xmlHttpRequest
	2Dデザイン・特殊効果
	2Dワークス・特殊効果
	2D・モニターデザイン
	2D・特殊効果
	3Dディレクター・モデリング
	3Dディレクター・モデルデザイン
	3Dメカニックデザイン・3D監修
	3Dヴィジュアル・ディレクター
	3D・CG制作・モニターグラフィックス
	CAD・メカニックデザイン
	[
	{
	"num": "n01440764",
	"en": "tench",
	"ja": "テンチ"
	},
	{
	"num": "n01443537",
	"en": "goldfish",
	"ja": "金魚"
	{0: 'tench, Tinca tinca',
	1: 'goldfish, Carassius auratus',
	2: 'great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias',
	3: 'tiger shark, Galeocerdo cuvieri',
	4: 'hammerhead, hammerhead shark',
	5: 'electric ray, crampfish, numbfish, torpedo',
	6: 'stingray',
	7: 'cock',
	8: 'hen',
	9: 'ostrich, Struthio camelus',