主要安装方法follow仓库中的Setup部分,部分操作需要修正。
- 安装apex时，可能存在当前cuda版本和pytorch编译cuda版本不匹配的报错，这时可以下载匹配版本的cuda driver只安装toolkit部分，然后用CUDA_HOME指定变异环境cuda进行apex的安装
Jianbin Chang shjwudp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """This script benchmarks tokenizer by counting text word segmentation efficiency. | |
| python tokenizer_benchmark.py \ | |
| --tokenizer_path TOKENIZER_MODEL_PATH \ | |
| --input_jsonl input.jsonl | |
| """ | |
| import argparse | |
| import json | |
| from transformers import AutoTokenizer | |
| from tqdm import tqdm |
shjwudp
/ install_bloom_deepspeed.md
Created
April 19, 2023 02:44
Install bigscience-workshop/Megatron-DeepSpeed
shjwudp
/ normally.log
Created
April 13, 2023 13:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (base) root@9fd6bda5c5df:~/wpang/chatglm_finetuning-main-20230406# bash start.sh | |
| Welcome 1 times | |
| INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpoghrns | |
| INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpoghrns/_remote_module_non_scriptable.py | |
| /home/cjiang/anaconda3/lib/python3.9/site-packages/pytorch_lightning/utilities/seed.py:47: LightningDeprecationWarning: `pytorch_lightning.utilities.seed.seed_everything` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.seed.seed_everything` instead. | |
| rank_zero_deprecation( | |
| INFO:lightning_fabric.utilities.seed:Global seed set to 42 | |
| INFO:pytorch_lightning.utilities.rank_zero:GPU available: True (cuda), used: True | |
| INFO:pytorch_lightning.utilities.rank_zero:TPU available: False, using: 0 TPU cores | |
| INFO:pytorch_lightning.utilities.rank_zero:IPU available: False, using: 0 IPUs |
shjwudp
/ gradient_accumulate.log
Created
April 13, 2023 13:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (base) root@9fd6bda5c5df:~/wpang/chatglm_finetuning-main-20230406# bash start.sh | |
| Welcome 1 times | |
| INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuiazaspy | |
| INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuiazaspy/_remote_module_non_scriptable.py | |
| /home/cjiang/anaconda3/lib/python3.9/site-packages/pytorch_lightning/utilities/seed.py:47: LightningDeprecationWarning: `pytorch_lightning.utilities.seed.seed_everything` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.seed.seed_everything` instead. | |
| rank_zero_deprecation( | |
| INFO:lightning_fabric.utilities.seed:Global seed set to 42 | |
| INFO:pytorch_lightning.utilities.rank_zero:GPU available: True (cuda), used: True | |
| INFO:pytorch_lightning.utilities.rank_zero:TPU available: False, using: 0 TPU cores | |
| INFO:pytorch_lightning.utilities.rank_zero:IPU available: False, using: 0 IPUs |
// 输入key-value序列，输出排序+去重后的序列
std::vector<std::pair<std::int64_t, std::string>>
sort(std::vector<std::pair<std::int64_t, std::string>>& A) {
std::vector<std::pair<std::int64_t, std::string>> result;
result.reserve(A.size());
std::sort(A.begin(), A.end());
shjwudp
/ gist:8c08f5e05550297d859cd074cddd1c84
Created
December 16, 2022 10:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| For writing task | |
| ``` | |
| Children who are brought up in families that do not have large amounts of money are | |
| better prepared to deal with the problems of adult life than children brought up by | |
| wealthy parents. | |
| To what extent do you agree or disagree with this opinion? | |
| ``` | |
| I write this: | |
| ``` | |
| Children who are brought up in famillies that wealthy parents better than parents has not large amounts of money. It is exactly NOT true. Good life or bad life can not determined by money. There are wealth in world has bad life, and there are people has good life from no money. Jay Chou is leader man in chinese music scene, he have a big success on music, enjoy life, be cool, play basketball, act a movie, marry a lovely woman. It's generally agreed that he had a pretty good life. But he do not has large amounts of money when he become a adult. It is not strange that people attributed their bad encounter to they have not a lot of money, but true things is how do you treat your life. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"msgsubject": ["\n 主题：本区是各类读者的推荐找书点评专区，平安共处，文明交流 ", "\n "], "comment": {"author": ["\n ", "№0 ", "☆☆☆", "管理员|", "6b5c6827", "♥♥♥", " ", "于", "2021-04-09 11:57:18留言", "☆☆☆ ", "\n ", "\n ", "加书签", "\n ", "举报", "\n ", "\n ", "\n "], "text": ["未发布版规前，百无禁忌，但是不可以发布娱乐圈、政治、人身攻击、色情等内容。 \r", "不可圈地赶人。 \r", "不可互<>暖互<>收。 \r", "不可宣群。\r", "不可抽卡算命、封建迷信。\r", "大家互相碰撞寻找最终规则。\r", "举报删贴功能已上线，每个贴子右下方有举报按钮"]}, "replies": [{"quote": [], "text": ["d"], "author": ["\n ", "№1", " ☆☆☆", "。|", "8cd7978f", "♥♥♥", "于", "2021-04- |
This file has been truncated, but you can view the full file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"user": "阮阮扫文推文", "text": "#言情推文# 书单推荐：男主是绿茶的言情文合集，欢迎大家排雷推荐[给你小心心][给你小心心]\\n1.《从此男主改拿绿茶剧本》作者：银发死鱼眼\\n2.《男扮女装的绿茶想攻略我》作者：锦橙 \\n3.《驸马如手足，情郎如衣服》作者：青色兔子\\n4.《别对我动心》作者：翘摇\\n5.《以理服人》作者：月下蝶影\\n6.《女扮男装后我拿了男主剧本》作者：别寒\\n7.《多宠着我点》作者：竹已\\n8.《将军》作者：苏他\\n9.《是祸躲不过》作者：丹青手\\n10.《君子坦荡荡 小人长嘤嘤》作者：坡西米\\n11.《圣僧中蛊之后[穿书]》作者：下限君一路好走\\n12.《被渣后绿茶影帝想撬走我》作者：四藏\\n13.《新欢》作者：南绫\\n14.《借我温柔》作者：红刺北\\n15.《我那夫君，柔弱不能自理》作者：杯雪 \\n16.《我的作精男友》作者：爱磕瓜子的葵花\\n17.《他真的很绿茶》作者：撒空空\\n18.《男主这朵黑心莲》作者：亮若星辰", "comments": [{"text": "马", "user": "每天只想把呆妞锁死", "sub_floor": []}, {"text": "4女主前边真的是太舔了[跪了][裂开]怎么能那么舔啊[裂开]看不下去了", "user": "最没力气", "sub_floor": []}, {"text": "👀", "user": "紫迷迭", "sub_floor": []}, {"text": "蹲2", "user": "伏地魔是我小弟", "sub_floor": []}, {"text": "1的女主好油腻…", "user": "养乐多喝几罐", "sub_floor": []}, {"text": "蹲蹲2，好看吗", "user": "旧日娇", "sub_floor": [{"text": "回复@旧日娇:前面挺搞笑的，后面女主知道男主男人身份以后感觉有点崩，看前面80%图个乐还可以的", "user": "有一点苦夏", "sub_floor": []}]}, {"text": "11为什么我搜到的文案不是这个?", "user": "月落星拾", "sub_floor": [{"text": "回复@月落星拾:去作者专栏看了一下，11这本应该叫《公主与圣僧二三事@阮阮扫文推文 |
shjwudp
/ topic_and_comments.jsonl
Created
November 15, 2022 08:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "status": { | |
| "user": { | |
| "screen_name": "阿七的推文分享", | |
| "id": 2998871231 | |
| }, | |
| "id": 4836085669298851, | |
| "text": "#求文# 🙏🙏🙏求名字\n求文！很早以前看过的了\n\n重生文，民国时期？\n女主前世和渣男去国外，流c+被抛弃。\n重生也是在国外，（不记得孩子还活着不），托人买了鸡这些补身体，休养好之后找了个金融公司的清洁工工作，暗地里炒股赚了钱回国，然后和渣男离婚，揭露渣男和白莲花小三 " | |
| }, | |
| "comments": [ |
This file has been truncated, but you can view the full file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "total_number": 3, | |
| "next_cursor": 0, | |
| "comments": [ | |
| { | |
| "idstr": "4836096133571378", | |
| "rootid": 4836087228535222, | |
| "disable_reply": 0, | |
| "created_at": "Tue Nov 15 15:50:31 +0800 2022", | |
| "mid": "4836096133571378", |