This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
⭐ Total Stars: 57 | |
➕ Total Commits: 680 | |
🔀 Total PRs: 11 | |
🚩 Total Issues: 7 | |
📦 Contributed to: 10 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🌞 Morning 131 commits ███████▏░░░░░░░░░░░░░ 34.3% | |
🌆 Daytime 179 commits █████████▊░░░░░░░░░░░ 46.9% | |
🌃 Evening 69 commits ███▊░░░░░░░░░░░░░░░░░ 18.1% | |
🌙 Night 3 commits ▏░░░░░░░░░░░░░░░░░░░░ 0.8% |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
# -*- coding: utf-8 -*- | |
from argparse import ArgumentParser | |
import torch | |
import torch.distributed as dist | |
from torch.nn.parallel import DistributedDataParallel as DDP | |
from torch.utils.data import DataLoader, Dataset | |
from torch.utils.data.distributed import DistributedSampler | |
from transformers import BertForMaskedLM |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/** | |
* Replace a PositionedImage by its ID. | |
* | |
* @param {Paragraph|ListItem} anchor The element (Paragraph or ListItem) to which the PositionedImage is anchored. | |
* @param {Number} positionedImageId The ID of the PositionedImage. | |
* @param {Blob} image The image used to replace the "old" PositionedImage with. | |
*/ | |
function ReplacePositionedImage( anchor, positionedImageId, image ) { | |
// get the positioned image by its ID | |
var positionedImage = anchor.getPositionedImage(positionedImageId); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
# -*- coding: utf-8 -*- | |
import timeit | |
import torch | |
from torch.utils.data import DataLoader, SequentialSampler, TensorDataset | |
from transformers import DistilBertForMaskedLM | |
from tqdm import tqdm | |
NUM_EXAMPLES = 128 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
# -*- coding: utf-8 -*- | |
import sys | |
from pathlib import Path | |
from tokenizers import BertWordPieceTokenizer | |
def main(): | |
in_file = Path(sys.argv[1]) |
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
implementation | mean execution time | |
---|---|---|
submit | 1min 8s | |
map | 1min 9s | |
encode_batch | 10.6s |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
implementation | mean execution time | |
---|---|---|
transformers | 6min 42s | |
tokenizers | 45.6s |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
# -*- coding: utf-8 -*- | |
import sys | |
from pathlib import Path | |
from blingfire import text_to_sentences | |
def main(): | |
wiki_dump_file_in = Path(sys.argv[1]) |