A Python script for splitting text into parts with controlled (limited) length in tokens. This script utilizes the tiktoken
library for encoding and decoding text.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
### steps #### | |
# verify the system has a cuda-capable gpu | |
# download and install the nvidia cuda toolkit and cudnn | |
# setup environmental variables | |
# verify the installation | |
### | |
### to verify your gpu is cuda enable check |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"env": { | |
"browser": true, | |
"node": true, | |
"es6": true | |
}, | |
"plugins": ["react"], | |
"ecmaFeatures": { |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pathlib | |
import hydra | |
hydra._internal.hydra.GlobalHydra().clear() | |
config_dir = pathlib.Path('/path/to/configs/') | |
hydra.experimental.initialize(config_dir=config_dir) | |
cfg = hydra.experimental.compose(config_file='config.yaml', overrides=[]) | |
print(cfg.pretty()) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/system/xbin/bash | |
#Based on https://github.com/termux/termux-app/issues/77 | |
export PREFIX='/data/data/com.termux/files/usr' | |
export HOME='/data/data/com.termux/files/home' | |
export LD_LIBRARY_PATH='/data/data/com.termux/files/usr/lib' | |
export PATH="/data/data/com.termux/files/usr/bin:/data/data/com.termux/files/usr/bin/applets:$PATH" | |
export LANG='en_US.UTF-8' | |
export SHELL='/data/data/com.termux/files/usr/bin/bash' | |
export BIN='/data/data/com.termux/files/usr/bin' | |
export TERM=vt220 |
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#Add e5-instruct-mistral layers, so they naming is different than | |
# original mistral instruct one | |
from __future__ import annotations | |
from typing import Sequence | |
from .constants import MODEL_ARCH, MODEL_TENSOR, MODEL_TENSORS, TENSOR_NAMES | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
// http://eslint.org/docs/rules/ | |
"ecmaFeatures": { | |
"binaryLiterals": false, // enable binary literals | |
"blockBindings": false, // enable let and const (aka block bindings) | |
"defaultParams": false, // enable default function parameters | |
"forOf": false, // enable for-of loops | |
"generators": false, // enable generators | |
"objectLiteralComputedProperties": false, // enable computed object literal property names |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
" Beginners .vimrc | |
" v0.1 2012-10-22 Philip Thrasher | |
" | |
" Important things for beginners: | |
" * Start out small... Don't jam your vimrc full of things you're not ready to | |
" immediately use. | |
" * Read other people's vimrc's. | |
" * Use a plugin manager for christ's sake! (I highly recommend vundle) | |
" * Spend time configuring your editor... It's important. Its the tool you | |
" spend 8 hours a day crafting your reputation. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# This hook is particularly useful when ablating layers | |
class ContextHook: | |
def __init__(self, layer): | |
self.layer = layer | |
def __enter__(self): | |
self.handle = self.layer.register_forward_hook(self.hook) | |
return self |
NewerOlder