Skip to content

Instantly share code, notes, and snippets.

@seandearnaley
seandearnaley / model_metrics_may_25_24.csv
Created May 25, 2024 20:16
Model Metrics (Updated May 25th 2024)
Model Name Article Key Inference Rate (s) Valid JSON Rate Sentiment Variance Mean Sentiment Mean Confidence
phi3-4k-sentiment-may-24-2024-2epoches-unsloth.Q4_K_M f76ab0da 2.37 1.0 0.0 0.61 0.8
phi3-4k-sentiment-may-24-2024-2epoches-unsloth.Q4_K_M 3d6a60c0 1.88 1.0 0.01 -0.46 0.8
phi3-4k-sentiment-may-24-2024-2epoches-unsloth.Q4_K_M 136f0dde 2.62 1.0 0.0 0.01 0.9
phi3-4k-sentiment-may-24-2024-2epoches-unsloth.Q4_K_M ce972942 1.69 1.0 0.0 0.48 0.77
phi3-4k-sentiment-may-24-2024-2epoches-unsloth.Q4_K_M 028bdea3 2.66 1.0 0.0 -0.41 0.8
phi3-4k-sentiment-may-24-2024-2epoches-unsloth.Q4_K_M 6048aefc 2.7 1.0 0.02 0.56 0.79
phi3-4k-sentiment-may-24-2024-2epoches-unsloth.Q4_K_M 1e2173be 1.99 0.93 0.01 0.57 0.78
phi3-4k-sentiment-may-24-2024-2epoches-unsloth.Q4_K_M 3e1daab6 4.24 1.0 0.11 0.38 0.88
phi3-4k-sentiment-may-24-2024-2epoches-unsloth.Q4_K_M abc67ae2 2.12 1.0 0.0 0.61 0.8
@seandearnaley
seandearnaley / Modefile
Last active May 17, 2024 20:13
Ollama Modelfile
FROM ./llama3-8b-sentiment-may-3-2024-unsloth.Q4_K_M.gguf
SYSTEM """
You are an advanced AI assistant created to perform sentiment analysis on text. Your task is to carefully read the text and analyze the sentiment it expresses towards the potential future stock value of any company mentioned. Analyze the sentiment of this text and respond with the appropriate JSON:
"""
TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>
@seandearnaley
seandearnaley / statistical_comparisons.csv
Created May 17, 2024 19:27
Statistical Comparisons
Comparison F-Statistic F-Test P-Value T-Statistic T-Test P-Value
llama3_8b-instruct-fp16 vs llama3_8b-instruct-sentiment_analysis-fp16 8.235 0.005 -2.87 0.005
llama3_8b-instruct-q4_K_M vs llama3_8b-instruct-sentiment_analysis-q4_K_M 5.184 0.024 -2.277 0.024
llama3_8b-instruct-q5_K_M vs llama3_8b-instruct-sentiment_analysis-q5_K_M 18.747 0.0 -4.33 0.0
llama3_8b-instruct-q8_0 vs llama3_8b-instruct-sentiment_analysis-q8_0 6.821 0.01 -2.612 0.01
mistral_7b-instruct-fp16 vs llama3_8b-instruct-fp16 1.558 0.214 1.248 0.214
mistral_7b-instruct-q4_K_M vs llama3_8b-instruct-q4_K_M 0.372 0.542 0.61 0.542
mistral_7b-instruct-q5_K_M vs llama3_8b-instruct-q5_K_M 4.425 0.037 2.104 0.037
mistral_7b-instruct-q8_0 vs llama3_8b-instruct-q8_0 0.712 0.4 0.844 0.4
dolphin-mistral_7b-v2.8-fp16 vs mistral_7b-instruct-fp16 0.007 0.935 0.081 0.935
@seandearnaley
seandearnaley / model_metrics.csv
Last active May 21, 2024 19:48
Model Metrics
Model Name Article Key Inference Rate (s) Valid JSON Rate Sentiment Variance Mean Sentiment Mean Confidence
llama3_8b-instruct-sentiment_analysis-q4_K_M 63f711be 5.41 1.0 0.01 0.33 0.78
llama3_8b-instruct-sentiment_analysis-q4_K_M 1ed47412 1.9 1.0 0.02 -0.14 0.6
llama3_8b-instruct-sentiment_analysis-q4_K_M 5852d8b8 2.3 1.0 0.01 -0.53 0.8
llama3_8b-instruct-sentiment_analysis-q4_K_M 6f5003f2 4.02 1.0 0.06 0.56 0.81
llama3_8b-instruct-sentiment_analysis-q4_K_M 7be9fbd1 4.4 1.0 0.0 0.6 0.8
llama3_8b-instruct-sentiment_analysis-q4_K_M 7fe80a1c 2.68 1.0 0.0 0.6 0.8
llama3_8b-instruct-sentiment_analysis-q4_K_M bcdd3918 3.44 1.0 0.01 0.04 0.64
llama3_8b-instruct-sentiment_analysis-q4_K_M 302c8a97 1.51 1.0 0.0 0.0 0.71
llama3_8b-instruct-sentiment_analysis-q4_K_M 7be44b2c 1.54 1.0 0.26 -0.49 0.71
@seandearnaley
seandearnaley / whispercc.py
Created April 20, 2023 01:55
Whisper.cpp Transcribe Example with Execution Time
"""WhisperCC binding module"""
import time
from pathlib import Path
from whispercpp import Whisper
from app.audio_tools import NdArray, convert_to_float_array, decode_audio
class WhisperTranscriber:
@seandearnaley
seandearnaley / main.py
Created April 20, 2023 01:53
Whisper YouTube Transcriber (GPU)
"""Example of using the Whisper library to detect the language and transcribe."""
import os
import sys
import time
from typing import Optional
import torch
import whisper
from pytube import YouTube
@seandearnaley
seandearnaley / whisper_avg_benchmarks.csv
Created April 20, 2023 01:39
Benchmark Results + Transcriptions for Whisper Transcriptions
We can make this file beautiful and searchable if this error is corrected: Unclosed quoted field in line 2.
Model,"CPU: Macbook M1 Pro (16gb, Nov 2020)","CPU: Mac M2 Pro (Mini 32gb, Jan 2023)","CPU: Windows 11 via Ubuntu 22 (32gb, Oct 2018)","GPU: RTX 2080TI (12gb VRAM, Sep 2018)","GPU: RTX 4090 (24gb, Oct 2022)","GPU: A100 (80GB 16 vCPU 62 GB RAM, Jun 2020)",Transcription
tiny.en,0:00:22.794,0:00:21.477,0:00:45.639,0:00:32.500,0:00:22.748,0:00:18.552,"One of the biggest mysteries of our existence is also one of the biggest mysteries of physics, time. We experienced time as passing with a special moment that we call ""now"". Now you're watching this video. Half an hour ago you were doing something else. Whatever you did, there's no way to change it. And what you will do in half an hour is up to you. At least that's how we perceive time. But what physics tells us about time is very different from our perception. The person who figured this out was none other than Albert Einstein. I know that guy again turns out he kind of knew it all. What did Einstein teach us about the past, the present and the future? That's what
@seandearnaley
seandearnaley / models.csv
Last active September 4, 2023 13:55
Table of LLMs
Model Developer Context Window Size (Tokens) Rough Number of Words
BERT-Large Google 512 128-256
RoBERTa-Large Facebook AI 512 128-256
GPT-2 OpenAI 1024 256-512
GPT-3 OpenAI 2048 512-1024
T5-11B Google 2048 512-1024
LLaMA Meta 2048 512-1024
GPT 3.5 (4k) OpenAI 4096 1024-2048
GPT 3.5 (16k) OpenAI 16384 4096-8192
GPT-4 (8k) OpenAI 8192 2048-4096
@seandearnaley
seandearnaley / JungViz1.yml
Last active May 28, 2022 17:12
Simplified Disco Diffusion YAML for Video Input
# Models
RN101: false
RN50: true
RN50x16: false
RN50x4: false
RN50x64: false
ViTB16: true
ViTB32: true
ViTL14: false
ViTL14_336: false
@seandearnaley
seandearnaley / syntheticAndroid2D-12.yml
Created May 26, 2022 13:53
Simplified Disco Diffusion Template for Human Being - Magma
# Models
RN101: false
RN50: true
RN50x16: false
RN50x4: false
RN50x64: false
ViTB16: true
ViTB32: true
ViTL14: false
ViTL14_336: false