Skip to content

Instantly share code, notes, and snippets.

@derwiki

derwiki/199.log

Last active June 21, 2023 20:28
Show Gist options
  • Save derwiki/99079de4cfcb4f196b1ca561b3d77a9f to your computer and use it in GitHub Desktop.
Save derwiki/99079de4cfcb4f196b1ca561b3d77a9f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
199.log
➜ hallucinations python3 analyze.py
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":{"value":75,"units":"F"}}
1: {"temperature":75,"unit":"F"}
1: {"temperature":75,"unit":"F"}
1: {"temperature":{"value":75,"units":"F"}}
1: {"temperature":75,"units":"F"}
1: {"current_temperature":{"value":75,"unit":"F"}}
1: {"Temperature":{"Fahrenheit":75}}
1: {"reading":"75F"}
1: {"temperature":75,"unit":"F"}
1: {"temperature":75,"unit":"F","timestamp":"2021-02-10T14:30:00Z"}
1: {"temperature":{"degrees":75,"unit":"F"}}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"temperature":75,"unit":"Fahrenheit"}
1: { "temperature":{ "value":75, "unit":"Fahrenheit" }}
1: {"temperature":75}
1: {"temperature":{"current":75,"unit":"F"}}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":75,"unit":"F","location":"Unknown"}
2: {"temperature":{"value":75,"unit":"F"}}
2: {"temperature":{"value":75,"unit":"F"}}
2: {"temperature":75,"unit":"F"}
2: {"temperature":75}
3: {"temperature":75,"unit":"F"}
4: {"temperature":{"value":"75","unit":"F"}}
5: {"temperature":"75F"}
5: {"temperature":"75F"}
6: {"temperature":75,"unit":"F"}
6: {"temperature":{"value":75,"unit":"F"}}
6: {"temperature":"75F"}
8: {"temperature":{"value":75,"unit":"Fahrenheit"}}
9: {"temperature":75}
10: {"temperature":{"value":75,"unit":"F"}}
20: {"temperature":75,"unit":"F"}
90: {"temperature":{"value":75,"unit":"F"}}
199
Raw
299.log
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":{"value":75,"units":"F"}}
1: {"temperature":75,"unit":"F"}
1: {"temperature":75,"unit":"F"}
1: { "temperature":{ "value":75, "unit":"F" }}
1: {"temperature":75,"units":"F"}
1: {"current_temperature":{"value":75,"unit":"F"}}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"Temperature":{"Fahrenheit":75}}
1: {"reading":"75F"}
1: {"temperature":75,"unit":"F"}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"temperature":75,"unit":"F","timestamp":"2021-02-10T14:30:00Z"}
1: {"temperature":{"degrees":75,"unit":"F"}}
1: {"temperature":75,"units":"F"}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: { "temperature":{ "value":75, "unit":"Fahrenheit" }}
1: {"temperature":75,"temperature_unit":"F"}
1: {"temperature":75}
1: {"temperature":{"current":75,"unit":"F"}}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":75,"unit":"Fahrenheit"}
1: {"temperature":75,"unit":"F","location":"Unknown"}
1: {"temperature":{"value":75,"unit":"F"}}
2: {"temperature":"75F"}
2: {"temperature":{"value":75,"unit":"F"}}
2: {"temperature":{"value":75,"unit":"F"}}
2: {"temperature":75,"unit":"Fahrenheit"}
2: {"temperature":75}
3: {"temperature":{"value":75,"units":"F"}}
4: {"temperature":75,"unit":"F"}
4: {"temperature":75,"unit":"F"}
5: {"temperature":{"value":"75","unit":"F"}}
7: {"temperature":"75F"}
8: {"temperature":"75F"}
9: {"temperature":75,"unit":"F"}
9: {"temperature":{"value":75,"unit":"F"}}
11: {"temperature":75}
14: {"temperature":{"value":75,"unit":"F"}}
16: {"temperature":{"value":75,"unit":"Fahrenheit"}}
17: {"temperature":"75F"}
26: {"temperature":75,"unit":"F"}
131: {"temperature":{"value":75,"unit":"F"}}
299
Raw
399.log
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":"75F"}
1: {"temperature":{"value":75,"units":"F"}}
1: {"temperature":75,"unit":"F"}
1: {"temperature":{"degrees":75,"unit":"F"}}
1: { "temperature":{ "value":75, "unit":"F" }}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":75,"units":"F"}
1: {"current_temperature":{"value":75,"unit":"F"}}
1: {"Temperature":{"Fahrenheit":75}}
1: {"reading":"75F"}
1: {"temperature":75,"unit":"F"}
1: {"temperature":{"value":"75","unit":"Fahrenheit"}}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"temperature":"75F"}
1: {"Temperature":{"Value":75,"Unit":"F"}}
1: {"temperature":75,"unit":"F","timestamp":"2021-02-10T14:30:00Z"}
1: {"location":"NewYork","temperature":{"value":75,"unit":"F"}}
1: {"temperature":{"degrees":75,"unit":"F"}}
1: {"temperature":75,"units":"F"}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: { "temperature":{ "value":75, "unit":"Fahrenheit" }}
1: {"temperature":75,"temperature_unit":"F"}
1: {"temperature":75}
1: {"temperature":{"current":75,"unit":"F"}}
1: {"temperature":"75F"}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":75,"unit":"Fahrenheit"}
1: {"temperature":75,"unit":"F","location":"Unknown"}
1: {"temperature":{"value":75,"unit":"F"}}
2: {"temperature":"75F"}
2: {"temperature":{"value":75,"unit":"F"}}
2: {"temperature":75,"unit":"F"}
2: {"temperature":{"value":75,"unit":"Fahrenheit"}}
2: {"temperature":75}
3: {"temperature":{"value":75,"unit":"F"}}
4: {"temperature":{"value":75,"units":"F"}}
4: {"temperature":75,"unit":"Fahrenheit"}
5: {"temperature":75,"unit":"F"}
6: {"temperature":75,"unit":"F"}
6: {"temperature":{"value":"75","unit":"F"}}
8: {"temperature":"75F"}
9: {"temperature":"75F"}
10: {"temperature":{"value":75,"unit":"F"}}
12: {"temperature":75}
13: {"temperature":75,"unit":"F"}
16: {"temperature":{"value":75,"unit":"F"}}
19: {"temperature":{"value":75,"unit":"Fahrenheit"}}
23: {"temperature":"75F"}
38: {"temperature":75,"unit":"F"}
181: {"temperature":{"value":75,"unit":"F"}}
398
Raw
497.log
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":"75F"}
1: {"temperature":{"value":75,"units":"F"}}
1: {"temperature":75,"unit":"F"}
1: {"temperature":{"degrees":75,"unit":"F"}}
1: { "temperature":{ "value":75, "unit":"F" }}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":75,"units":"F"}
1: {"current_temperature":{"value":75,"unit":"F"}}
1: {"temperature":75,"units":"Fahrenheit"}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"Temperature":{"Fahrenheit":75}}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"reading":"75F"}
1: {"temperature":75,"unit":"F"}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"temperature":"75F"}
1: {"Temperature":{"Value":75,"Unit":"F"}}
1: {"temperature":75,"unit":"F","timestamp":"2021-02-10T14:30:00Z"}
1: {"location":"NewYork","temperature":{"value":75,"unit":"F"}}
1: {"temperature":{"degrees":75,"unit":"F"}}
1: {"Temperature":{"value":75,"unit":"F"}}
1: {"temperature":75,"units":"F"}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: { "temperature":{ "value":75, "unit":"Fahrenheit" }}
1: {"temperature":75,"temperature_unit":"F"}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":75}
1: {"temperature":75,"unit":"F"}
1: {"temperature":{"current":75,"unit":"F"}}
1: {"temperature":"75F"}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":75,"unit":"Fahrenheit"}
1: {"temperature":75,"unit":"F","location":"Unknown"}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
2: {"temperature":"75F"}
2: {"temperature":{"unit":"F","value":75}}
2: {"temperature":{"value":75,"unit":"F"}}
2: {"temperature":{"value":"75","unit":"Fahrenheit"}}
2: {"temperature":{"value":75,"unit":"Fahrenheit"}}
2: ```{"temperature":{"value":75,"unit":"F"}}```
2: {"temperature":75}
3: {"temperature":{"value":75,"unit":"F"}}
3: {"temperature":75,"unit":"F"}
6: {"temperature":75,"unit":"F"}
6: {"temperature":{"value":75,"units":"F"}}
6: {"temperature":75,"unit":"Fahrenheit"}
8: {"temperature":{"value":"75","unit":"F"}}
9: {"temperature":"75F"}
10: {"temperature":75,"unit":"F"}
10: {"temperature":"75F"}
10: {"temperature":{"value":75,"unit":"F"}}
14: {"temperature":75}
14: {"temperature":75,"unit":"F"}
17: {"temperature":{"value":75,"unit":"F"}}
23: {"temperature":{"value":75,"unit":"Fahrenheit"}}
33: {"temperature":"75F"}
51: {"temperature":75,"unit":"F"}
222: {"temperature":{"value":75,"unit":"F"}}
497
Raw
99.log
➜ hallucinations python3 analyze.py
1: {"temperature":{"value":75,"units":"F"}}
1: {"temperature":75,"unit":"F"}
1: {"temperature":{"value":75,"units":"F"}}
1: {"temperature":75,"units":"F"}
1: {"current_temperature":{"value":75,"unit":"F"}}
1: {"reading":"75F"}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"temperature":{"degrees":75,"unit":"F"}}
1: {"temperature":{"value":75,"unit":"Fahrenheit"}}
1: {"temperature":75,"unit":"F"}
1: { "temperature":{ "value":75, "unit":"Fahrenheit" }}
1: {"temperature":{"current":75,"unit":"F"}}
1: {"temperature":{"value":75,"unit":"F"}}
1: {"temperature":75,"unit":"F","location":"Unknown"}
2: {"temperature":75,"unit":"F"}
2: {"temperature":{"value":"75","unit":"F"}}
3: {"temperature":"75F"}
3: {"temperature":"75F"}
3: {"temperature":"75F"}
3: {"temperature":75,"unit":"F"}
5: {"temperature":{"value":75,"unit":"F"}}
6: {"temperature":75}
7: {"temperature":{"value":75,"unit":"F"}}
9: {"temperature":75,"unit":"F"}
42: {"temperature":{"value":75,"unit":"F"}}
99
Raw
analyze.py
import os
# Initialize an empty dictionary for the histogram
histogram = {}
def compute_hash(file_path):
with open(file_path, "r") as f:
return f.read()[len("\nA(gpt-3.5-turbo): "):]
# List all files in the current directory
for filename in os.listdir('.'):
# Check if the filename starts with 'test-' and ends with '.txt'
if filename.startswith('test-') and filename.endswith('.txt'):
# Compute hash of the file
file_contents = compute_hash(filename)
# If the hash is already in the histogram, increment the count
if file_contents in histogram:
histogram[file_contents] += 1
# Else, add the hash to the histogram with a count of 1
else:
histogram[file_contents] = 1
# Print the histogram
for file_contents, cnt in sorted(histogram.items(), key=lambda item: item[1]):
formatted_contents = file_contents.replace("\n", " ").replace(" ", "")
print(f"{cnt}: {formatted_contents}")
print(sum(histogram.values()))
Raw
run.sh
PROMPT="Construct a JSON example that expresses the temperature is 75F"
for i in `seq 100`; do
TIMESTAMP=$(ruby -e 'puts (Time.now.to_f * 1000).to_i')
(echo $TIMESTAMP ; echo $PROMPT | LLM_MODELS=gpt-3.5-turbo /Users/adam/src/askgpt/main --skip-history > "test-$TIMESTAMP.txt" ) &
sleep 0.3
done
@derwiki
Copy link
Author

derwiki commented Jun 21, 2023 

I did 5 runs and the unique result set got larger each time, and the most popular response got less popular each time.
Prompt was: Construct a JSON example that expresses the temperature is 75F
here's a sample of the histogram with 498 completions
2: {"temperature":{"value":"75","unit":"Fahrenheit"}}
2: {"temperature":{"value":75,"unit":"Fahrenheit"}}
2: ```{"temperature":{"value":75,"unit":"F"}}```
2: {"temperature":75}
3: {"temperature":{"value":75,"unit":"F"}}
3: {"temperature":75,"unit":"F"}
6: {"temperature":75,"unit":"F"}
6: {"temperature":{"value":75,"units":"F"}}
6: {"temperature":75,"unit":"Fahrenheit"}
8: {"temperature":{"value":"75","unit":"F"}}
9: {"temperature":"75F"}
10: {"temperature":75,"unit":"F"}
10: {"temperature":"75F"}
10: {"temperature":{"value":75,"unit":"F"}}
14: {"temperature":75}
14: {"temperature":75,"unit":"F"}
17: {"temperature":{"value":75,"unit":"F"}}
23: {"temperature":{"value":75,"unit":"Fahrenheit"}}
33: {"temperature":"75F"}
51: {"temperature":75,"unit":"F"}
222: {"temperature":{"value":75,"unit":"F"}}
497

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment