Run falcon-40b-instruct in conversational mode

falcon_40b_instruct_chat.py

import torch
import transformers
from transformers import AutoTokenizer, AutoModelForCausalLM


model = "tiiuae/falcon-40b-instruct"

tokenizer = AutoTokenizer.from_pretrained(model, padding_side="left")

pipeline = transformers.pipeline(
    "conversational",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)

# Without setting newline as bad word, the model sometimes would answer a question,
# then generate a question, and answer it as well, all withen the same response from
# the model for one user question. While this is a workaround, as this means model
# is destined to fail in many cases, it works for this simple use case
bad_list = tokenizer(["\n"], add_special_tokens=False)

# The simplest version of prompt that worked for me was to instruct the model, then
# Add a sample question, and answer it on behalf of the model
INITIAL_USER = (
    "You are an artificial intelligence assistant. The assistant gives helpful,"
    " detailed, and polite answers to the user's questions. Your only source of"
    ' knowledge is following article between quote marks "The number of buildings in'
    " the neighborhood is 80 buildings. It is a new neighborhood with modern"
    " facilities. Each building has 24 appartments. The streets are wide to reflect"
    " good on the drivers. The sides of the roads are all planted with palm trees. The"
    " buildings are of the colours brown, orange, and off-white. The appartment is"
    ' expected to cost as little as 400,000 AED."\nHow many buildings are there?'
)
INITIAL_AI = "There are 80 new buildings in the new neighborhood"

conversation = transformers.Conversation(INITIAL_USER)
conversation.mark_processed()
conversation.append_response(INITIAL_AI)

print(f"[USER]: {INITIAL_USER}")
print(f"[ AI ]: {INITIAL_AI}")

for msg in [
    "What is the total number of flats in the new neighborhood?",
    "What colours are the building?",
    "Is red one of the colours used for the buildings?",
]:
    print(f"[USER]: {msg}")
    conversation.add_user_input(msg)
    conversation = pipeline(
        conversation,
        top_k=10,
        temperature=0.2,
        # Limit is bound to number of questions; Adding more questions require
        # increasing this value
        max_length=400,
        bad_words_ids=bad_list.input_ids,
    )
    print(f"[ AI ]: {conversation.generated_responses[-1]}")

output

[USER]: You are an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. Your only source of knowledge is following article between quote marks "The number of buildings in the neighborhood is 80 buildings. It is a new neighborhood with modern facilities. Each building has 24 appartments. The streets are wide to reflect good on the drivers. The sides of the roads are all planted with palm trees. The buildings are of the colours brown, orange, and off-white. The appartment is expected to cost as little as 400,000 AED."
How many buildings are there?
[ AI ]: There are 80 new buildings in the new neighborhood
[USER]: What is the total number of flats in the new neighborhood?
[ AI ]: The total number of flats in the new neighborhood is 1,920. (80 buildings * 24 flats per building)
[USER]: What colours are the building?
[ AI ]: The buildings are of the colours brown, orange, and off-white.
[USER]: Is red one of the colours used for the buildings?
[ AI ]: The article does not mention red as one of the colours used for the buildings.

Takeaways:

• Model falcon-40b-instruct can be used as-is with many simple tasks.
• In other experiments it proved to have better knowledge in areas I found vicuna to fail (e.g. Geography).
• The model did math (Q1) which I didn't expect to be done well, given that even more established models struggle with it at some events.
• We are witnessing the future!