LLM notes

llm.md

Notebook

Theory

LLM

• Build LLM from scratch in Python using createllm package - https://pythonscholar.com/build-a-large-language-model-from-scratch/
• Private LLM using databricks-dolly-15k (approximately 15,000 instruction/response fine-tuning records) - https://www.leewayhertz.com/build-private-llm/
• The Emergence Of Large Language Model (LLM) API Build Frameworks - https://cobusgreyling.medium.com/the-emergence-of-large-language-model-llm-api-build-frameworks-78d83d68eeda
• Corpus size and LLM - https://genai.stackexchange.com/q/613/2269
• Restrict LLM responses to specific dataset - https://genai.stackexchange.com/q/167/2269
• Fine tuning the LLaVA Vision LLM on AWS - https://medium.com/@mr.sean.ryan/fine-tuning-the-llava-vision-llm-on-aws-2ba46b7dcec9
• Time-LLM: Reprogram an LLM for Time Series Forecasting - https://towardsdatascience.com/time-llm-reprogram-an-llm-for-time-series-forecasting-e2558087b8ac
• Running Your Very Own Local LLM - https://yc.prosetech.com/running-your-very-own-local-llm-6d4db99c0611
• What are 1-bit LLMs? - https://medium.com/data-science-in-your-pocket/what-are-1-bit-llms-3f2ae4b40fdf
• Implementing the Transformer Encoder from Scratch in TensorFlow and Keras - https://machinelearningmastery.com/implementing-the-transformer-encoder-from-scratch-in-tensorflow-and-keras/
• Unleashing the Power of Language Models: A Deep Dive into Language Foundation Model Tuning Strategies - https://ai.plainenglish.io/unleashing-the-power-of-language-models-a-deep-dive-into-language-foundation-model-tuning-4f1e96be7ddf
• Understanding Large Language Models - Words vs Tokens - https://kelvin.legal/understanding-large-language-models-words-versus-tokens/
• Hands-On LangChain for LLM Applications Development: Output Parsing - https://pub.towardsai.net/hands-on-langchain-for-llm-applications-development-output-parsing-876354434462
• What are foundation models? - https://research.ibm.com/blog/what-are-foundation-models
• Your Guide to the LLM Ecosystem - https://pub.aimind.so/your-guide-to-the-llm-ecosystem-f67826c84be8
• Large Language Model (LLM) Stack — Version 5 - https://cobusgreyling.medium.com/large-language-model-llm-stack-version-5-5a9306870e7f
• Building a Million-Parameter LLM from Scratch Using Python - https://levelup.gitconnected.com/building-a-million-parameter-llm-from-scratch-using-python-f612398f06c2
• Deploy your own Open-Source Language Model: A Comprehensive Guide - https://blog.zhaw.ch/artificial-intelligence/2023/04/20/deploy-your-own-open-source-language-model/
• Best practices for building LLMs - https://stackoverflow.blog/2024/02/07/best-practices-for-building-llms/
• Understanding Encoder And Decoder LLMs - https://magazine.sebastianraschka.com/p/understanding-encoder-and-decoder
• How does the (decoder-only) transformer architecture work? - https://ai.stackexchange.com/q/40179/75530
• LLM Agents - https://www.promptingguide.ai/research/llm-agents
• https://python.langchain.com/docs/use_cases/sql/
• Building an LLM Stack, Part 1: Implementing Encoders and Decoders - https://deepgram.com/learn/building-an-llm-stack-1-implementing-encoders-and-decoders

LLM metrics

• The Secrets of Large Language Models Parameters
• Model Cards for Model Reporting
• A simple way to create ML Model Cards in Python
• HF Model Card writing tool
• Model Card Guidebook

LLM Customizations

• Mastering LLM Techniques: Customization
• The architecture of today’s LLM applications

LLM Evaluation

• How to choose your LLM - https://community.aws/posts/how-to-choose-your-llm
• How I selected my GenAI and Large Language Model (LLM) Platform - https://medium.com/@nayan.j.paul/how-i-selected-my-genai-and-large-language-model-llm-platform-cfe6da358b25
• LLM Evaluation: Benchmarking Performance and Metrics - https://aisera.com/blog/llm-evaluation/
• LLM Evaluation: Everything You Need To Run, Benchmark LLM Evals - https://arize.com/blog-course/llm-evaluation-the-definitive-guide/
• W&B Prompts - https://docs.wandb.ai/guides/prompts
• Evaluating Large Language Model (LLM) systems: Metrics, challenges, and best practices - https://medium.com/data-science-at-microsoft/evaluating-llm-systems-metrics-challenges-and-best-practices-664ac25be7e5
• Decoding LLM Performance: A Guide to Evaluating LLM Applications - https://amagastya.medium.com/decoding-llm-performance-a-guide-to-evaluating-llm-applications-e8d7939cafce
• Amazon Bedrock Model Evaluation - https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation.html
• How to Evaluate LLMs: A Complete Metric Framework - https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/how-to-evaluate-llms-a-complete-metric-framework/
• MLflow LLM Evaluate - https://mlflow.org/docs/latest/llms/llm-evaluate/index.html

LLM fine-tuning

• Comprehensive Guide for Finetuning - https://genai.stackexchange.com/q/564/2269
• Fine tuning pipeline for open-source LLMs - https://paulabartabajo.substack.com/p/lets-fine-tune-an-open-source-llm
• When Should You Fine-Tune LLMs? - https://towardsdatascience.com/when-should-you-fine-tune-llms-2dddc09a404a
• Large Language Models: to Fine-tune or not to Fine-tune? - https://www.ml6.eu/blogpost/fine-tuning-large-language-models
• Fine tuning: what is it good for? - https://community.openai.com/t/fine-tuning-what-is-it-good-for/428080?u=nsubrahm

RAG

• What is RAG? - https://aws.amazon.com/what-is/retrieval-augmented-generation/
• Retrieval Augmented Generation (RAG) - https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-customize-rag.html
• Retrieval augmented generation: Keeping LLMs relevant and current - https://stackoverflow.blog/2023/10/18/retrieval-augmented-generation-keeping-llms-relevant-and-current/
• Retrieval Augmented Generation (RAG) - https://www.promptingguide.ai/techniques/rag
• Build Industry-Specific LLMs Using Retrieval Augmented Generation - https://towardsdatascience.com/build-industry-specific-llms-using-retrieval-augmented-generation-af9e98bb6f68
• RAG Vs Fine tuning Vs Both - https://medium.com/@ramprasathsee/rag-vs-fine-tuning-vs-both-3cb25857d921
• GraphRAG: Unlocking LLM discovery on narrative private data - https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
• What is a Vector Database? - https://zilliz.com/learn/what-is-vector-database
• Knowledge Bases for Amazon Bedrock - https://aws.amazon.com/bedrock/knowledge-bases/
• Retrieval Augmented Generation (RAG) Architecture based on AWS - https://shabarish033.medium.com/retrieval-augmented-generation-rag-architecture-based-on-aws-fc449b708b04

GenAI stack

• What every investor should know about the GenAI tech stack - https://raphaelledornano.medium.com/what-every-investor-should-know-about-the-genai-tech-stack-813cc04a5249
• Understanding the GenAI Tech Stack : Part 3 — Foundation Models - https://raphaelledornano.medium.com/understanding-the-genai-tech-stack-part-3-foundation-models-8c3a9ad2c49b
• Understanding the GenAI Tech Stack : Part 4 — Application Models - https://raphaelledornano.medium.com/understanding-the-genai-tech-stack-part-4-application-models-8a92fc30e3ef
• GenAI Dev Stack, LLMOps & Vector Databases! - https://www.linkedin.com/pulse/genai-dev-stack-llmops-vector-databases-pavan-belagatti-wmcmc/
• GenAI Stack Walkthrough: Behind the Scenes With Neo4j, LangChain, and Ollama in Docker - https://neo4j.com/developer-blog/genai-app-how-to-build/
• Generative AI Tech Stack: A Complete Guide - https://flyaps.com/blog/generative-ai-tech-stack-a-complete-guide/
• Understanding Generative AI: A Tech Stack Breakdown - https://www.orioninc.com/blog/understanding-generative-ai-a-tech-stack-breakdown/
• The LLM App Stack — 2024 - https://medium.com/plain-simple-software/the-llm-app-stack-2024-eac28b9dc1e7

Prompt Engineering

• What is Prompt Engineering? - https://aws.amazon.com/what-is/prompt-engineering/
• Prompt engineering for foundation models - https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-customize-prompt-engineering.html
• What is prompt engineering? - https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-prompt-engineering.html
• Prompt engineering guidelines - https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-engineering-guidelines.html
• Best Practices for Prompt Engineering with Amazon CodeWhisperer - https://aws.amazon.com/blogs/devops/best-practices-for-prompt-engineering-with-amazon-codewhisperer/

Implementations - AWS

Fine-tuning

• Fine-tune Llama 2 using QLoRA and Deploy it on Amazon SageMaker with AWS Inferentia2 - https://aws.amazon.com/blogs/machine-learning/fine-tune-llama-2-using-qlora-and-deploy-it-on-amazon-sagemaker-with-aws-inferentia2/
• Deploying and fine tuning LLMs in AWS for job listing summarisation - https://medium.com/stepstone-tech/deploying-and-fine-tuning-llms-in-aws-for-job-listing-summarisation-c3e16e00aab3
• Instruction fine-tuning for FLAN T5 XL with Amazon SageMaker Jumpstart - https://aws.amazon.com/blogs/machine-learning/instruction-fine-tuning-for-flan-t5-xl-with-amazon-sagemaker-jumpstart/
• Fine-tune your LLM with AWS Sagemaker: build the best D&D assistant with generative AI - https://medium.com/@jeremyarancio/fine-tune-your-llm-with-aws-sagemaker-build-the-best-d-d-assistant-with-generative-ai-e78b507ae575
• Fine-Tuning LLMs with Amazon Bedrock - https://towardsaws.com/fine-tuning-llms-with-amazon-bedrock-739757479f47
• Fast and cost-effective LLaMA 2 fine-tuning with AWS Trainium - https://aws.amazon.com/blogs/machine-learning/fast-and-cost-effective-llama-2-fine-tuning-with-aws-trainium/
• Customize models in Amazon Bedrock with your own data using fine-tuning and continued pre-training - https://aws.amazon.com/blogs/aws/customize-models-in-amazon-bedrock-with-your-own-data-using-fine-tuning-and-continued-pre-training/
• Fine-tune LLaMA 2 (7-70B) on Amazon SageMaker - https://www.philschmid.de/sagemaker-llama2-qlora
• Fine-tune Falcon 7B and other LLMs on Amazon SageMaker with @remote decorator - https://aws.amazon.com/blogs/machine-learning/fine-tune-falcon-7b-and-other-llms-on-amazon-sagemaker-with-remote-decorator/
• Fine-tune and Deploy Mistral 7B with Amazon SageMaker JumpStart - https://aws.amazon.com/blogs/machine-learning/fine-tune-and-deploy-mistral-7b-with-amazon-sagemaker-jumpstart/
• Domain-adaptation Fine-tuning of Foundation Models in Amazon SageMaker JumpStart on Financial data - https://aws.amazon.com/blogs/machine-learning/domain-adaptation-fine-tuning-of-foundation-models-in-amazon-sagemaker-jumpstart-on-financial-data/
• AWS performs fine-tuning on a Large Language Model (LLM) to classify toxic speech for a large gaming company - https://aws.amazon.com/blogs/machine-learning/aws-performs-fine-tuning-on-a-large-language-model-llm-to-classify-toxic-speech-for-a-large-gaming-company/
• Interactively fine-tune Falcon-40B and other LLMs on Amazon SageMaker Studio notebooks using QLoRA - https://aws.amazon.com/blogs/machine-learning/interactively-fine-tune-falcon-40b-and-other-llms-on-amazon-sagemaker-studio-notebooks-using-qlora/
• Preprocess and fine-tune LLMs quickly and cost-effectively using Amazon EMR Serverless and Amazon SageMaker - https://aws.amazon.com/blogs/big-data/preprocess-and-fine-tune-llms-quickly-and-cost-effectively-using-amazon-emr-serverless-and-amazon-sagemaker/
• Model management for LoRA fine-tuned models using Llama2 and Amazon SageMaker - https://aws.amazon.com/blogs/machine-learning/model-management-for-lora-fine-tuned-models-using-llama2-and-amazon-sagemaker/
• Fine-Tuning an Open Source LLM in Amazon SageMaker with W&B - https://wandb.ai/capecape/aws_llm_workshop/reports/Fine-Tuning-an-Open-Source-LLM-in-Amazon-SageMaker-with-W-B--Vmlldzo1Njk4MDc1
• fine-tuning-llm-with-domain-knowledge - https://github.com/aws-samples/fine-tuning-llm-with-domain-knowledge
• Llama 2 foundation models from Meta are now available in Amazon SageMaker JumpStart - https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/
• Build scalable and serverless RAG workflows with a vector engine for Amazon OpenSearch Serverless and Amazon Bedrock Claude models - https://aws.amazon.com/blogs/big-data/build-scalable-and-serverless-rag-workflows-with-a-vector-engine-for-amazon-opensearch-serverless-and-amazon-bedrock-claude-models/
• Train Llama2 with AWS Trainium on Amazon EKS - https://aws.amazon.com/blogs/containers/train-llama2-with-aws-trainium-on-amazon-eks/

LLMOps

• FMOps/LLMOps: Operationalize generative AI and differences with MLOps - https://aws.amazon.com/blogs/machine-learning/fmops-llmops-operationalize-generative-ai-and-differences-with-mlops/
• Training large language models on Amazon SageMaker: Best practices - https://aws.amazon.com/blogs/machine-learning/training-large-language-models-on-amazon-sagemaker-best-practices/
• Evaluate large language models for quality and responsibility - https://aws.amazon.com/blogs/machine-learning/evaluate-large-language-models-for-quality-and-responsibility/
• Operationalize LLM Evaluation at Scale using Amazon SageMaker Clarify and MLOps services - https://aws.amazon.com/blogs/machine-learning/operationalize-llm-evaluation-at-scale-using-amazon-sagemaker-clarify-and-mlops-services/
• Improving your LLMs with RLHF on Amazon SageMaker - https://aws.amazon.com/blogs/machine-learning/improving-your-llms-with-rlhf-on-amazon-sagemaker/

RAG

• Serverless Retrieval Augmented Generation (RAG) on AWS - https://community.aws/content/2d1B5srtVqbVYnlm9ixKNJf4p1M/serverless-retrieval-augmented-generation-rag-on-aws?lang=en
• Build a secure enterprise application with Generative AI and RAG using Amazon SageMaker JumpStart - https://aws.amazon.com/blogs/machine-learning/build-a-secure-enterprise-application-with-generative-ai-and-rag-using-amazon-sagemaker-jumpstart/
• Mitigate hallucinations through Retrieval Augmented Generation using Pinecone vector database & Llama-2 from Amazon SageMaker JumpStart - https://aws.amazon.com/blogs/machine-learning/mitigate-hallucinations-through-retrieval-augmented-generation-using-pinecone-vector-database-llama-2-from-amazon-sagemaker-jumpstart/
• Simplify access to internal information using Retrieval Augmented Generation and LangChain Agents - https://aws.amazon.com/blogs/machine-learning/simplify-access-to-internal-information-using-retrieval-augmented-generation-and-langchain-agents/
• Question answering using Retrieval Augmented Generation with foundation models in Amazon SageMaker JumpStart - https://aws.amazon.com/blogs/machine-learning/question-answering-using-retrieval-augmented-generation-with-foundation-models-in-amazon-sagemaker-jumpstart/
• Knowledge Bases for Amazon Bedrock now supports hybrid search - https://aws.amazon.com/blogs/machine-learning/knowledge-bases-for-amazon-bedrock-now-supports-hybrid-search/
• Implementing RAG with Amazon Bedrock and AWS Lambda - https://heeki.medium.com/implementing-rag-with-amazon-bedrock-and-aws-lambda-db1476089b23
• Use RAG for drug discovery with Knowledge Bases for Amazon Bedrock - https://aws.amazon.com/blogs/machine-learning/use-rag-for-drug-discovery-with-knowledge-bases-for-amazon-bedrock/
• Boosting RAG-based intelligent document assistants using entity extraction, SQL querying, and agents with Amazon Bedrock - https://aws.amazon.com/blogs/machine-learning/boosting-rag-based-intelligent-document-assistants-using-entity-extraction-sql-querying-and-agents-with-amazon-bedrock/
• Create RAG AI on AWS with OpenSearch - https://caylent.com/blog/building-a-rag-with-open-search-serverless-and-lang-chain
• Develop advanced generative AI chatbots by using RAG and ReAct prompting - https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/develop-advanced-generative-ai-chatbots-by-using-rag-and-react-prompting.html
• Build and scale generative AI applications with Amazon Bedrock - https://workshops.aws/categories/Prompt%20Engineering
• Generative AI - Prompt Engineering - https://github.com/build-on-aws/generative-ai-prompt-engineering

LangChain

• Quickly build high-accuracy Generative AI applications on enterprise data using Amazon Kendra, LangChain, and large language models - https://aws.amazon.com/blogs/machine-learning/quickly-build-high-accuracy-generative-ai-applications-on-enterprise-data-using-amazon-kendra-langchain-and-large-language-models/

Others

• Build a contextual chatbot application using Knowledge Bases for Amazon Bedrock - https://aws.amazon.com/blogs/machine-learning/build-a-contextual-chatbot-application-using-knowledge-bases-for-amazon-bedrock/
• Building with Amazon Bedrock and LangChain - https://catalog.workshops.aws/building-with-amazon-bedrock/en-US
• Empowering Software Conversations with GenAI - https://medium.com/@khandelwal.praful/empowering-software-conversations-with-genai-2cea89e80a86
• Generative AI Application Builder on AWS - https://aws.amazon.com/solutions/implementations/generative-ai-application-builder-on-aws/
• Building a Powerful Question Answering Bot with Amazon SageMaker JumpStart, Amazon OpenSearch, Streamlit, and LangChain: A Step-by-Step Guide- https://github.com/aws-samples/llm-apps-workshop/blob/main/blogs/rag/blog_post.md
• Architect defense-in-depth security for generative AI applications using the OWASP Top 10 for LLMs - https://aws.amazon.com/blogs/machine-learning/architect-defense-in-depth-security-for-generative-ai-applications-using-the-owasp-top-10-for-llms/

Implementations - others

• Build a Chatbot on Your CSV Data With LangChain and OpenAI - https://betterprogramming.pub/build-a-chatbot-on-your-csv-data-with-langchain-and-openai-ed121f85f0cd
• Building a Multi-Document Reader and Chatbot With LangChain and ChatGPT - https://betterprogramming.pub/building-a-multi-document-reader-and-chatbot-with-langchain-and-chatgpt-d1864d47e339
• LangChain with FAISS - https://python.langchain.com/docs/integrations/vectorstores/faiss
• How to create Embeddings for Open Source-LLM’s - https://bagisto.com/en/embeddings-in-large-language-models/

Hugging Face

• Total noob’s intro to Hugging Face Transformers
• Hugging Face Transformers - Hugging Face Transformers is an open-source Python library that provides access to thousands of pre-trained Transformers models for natural language processing (NLP), computer vision, audio tasks, and more. It simplifies the process of implementing Transformer models by abstracting away the complexity of training or deploying models in lower level ML frameworks like PyTorch, TensorFlow and JAX.
• Hugging Face Hub - The Hugging Face Hub is a collaboration platform that hosts a huge collection of open-source models and datasets for machine learning, think of it being like Github for ML. The hub facilitates sharing and collaborating by making it easy for you to discover, learn, and interact with useful ML assets from the open-source community. The hub integrates with, and is used in conjunction with the Transformers library, as models deployed using the Transformers library are downloaded from the hub.
• Hugging Face Spaces - Spaces from Hugging Face is a service available on the Hugging Face Hub that provides an easy to use GUI for building and deploying web hosted ML demos and apps. The service allows you to quickly build ML demos, upload your own apps to be hosted, or even select a number of pre-configured ML applications to deploy instantly.

Tools

• ollama - https://ollama.com/
• LLMStudio - https://h2o.ai/platform/ai-cloud/make/llm-studio/
• LMStudio - https://lmstudio.ai/
• nanoGPT - https://github.com/karpathy/nanoGPT
• PandasAI - https://docs.pandas-ai.com/en/latest/
• docker/genai-stack - https://github.com/docker/genai-stack
• huggingface - https://github.com/huggingface
• evals - https://github.com/openai/evals
• confident-ai/deepeval - https://github.com/confident-ai/deepeval

Miscellaneous

According to Alireza Goudarzi, senior researcher of machine learning (ML) for GitHub Copilot: “LLMs are not trained to reason. They’re not trying to understand science, literature, code, or anything else. They’re simply trained to predict the next token in the text.” Source

StackOverflow