test_latency_prompts.py

# 500 token
FIVE_HUNDRED_TOKEN_QUESTION = """Imagine a scenario where a medium-sized, technologically advanced company, located in the heart of Silicon Valley and specializing in the development of autonomous drone technology, is facing a strategic crossroads due to recent shifts in both market dynamics and regulatory landscapes. As the lead AI strategist, you are tasked with assessing the potential impact of integrating next-generation artificial intelligence systems into their current operations. The company's CEO, with a keen interest in leveraging AI for enhancing product capabilities and operational efficiency, has sought your expert advice. Considering the latest advancements in AI, particularly in machine learning algorithms and data processing capabilities, you are to provide a detailed analysis. This analysis should cover several critical aspects:
Evaluate the potential benefits and challenges associated with incorporating machine learning into the company's drone navigation systems, including issues related to data privacy and algorithmic bias.
Discuss how AI could be used to optimize supply chain management for this company, considering the complexities of global logistics and the importance of minimizing delays and costs.
Explore the possibilities of using AI-driven predictive maintenance in drones to reduce downtime and increase longevity, detailing the types of data required and the feasibility of real-time data analysis.
Consider the implications of international regulatory frameworks on the deployment of AI technologies in drones, with a particular focus on the U.S., European Union, and China. Include a discussion on how these regulations might influence the company's operational strategies.
Assess the competitive landscape by identifying major players in the autonomous drone market who also integrate AI technologies. Compare their strategies, market share, and technological advancements with those of the company.
Provide strategic recommendations on whether the company should pursue aggressive expansion into AI-driven drone technologies or adopt a more cautious approach due to the associated risks and uncertainties.
Your analysis should draw upon the latest research and case studies in the field, incorporate statistical data where relevant, and conclude with a series of actionable insights that can guide the CEO's decision-making process. Additionally, consider any ethical considerations and the potential societal impacts of deploying advanced AI systems in autonomous drones. How would you approach this complex task, and what methodologies would you employ to ensure a comprehensive and balanced assessment?"""

# 1k token
ONE_THOUSAND_TOKEN_QUESTION = """In the context of evolving technological landscapes, especially with the advent of artificial intelligence (AI) and machine learning (ML) influencing multiple sectors, a nuanced understanding of these technologies' socio-economic, ethical, and psychological impacts is crucial. Considering your capabilities as a sophisticated language model developed by OpenAI, I have several layered questions that require a detailed exploration of these aspects:
Technological Impact: How have AI and ML specifically transformed the following industries: healthcare, automotive, and finance? Please provide a detailed analysis of the advancements in each sector, focusing on the implementation of AI technologies like deep learning and neural networks. Highlight any significant breakthroughs that have revolutionized these industries.
Economic Implications: Delve into the economic effects that AI and ML have had globally. Discuss both the positive and negative impacts, particularly in terms of job creation and displacement. How do these technologies affect economic disparities between developed and developing nations? Moreover, explore the potential long-term economic outcomes if AI continues to advance at its current pace.
Ethical Considerations: Address the ethical dilemmas posed by AI in areas such as privacy, surveillance, and decision-making. What are the major concerns about AI's role in decision-making processes in critical sectors like criminal justice and healthcare? Discuss the balance between leveraging AI for benefits while safeguarding individual rights and freedoms.
Psychological and Societal Changes: What are the psychological impacts of widespread AI and ML adoption on individuals and society? Analyze the changes in human behavior and interaction as AI becomes more integrated into daily life. How does the interaction with AI influence our perceptions of self and others, and what might be the long-term societal effects?
Future Trends and Predictions: Based on current developments and your vast dataset, what predictions can you make about the future of AI and ML? Highlight potential groundbreaking technologies or methodologies in AI that could emerge in the next decade. Additionally, consider the implications of these advancements on global stability, particularly in relation to cybersecurity and international relations.
Personalization of AI: How is AI being tailored to individual needs in sectors like education and personalized medicine? Discuss the technologies involved, the data privacy concerns that arise, and the potential for these personalized approaches to enhance individual outcomes in health and education.
Regulation and Governance: How should governments and international bodies regulate the development and deployment of AI to ensure it is safe, ethical, and beneficial? Analyze the current regulatory frameworks in place in leading AI research nations and discuss the gaps that still exist. What specific policies would you recommend to address these gaps, and how should these policies adapt to the rapid pace of AI innovation?
Impact on Education Systems: With AI's integration into educational tools and platforms, what are the transformative impacts observed in learning environments from primary to tertiary education? Explore how AI is reshaping teaching methodologies, student engagement, and personalized learning experiences. What are the challenges and potential risks of relying heavily on AI in educational settings?
AI in Creative Industries: How is AI being utilized in the arts, music, and entertainment industries, and what are the implications of this trend for creators and the industry structure? Discuss specific examples where AI has played a critical role in content creation, such as in music composition, film production, and visual arts, and debate the potential for AI to replace human creativity.
Ethics of AI in Warfare and Defense: Explore the ethical considerations of using AI in military and defense applications. What are the risks and benefits of autonomous weapons systems, surveillance technologies, and cyber defense mechanisms powered by AI? Discuss the international perspective on AI warfare, including any existing treaties and agreements, and the moral dilemmas these technologies pose.
AI's Role in Climate Change: Analyze how AI and ML are being used to combat climate change and environmental degradation. Detail the types of AI applications that are currently in use for monitoring, predicting, and mitigating environmental impacts. What are the limitations of these technologies, and what breakthroughs are needed to enhance their effectiveness in the fight against global warming?
Each of these questions demands a deep dive into complex and interconnected themes. Please elaborate extensively on each point, using relevant data and examples to support your discussions, and consider the multifaceted implications of AI and ML's integration into modern life."""

# 5k token
FIVE_THOUSAND_TOKEN_QUESTION = """Summarize the following: 
The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks. Unfortunately, existing LM pipelines are typically implemented using hard-coded
“prompt templates”, i.e. lengthy strings discovered via trial and error. Toward a
more systematic approach for developing and optimizing LM pipelines, we introduce DSPy, a programming model that abstracts LM pipelines as text transformation graphs, i.e. imperative computation graphs where LMs are invoked through
declarative modules. DSPy modules are parameterized, meaning they can learn
(by creating and collecting demonstrations) how to apply compositions of prompting, finetuning, augmentation, and reasoning techniques. We design a compiler
that will optimize any DSPy pipeline to maximize a given metric. We conduct
two case studies, showing that succinct DSPy programs can express and optimize
sophisticated LM pipelines that reason about math word problems, tackle multihop retrieval, answer complex questions, and control agent loops. Within minutes
of compiling, a few lines of DSPy allow GPT-3.5 and llama2-13b-chat to selfbootstrap pipelines that outperform standard few-shot prompting (generally by
over 25% and 65%, respectively) and pipelines with expert-created demonstrations (by up to 5–46% and 16–40%, respectively). On top of that, DSPy programs compiled to open and relatively small LMs like 770M-parameter T5 and
llama2-13b-chat are competitive with approaches that rely on expert-written
prompt chains for proprietary GPT-3.5.
DSPy is available at https://github.com/stanfordnlp/dspy.
1 INTRODUCTION
Language models (LMs) are enabling researchers to build NLP systems at higher levels of abstraction and with lower data requirements than ever before (Bommasani et al., 2021). This is fueling an
exploding space of “prompting” techniques—and lightweight finetuning techniques—for adapting
LMs to new tasks (Kojima et al., 2022), eliciting systematic reasoning from them (Wei et al., 2022;
Wang et al., 2022b), and augmenting them with retrieved sources (Guu et al., 2020; Lazaridou et al.,
2022; Khattab et al., 2022) or with tools (Yao et al., 2022; Schick et al., 2023). Most of these techniques are explored in isolation, but interest has been growing in building multi-stage pipelines and
agents that decompose complex tasks into more manageable calls to LMs in an effort to improve
performance (Qi et al., 2019; Khattab et al., 2021a; Karpas et al., 2022; Dohan et al., 2022; Khot
et al., 2022; Khattab et al., 2022; Chen et al., 2022; Pourreza & Rafiei, 2023; Shinn et al., 2023).
Unfortunately, LMs are known to be sensitive to how they are prompted for each task, and this is
exacerbated in pipelines where multiple LM calls have to interact effectively. As a result, the LM
1
arXiv:2310.03714v1 [cs.CL] 5 Oct 2023
Preprint
calls in existing LM pipelines and in popular developer frameworks are generally implemented using
hard-coded ‘prompt templates’, that is, long strings of instructions and demonstrations that are hand
crafted through manual trial and error. We argue that this approach, while pervasive, can be brittle
and unscalable—conceptually akin to hand-tuning the weights for a classifier. A given string prompt
might not generalize to different pipelines or across different LMs, data domains, or even inputs.
Toward a more systematic approach to designing AI pipelines, we introduce the DSPy programming
model.1 DSPy pushes building new LM pipelines away from manipulating free-form strings and
closer to programming (composing modular operators to build text transformation graphs) where a
compiler automatically generates optimized LM invocation strategies and prompts from a program.
We draw inspiration from the consensus that emerged around neural network abstractions (Bergstra
et al., 2013), where (1) many general-purpose layers can be modularly composed in any complex
architecture and (2) the model weights can be trained using optimizers instead of being hand-tuned.
To this end, we propose the DSPy programming model (Sec 3). We first translate string-based
prompting techniques, including complex and task-dependent ones like Chain of Thought (Wei et al.,
2022) and ReAct (Yao et al., 2022), into declarative modules that carry natural-language typed signatures. DSPy modules are task-adaptive components—akin to neural network layers—that abstract
any particular text transformation, like answering a question or summarizing a paper. We then parameterize each module so that it can learn its desired behavior by iteratively bootstrapping useful
demonstrations within the pipeline. Inspired directly by PyTorch abstractions (Paszke et al., 2019),
DSPy modules are used via expressive define-by-run computational graphs. Pipelines are expressed
by (1) declaring the modules needed and (2) using these modules in any logical control flow (e.g.,
if statements, for loops, exceptions, etc.) to logically connect the modules.
We then develop the DSPy compiler (Sec 4), which optimizes any DSPy program to improve quality
or cost. The compiler inputs are the program, a few training inputs with optional labels, and a validation metric. The compiler simulates versions of the program on the inputs and bootstraps example
traces of each module for self-improvement, using them to construct effective few-shot prompts
or finetuning small LMs for steps of the pipeline. Optimization in DSPy is highly modular: it is
conducted by teleprompters,
2 which are general-purpose optimization strategies that determine how
the modules should learn from data. In this way, the compiler automatically maps the declarative
modules to high-quality compositions of prompting, finetuning, reasoning, and augmentation.
Programming models like DSPy could be assessed along many dimensions, but we focus on the role
of expert-crafted prompts in shaping system performance. We are seeking to reduce or even remove
their role through DSPy modules (e.g., versions of popular techniques like Chain of Thought) and
teleprompters. We report on two expansive case studies: math word problems (GMS8K; Cobbe et al.
2021) and multi-hop question answering (HotPotQA; Yang et al. 2018) with explorations of chain
of thought, multi-chain reflection, multi-hop retrieval, retrieval-augmented question answering, and
agent loops. Our evaluations use a number of different compiling strategies effectively and show
that straightforward DSPy programs outperform systems using hand-crafted prompts, while also
allowing our programs to use much smaller and hence more efficient LMs effectively.
Overall, this work proposes the first programming model that translates prompting techniques into
parameterized declarative modules and introduces an effective compiler with general optimization strategies (teleprompters) to optimize arbitrary pipelines of these modules. Our main contributions are empirical and algorithmic: with DSPy, we have found that we can implement very
short programs that can bootstrap self-improving multi-stage NLP systems using LMs as small as
llama2-13b-chat and T5-Large (770M parameters). Without hand-crafted prompts and within
minutes to tens of minutes of compiling, compositions of DSPy modules can raise the quality of
simple programs from 33% to 82% (Sec 6) and from 32% to 46% (Sec 7) for GPT-3.5 and, similarly, from 9% to 47% (Sec 6) and from 22% to 41% (Sec 7) for llama2-13b-chat.
1DSPy is pronounced dee-ess-pie. It’s the second iteration of our earlier Demonstrate–Search–Predict
framework (DSP; Khattab et al. 2022). This paper introduces the key concepts in DSPy. For more extensive and
up-to-date documentation of the framework, we refer readers to https://github.com/stanfordnlp/dspy.
2We derive the name tele-prompters from the notion of abstracting and automating the task of prompting,
in particular, such that it happens at a distance, without manual intervention.
2
Preprint
2 RELATED WORK
This work is inspired by the role that Torch (Collobert et al., 2002), Theano (Bergstra et al., 2010;
2011; Al-Rfou et al., 2016), Chainer (Tokui et al., 2015), and others played in the development in
deep learning by providing powerful abstractions. A similar transformation is emerging with higherlevel pipelines of LMs, and we are seeking to offer a solid conceptual framework and programming
abstractions for what we call foundation model programming. We draw on differentiable programming (Wang et al., 2018) but applied to LM calls rather than neural networks, and borrow syntactic
elements from PyTorch (Paszke et al., 2019).
In-context learning (McCann et al. 2018; Radford et al. 2018; Brown et al. 2020) is a key mechanism
for foundation model programming. A growing body of work has revealed that, especially with
instruction tuning (Ouyang et al., 2022), we can elicit sophisticated behavior via prompting (Wei
et al., 2022; Wang et al., 2022b; Press et al., 2022; Yao et al., 2022; Khot et al., 2022; Madaan et al.,
2023). Similarly, forms of weak supervision that would normally require task-specific (Khattab
et al., 2021a;b) or hand-built (Ratner et al., 2016; Hancock et al., 2018) heuristics are now done by
LMs (Wang et al., 2022b; Zelikman et al., 2022; Zhang et al., 2022; Shao et al., 2023).
In-context learning methods now routinely invoke tools, leading to LM pipelines that use retrieval
models (Chen et al., 2017; Lewis et al., 2020; Guu et al., 2020; Lazaridou et al., 2022; Izacard et al.,
2022), multimodal foundation models, and more traditional tools like APIs (Nakano et al., 2021)
and calculators. A number of toolkits have been developed to facilitate this, including LangChain
(Chase, 2022), Semantic Kernel (Microsoft, 2023), LlamaIndex (Liu, 2022), and many other retrieval and agent libraries. These toolkits provide pre-packaged chains and agents that connect
LMs with numerous accessible tools. However, they suffer from the pervasive prompt engineering
challenges we address in DSPy: they express task-specific behavior through hand-written prompt
templates (for detailed discussion, see Appendix B).
Researchers are starting to apply discrete optimization and RL to find effective prompts, generally
for a single logical LM call (Guo et al., 2023; Pryzant et al., 2023; Huang et al., 2022; Yang et al.,
2023). DSPy seeks to generalize this space: it offers a rich framework for optimizing arbitrary
pipelines from high-level declarative signatures, by bootstrapping high-quality multi-stage demonstrations with constraints. In this framework, DSPy teleprompters may apply optimization using
model selection techniques like cross-validation or, in principle, with sophisticated techniques involving RL and LM feedback (Hu et al., 2023; Zhao et al., 2023a; Shinn et al., 2023) or learned or
Bayesian hyperparameter optimization methods (Bergstra et al., 2013; Akiba et al., 2019).
The present paper seeks to motivate DSPy as a programming model and to report new empirical
findings from applying the DSPy compiler. This is inspired by formative work by Bergstra et al.
(2010; 2013), Paszke et al. (2019), and Wolf et al. (2020), who support their respective programming
models with a mix of benchmark numbers and some qualitative measures. For the current paper, we
focus on showing that DSPy and its compiler allow us to build outstanding LM systems without
hand-crafted prompt strings, but instead from truly modular units, and that this opens up doors for
systematically exploring a rich design space at a very high programmatic level of abstraction.
3 THE DSPY PROGRAMMING MODEL
We present DSPy, which treats LMs as abstract devices for text generation,3
and optimizes their usage in arbitrary computational graphs. DSPy programs are expressed in Python: each program takes
the task input (e.g., a question to answer or a paper to summarize) and returns the output (e.g., an
answer or a summary) after a series of steps. DSPy contributes three abstractions toward automatic
optimization: signatures, modules, and teleprompters. Signatures abstract the input/output behavior
of a module; modules replace existing hand-prompting techniques and can be composed in arbitrary
pipelines; and teleprompters optimize all modules in the pipeline to maximize a metric.
3We assume access to one or more LMs, which consume a prompt string and return text completions. This
may be a promptable LM capable of in-context learning (e.g., GPT-3.5 or Llama2-7b) or a smaller finetuneable
LM (e.g., T5-base). An LM may be selected as the default; operations will use it unless configured otherwise.
3
Preprint
3.1 NATURAL LANGUAGE SIGNATURES CAN ABSTRACT PROMPTING & FINETUNING
Instead of free-form string prompts, DSPy programs use natural language signatures to assign work
to the LM. A DSPy signature is natural-language typed declaration of a function: a short declarative
spec that tells DSPy what a text transformation needs to do (e.g., “consume questions and return
answers”), rather than how a specific LM should be prompted to implement that behavior. More
formally, a DSPy signature is a tuple of input fields and output fields (and an optional instruction).
A field consists of field name and optional metadata.4
In typical usage, the roles of fields are inferred
by DSPy as a function of field names. For instance, the DSPy compiler will use in-context learning
to interpret question differently from answer and will iteratively refine its usage of these fields.
Signatures offer two benefits over prompts: they can be compiled into self-improving and pipelineadaptive prompts or finetunes. This is primarily done by bootstrapping (Sec 4) useful demonstrating
examples for each signature. Additionally, they handle structured formatting and parsing logic to
reduce (or, ideally, avoid) brittle string manipulation in user programs.
In practice, DSPy signatures can be expressed with a shorthand notation like question -> answer,
so that line 1 in the following is a complete DSPy program for a basic question-answering system
(with line 2 illustrating usage and line 3 the response when GPT-3.5 is the LM):
1 qa = dspy . Predict (" question -> answer ")
2 qa ( question =" Where is Guaran ´ı spoken ?")
3 # Out: Prediction ( answer = ’ Guaran ´ı is spoken mainly in South America . ’)
In the shorthand notation, each field’s name indicates the semantic role that the input (or output)
field plays in the transformation. DSPy will parse this notation and expand the field names into
meaningful instructions for the LM, so that english document -> french translation would
prompt for English to French translation. When needed, DSPy offers more advanced programming
interfaces for expressing more explicit constraints on signatures (Appendix A).
3.2 PARAMETERIZED & TEMPLATED MODULES CAN ABSTRACT PROMPTING TECHNIQUES
Akin to type signatures in programming languages, DSPy signatures simply define an interface and
provide type-like hints on the expected behavior. To use a signature, we must declare a module with
that signature, like we instantiated a Predict module above. A module declaration like this returns
a function having that signature.
The Predict Module The core module for working with signatures in DSPy is Predict (simplified
pseudocode in Appendix D.1). Internally, Predict stores the supplied signature, an optional LM to
use (initially None, but otherwise overrides the default LM for this module), and a list of demonstrations for prompting (initially empty). Like layers in PyTorch, the instantiated module behaves as
a callable function: it takes in keyword arguments corresponding to the signature input fields (e.g.,
question), formats a prompt to implement the signature and includes the appropriate demonstrations, calls the LM, and parses the output fields. When Predict detects it’s being used in compile
mode, it will also internally track input/output traces to assist the teleprompter at bootstrapping the
demonstrations.
Other Built-in Modules DSPy modules translate prompting techniques into modular functions that
support any signature, contrasting with the standard approach of prompting LMs with task-specific
details (e.g., hand-written few-shot examples). To this end, DSPy includes a number of more sophisticated modules like ChainOfThought, ProgramOfThought, MultiChainComparison, and ReAct.
5
These can all be used interchangeably to implement a DSPy signature. For instance, simply chang4
String descriptions of the task and the fields are also optional and usually omitted. Fields can carry optional
field prefix and description. By default, fields are assumed to hold free-form strings; we are actively exploring
optional data type as a way to specify constraints on valid values (e.g., bool or int) and more gracefully handle
formatting and parsing logic, though this feature is not core to DSPy at the time of writing.
5These modules generalize prompting techniques from the literature, respectively, by Wei et al. (2022),
Chen et al. (2022), Yoran et al. (2023), and Yao et al. (2022) and, in doing so, generalize the ideas on zero-shot
prompting and rationale self-generation from Kojima et al. (2022), Zelikman et al. (2022), Zhang et al. (2022),
and Huang et al. (2022) to parameterized modules that can bootstrap arbitrary multi-stage pipelines.
4
Preprint
ing Predict to ChainOfThought in the above program leads to a system that thinks step by step
before committing to its output field.
Importantly, all of these modules are implemented in a few lines of code by expanding the userdefined signature and calling Predict one or more times on new signatures as appropriate. For
instance, we show a simplified implementation of the built-in ChainOfThought below.
1 class ChainOfThought ( dspy . Module ) :
2 def __init__ ( self , signature ) :
3 # Modify signature from ‘* inputs -> * outputs ‘ to ‘* inputs -> rationale , * outputs ‘.
4 rationale_field = dspy . OutputField ( prefix =" Reasoning : Let ’s think step by step .")
5 signature = dspy . Signature ( signature ) . prepend_output_field ( rationale_field )
6
7 # Declare a sub - module with the modified signature .
8 self . predict = dspy . Predict ( signature )
9
10 def forward ( self , ** kwargs ) :
11 # Just forward the inputs to the sub - module .
12 return self . predict (** kwargs )
This is a fully-fledged module capable of learning effective few-shot prompting for any LM or task.
We contrast that with Appendix C, which copies long reasoning prompts hand-written by sources
ranging from recent research to popular prompting libraries.
Parameterization Uniquely, DSPy parameterizes these prompting techniques. To understand this
parameterization, observe that any LM call seeking to implement a particular signature needs to
specify parameters that include: (1) the specific LM to call (Chen et al., 2023), (2) the prompt instructions (Yang et al., 2023) and the string prefix of each signature field and, most importantly, (3)
the demonstrations used as few-shot prompts (for frozen LMs) or as training data (for finetuning).
We focus primarily on automatically generating and selecting useful demonstrations. In our case
studies, we find that bootstrapping good demonstrations gives us a powerful way to teach sophisticated pipelines of LMs new behaviors systematically.
Tools DSPy programs may use tools, which are modules that execute computation. We support retrieval models through a dspy.Retrieve module. At the time of writing, DSPy has built-in support
for ColBERTv2, Pyserini, and Pinecone retrievers, and we have explored experimental dspy.SQL
for executing SQL queries and dspy.PythonInterpreter for executing Python code in a sandbox.
Programs DSPy modules can be composed in arbitrary pipelines in a define-by-run interface. Inspired directly by PyTorch and Chainer, one first declares the modules needed at initialization, allowing DSPy to keep track of them for optimization, and then one expresses the pipeline with arbitrary
code that calls the modules in a forward method. As a simple illustration, we offer the following
simple but complete retrieval-augmented generation (RAG) system.
1 class RAG ( dspy . Module ) :
2 def __init__ ( self , num_passages =3) :
3 # ‘Retrieve ‘ will use the user ’s default retrieval settings unless overriden .
4 self . retrieve = dspy . Retrieve ( k = num_passages )
5 # ‘ChainOfThought ‘ with signature that generates answers given retrieval & question .
6 self . generate_answer = dspy . ChainOfThought (" context , question -> answer ")
7
8 def forward ( self , question ) :
9 context = self . retrieve ( question ) . passages
10 return self . generate_answer ( context = context , question = question )
To highlight modularity, we use ChainOfThought as a drop-in replacement of the basic Predict.
One can now simply write RAG()("Where is Guaran´ı spoken?") to use it. Notice that, if we
use a signature "context, question -> search query", we get a system that generates search
queries rather than answers.
3.3 TELEPROMPTERS CAN AUTOMATE PROMPTING FOR ARBITRARY PIPELINES
When compiling a DSPy program, we generally invoke a teleprompter, which is an optimizer that
takes the program, a training set, and a metric—and returns a new optimized program. Different
teleprompters (Sec 4) apply different strategies for optimization.
5
Preprint
In DSPy, training sets may be small, potentially a handful of examples, though larger data enables
more powerful optimization. Training examples may be incomplete, i.e., only input values are necessary. Labels for the pipeline steps are not required, unless they need to be used in the metric. In
practice, we typically assume labels only for (at most) the program’s final output, not the intermediate steps. This label-efficiency is critical for modularity: building a new pipeline in DSPy requires
simply recompiling the new pipeline’s code, not annotating data specific to the new pipeline.
Metrics can be simple notions like exact match (EM) or F1, but they can be entire DSPy programs
that balance multiple concerns. For example, we may compile the RAG module above against a
dataset of question–answer pairs qa trainset and the metric EM. The goal of optimization here is
to effectively bootstrap few-shot demonstrations. The following code achieves this:
1 # Small training set with only questions and final answers .
2 qa_trainset = [ dspy . Example ( question =" What is the capital of France ?", answer =" Paris ") ]
3
4 # The teleprompter will bootstrap missing labels : reasoning chains and retrieval contexts .
5 teleprompter = dspy . BootstrapFewShot ( metric = dspy . evaluate . answer_exact_match )
6 compiled_rag = teleprompter . compile ( RAG () , trainset = qa_trainset )
In this example, the BootstrapFewShot teleprompter (Sec 4, Appendix E.1) simulates RAG on the
training example(s). It will collect demonstrations of each module (i.e., examples of its input–output
behavior) that collectively lead to valid output (i.e., respecting the signatures and the metric).
If one wanted to push the compiled program to be extractive given its retrieved contexts, one could
define a custom metric to use in place of dspy.evaluate.answer exact match:
1 def answer_and_context_match ( example , pred , trace = None ) :
2 answer_match = dspy . evaluate . answer_exact_match ( example , pred )
3
4 # Is the prediction a substring of some passage ?
5 context_match = any (( pred . answer . lower () in c ) for c in pred . context )
6
7 return answer_match and context_match
Notice that behavior like this might be more accurately checked by another DSPy program that
checks for faithful grounding of answers. Such metrics are fully supported and encouraged in DSPy.
Teleprompters can be composed by specifying a teacher program. DSPy will sample demonstrations from this program for prompt optimization. This composition can enable very rich pipelines,
where expensive programs (e.g., complex expensive ensembles using large LMs) supervise cheap
programs (e.g., simple pipelines using smaller LMs). One may start with compiled rag from above
(say, compiled to use a large Llama2-13b-chat LM) but now fine-tune Flan-T5-large to create an
efficient program:
1 # Larger set of questions with *no labels *. Labels for all steps will be bootstrapped .
2 unlabeled_questions = [ dspy . Example ( question =" What is the capital of Germany ?") , ...]
3
4 # As we assumes no answer , we use ‘answer_passage_match ‘ to filter ungrounded answers .
5 finetuning_teleprompter = BootstrapFinetune ( metric = dspy . evaluate . answer_passage_match )
6
7 # We set ‘teacher = compiled_rag ‘ to compose . Bootstrapping will now use ‘compiled_rag ‘.
8 compiled_rag_via_finetune = finetuning_teleprompter . compile ( RAG () , teacher = compiled_rag ,
trainset = unlabeled_questions , target =’google /flan -t5 - large ’)
4 """