My Dialog

Paper_DeepSeekMath.ipynb

{"cells": [{"cell_type": "code", "metadata": {}, "source": ["import pymupdf4llm\n", "from IPython.display import Markdown, display\n", "md_text = pymupdf4llm.to_markdown(\"static/deepseek_math.pdf\")\n", "md_text[:100]"], "outputs": [{"output_type": "execute_result", "metadata": {}, "data": {"text/plain": ["'## **DeepSeekMath: Pushing the Limits of Mathematical** **Reasoning in Open Language Models**\\n\\nZhiho'"]}, "execution_count": 2}], "execution_count": null, "attachments": {}}, {"cell_type": "code", "metadata": {}, "source": ["def split_md_by_headings(text):\n", "    lines = text.split('\\n')\n", "    sections = []\n", "    current_section = []\n", "    current_title = \"Introduction\"\n", "    \n", "    for line in lines:\n", "        if line.startswith('###') or line.startswith('##') or line.startswith('#'):\n", "            if current_section: sections.append((current_title, '\\n'.join(current_section)))\n", "            current_title = line.strip('#').strip()\n", "            current_section = [line]\n", "        else: current_section.append(line)\n", "    \n", "    if current_section: sections.append((current_title, '\\n'.join(current_section)))\n", "    return sections\n", "\n", "sections = split_md_by_headings(md_text)\n", "print(f\"Found {len(sections)} sections\")\n", "[s[0] for s in sections[:10]]"], "outputs": [{"output_type": "stream", "name": "stdout", "text": ["Found 10 sections\n"]}, {"output_type": "execute_result", "metadata": {}, "data": {"text/plain": ["['**DeepSeekMath: Pushing the Limits of Mathematical** **Reasoning in Open Language Models**',\n", " '**Abstract**',\n", " '**1. Introduction**',\n", " '**2. Math Pre-Training**',\n", " '**3. Supervised Fine-Tuning**',\n", " '**4. Reinforcement Learning**',\n", " '**5. Discussion**',\n", " '**6. Conclusion, Limitation, and Future Work**',\n", " '**References**',\n", " '**A. Appendix**']"]}, "execution_count": 3}], "execution_count": null, "attachments": {}}, {"cell_type": "code", "metadata": {}, "source": ["def display_section(idx):\n", "    title, content = sections[idx]\n", "    display(Markdown(content))\n", "\n", "display_section(1)"], "outputs": [{"output_type": "display_data", "metadata": {}, "data": {"text/plain": ["<IPython.core.display.Markdown object>"], "text/markdown": ["### **Abstract**\n", "```\n", "\n", "Mathematical reasoning poses a significant challenge for language models due to its complex\n", "and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pretraining DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common\n", "Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an\n", "impressive score of 51.7% on the competition-level MATH benchmark without relying on\n", "external toolkits and voting techniques, approaching the performance level of Gemini-Ultra\n", "and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH.\n", "The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First,\n", "we harness the significant potential of publicly available web data through a meticulously\n", "engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization\n", "(GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning\n", "abilities while concurrently optimizing the memory usage of PPO.\n", "\n", "Figure 1 | Top1 accuracy of open-source models on the competition-level MATH benchmark\n", "(Hendrycks et al., 2021) without the use of external toolkits and voting techniques.\n", "\n", "\u2217 Core contributors.\n", "\n", " - Work done during internship at DeepSeek-AI.\n"]}, "transient": {}}], "execution_count": null, "attachments": {}}, {"cell_type": "code", "metadata": {}, "source": ["display_section(2)"], "outputs": [{"output_type": "display_data", "metadata": {}, "data": {"text/plain": ["<IPython.core.display.Markdown object>"], "text/markdown": ["##### **1. Introduction**\n", "\n", "Large language models (LLM) have revolutionized the approach to mathematical reasoning\n", "in artificial intelligence, spurring significant advancements in both the quantitative reasoning\n", "benchmark (Hendrycks et al., 2021) and the geometry reasoning benchmark (Trinh et al., 2024).\n", "Moreover, these models have proven instrumental in assisting humans in solving complex\n", "mathematical problems (Tao, 2023). However, cutting-edge models such as GPT-4 (OpenAI,\n", "2023) and Gemini-Ultra (Anil et al., 2023) are not publicly available, and the currently accessible\n", "open-source models considerably trail behind in performance.\n", "\n", "In this study, we introduce DeepSeekMath, a domain-specific language model that significantly outperforms the mathematical capabilities of open-source models and approaches the\n", "performance level of GPT-4 on academic benchmarks. To achieve this, we create the DeepSeekMath Corpus, a large-scale high-quality pre-training corpus comprising 120B math tokens. This\n", "dataset is extracted from the Common Crawl (CC) using a fastText-based classifier (Joulin et al.,\n", "2016). In the initial iteration, the classifier is trained using instances from OpenWebMath (Paster\n", "et al., 2023) as positive examples, while incorporating a diverse selection of other web pages to\n", "serve as negative examples. Subsequently, we employ the classifier to mine additional positive\n", "instances from the CC, which are further refined through human annotation. The classifier is\n", "then updated with this enhanced dataset to improve its performance. The evaluation results\n", "indicate that the large-scale corpus is of high quality, as our base model DeepSeekMath-Base\n", "7B achieves 64.2% on GSM8K (Cobbe et al., 2021) and 36.2% on the competition-level MATH\n", "dataset (Hendrycks et al., 2021), outperforming Minerva 540B (Lewkowycz et al., 2022a). In\n", "addition, the DeepSeekMath Corpus is multilingual, so we notice an improvement in Chinese\n", "mathematical benchmarks (Wei et al., 2023; Zhong et al., 2023). We believe that our experience\n", "in mathematical data processing is a starting point for the research community, and there is\n", "significant room for improvement in the future.\n", "\n", "DeepSeekMath-Base is initialized with DeepSeek-Coder-Base-v1.5 7B (Guo et al., 2024), as\n", "we notice that starting from a code training model is a better choice compared to a general\n", "LLM. Furthermore, we observe the math training also improves model capability on MMLU\n", "(Hendrycks et al., 2020) and BBH benchmarks (Suzgun et al., 2022), indicating it does not only\n", "enhance the model\u2019s mathematical abilities but also amplifies general reasoning capabilities.\n", "\n", "After pre-training, we apply mathematical instruction tuning to DeepSeekMath-Base with\n", "chain-of-thought (Wei et al., 2022), program-of-thought (Chen et al., 2022; Gao et al., 2023), and\n", "tool-integrated reasoning (Gou et al., 2023) data. The resulting model DeepSeekMath-Instruct\n", "7B beats all 7B counterparts and is comparable with 70B open-source instruction-tuned models.\n", "\n", "Furthermore, we introduce the Group Relative Policy Optimization (GRPO), a variant reinforcement learning (RL) algorithm of Proximal Policy Optimization (PPO) (Schulman et al., 2017).\n", "GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly\n", "reducing training resources. By solely using a subset of English instruction tuning data, GRPO\n", "obtains a substantial improvement over the strong DeepSeekMath-Instruct, including both\n", "in-domain (GSM8K: 82.9% \u2192 88.2%, MATH: 46.8% \u2192 51.7%) and out-of-domain mathematical\n", "tasks (e.g., CMATH: 84.6% \u2192 88.8%) during the reinforcement learning phase. We also provide\n", "a unified paradigm to understand different methods, such as Rejection Sampling Fine-Tuning\n", "(RFT) (Yuan et al., 2023a), Direct Preference Optimization (DPO) (Rafailov et al., 2023), PPO and\n", "GRPO. Based on such a unified paradigm, we find that all these methods are conceptualized as\n", "either direct or simplified RL techniques. We also conduct extensive experiments, e.g., online\n", "v.s. offline training, outcome v.s. process supervision, single-turn v.s. iterative RL and so on,\n", "\n", "2\n", "\n", "to deeply investigate the essential elements of this paradigm. At last, we explain why our RL\n", "boosts the performance of instruction-tuned models, and further summarize potential directions\n", "to achieve more effective RL based on this unified paradigm.\n", "\n", "**1.1. Contributions**\n", "\n", "Our contribution includes scalable math pre-training, along with the exploration and analysis of\n", "reinforcement learning.\n", "\n", "**Math Pre-Training at Scale**\n", "\n", " - Our research provides compelling evidence that the publicly accessible Common Crawl\n", "data contains valuable information for mathematical purposes. By implementing a meticulously designed data selection pipeline, we successfully construct the DeepSeekMath\n", "Corpus, a high-quality dataset of 120B tokens from web pages filtered for mathematical content, which is almost 7 times the size of the math web pages used by Minerva\n", "(Lewkowycz et al., 2022a) and 9 times the size of the recently released OpenWebMath\n", "(Paster et al., 2023).\n", "\n", " - Our pre-trained base model DeepSeekMath-Base 7B achieves comparable performance\n", "with Minerva 540B (Lewkowycz et al., 2022a), indicating the number of parameters is not\n", "the only key factor in mathematical reasoning capability. A smaller model pre-trained on\n", "high-quality data could achieve strong performance as well.\n", "\n", " - We share our findings from math training experiments. Code training prior to math\n", "training improves models\u2019 ability to solve mathematical problems both with and without\n", "tool use. This offers a partial answer to the long-standing question: _does code training_\n", "_improve reasoning abilities?_ We believe it does, at least for mathematical reasoning.\n", "\n", " - Although training on arXiv papers is common, especially in many math-related papers, it\n", "brings no notable improvements on all mathematical benchmarks adopted in this paper.\n", "\n", "**Exploration and Analysis of Reinforcement Learning**\n", "\n", " - We introduce Group Relative Policy Optimization (GRPO), an efficient and effective\n", "reinforcement learning algorithm. GRPO foregoes the critic model, instead estimating\n", "the baseline from group scores, significantly reducing training resources compared to\n", "Proximal Policy Optimization (PPO).\n", "\n", " - We demonstrate that GRPO significantly enhances the performance of our instructiontuned model DeepSeekMath-Instruct, by solely using the instruction-tuning data. Furthermore, we observe enhancements in the out-of-domain performance during the reinforcement learning process.\n", "\n", " - We provide a unified paradigm to understand different methods, such as RFT, DPO,\n", "PPO, and GRPO. We also conduct extensive experiments, e.g., online v.s. offline training,\n", "outcome v.s. process supervision, single-turn v.s. iterative reinforcement learning, and so\n", "on to deeply investigate the essential elements of this paradigm.\n", "\n", " - Based on our unified paradigm, we explore the reasons behind the effectiveness of reinforcement learning, and summarize several potential directions to achieve more effective\n", "reinforcement learning of LLMs.\n", "\n", "**1.2. Summary of Evaluations and Metrics**\n", "\n", " - **English and Chinese Mathematical Reasoning** : We conduct comprehensive assessments\n", "of our models on English and Chinese benchmarks, covering mathematical problems\n", "\n", "3\n", "\n", "from grade-school level to college level. English benchmarks include GSM8K (Cobbe\n", "et al., 2021), MATH (Hendrycks et al., 2021), SAT (Azerbayev et al., 2023), OCW Courses\n", "(Lewkowycz et al., 2022a), MMLU-STEM (Hendrycks et al., 2020). Chinese benchmarks\n", "include MGSM-zh (Shi et al., 2023), CMATH (Wei et al., 2023), Gaokao-MathCloze (Zhong\n", "et al., 2023), and Gaokao-MathQA (Zhong et al., 2023). We evaluate models\u2019 ability\n", "to generate self-contained text solutions without tool use, and also the ability to solve\n", "problems using Python.\n", "On English benchmarks, DeepSeekMath-Base is competitive with the closed-source Minerva 540B (Lewkowycz et al., 2022a), and surpasses all open-source base models (e.g., Mistral 7B (Jiang et al., 2023) and Llemma-34B (Azerbayev et al., 2023)), regardless of whether\n", "they\u2019ve undergone math pre-training or not, often by a significant margin. Notably,\n", "DeepSeekMath-Base is superior on Chinese benchmarks, likely because we don\u2019t follow\n", "previous works (Azerbayev et al., 2023; Lewkowycz et al., 2022a) to collect English-only\n", "math pre-training data, and also include high-quality non-English ones. With mathematical instruction tuning and reinforcement learning, the resulting DeepSeekMath-Instruct\n", "and DeepSeekMath-RL demonstrate strong performance, obtaining an accuracy of over\n", "50% on the competition-level MATH dataset for the first time within the open-source\n", "community.\n", "\n", " - **Formal Mathematics** : We evaluate DeepSeekMath-Base using the informal-to-formal\n", "theorem proving task from (Jiang et al., 2022) on miniF2F (Zheng et al., 2021) with Isabelle\n", "(Wenzel et al., 2008) chosen to be the proof assistant. DeepSeekMath-Base demonstrates\n", "strong few-shot autoformalization performance.\n", "\n", " - **Natural Language Understanding, Reasoning, and Code** : To build a comprehensive\n", "profile of models\u2019 general understanding, reasoning, and coding capabilities, we evaluate DeepSeekMath-Base on the Massive Multitask Language Understanding (MMLU)\n", "benchmark (Hendrycks et al., 2020) which encompasses 57 multiple-choice tasks covering\n", "diverse subjects, BIG-Bench Hard (BBH) (Suzgun et al., 2022) which consists of 23 challenging tasks that mostly require multi-step reasoning to solve, as well as HumanEval\n", "(Chen et al., 2021) and MBPP (Austin et al., 2021) which are widely used to evaluate code\n", "language models. Math pre-training benefits both language understanding and reasoning\n", "performance."]}, "transient": {}}], "execution_count": null, "attachments": {}}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "What is a critic model?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "A critic model is a component used in reinforcement learning algorithms, particularly in methods like Proximal Policy Optimization (PPO). \n", "\n", "In the context of training language models, the critic model serves as a \"judge\" that estimates how good or valuable a particular state or action is. Think of it like having a separate AI that watches what the main model is doing and gives it feedback on whether its responses are heading in the right direction.\n", "\n", "The critic helps the main model learn by providing baseline estimates - essentially saying \"this response is better or worse than average\" - which helps guide the learning process more efficiently.\n", "\n", "From the paper you've loaded, it mentions that their new GRPO method \"foregoes the critic model, instead estimating the baseline from group scores.\" This means they found a way to get similar benefits without needing this separate critic component, which saves computational resources.\n", "\n", "Would you like me to explain more about how this fits into the reinforcement learning process, or are you curious about another aspect of the paper?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Is it the same as reward model?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Great question! They're related but not exactly the same, though the terms are sometimes used interchangeably in different contexts.\n", "\n", "A **reward model** typically learns to assign scores to complete outputs - like rating how good a final answer is. It's often trained on human preferences to understand what makes one response better than another.\n", "\n", "A **critic model** in reinforcement learning estimates the \"value\" of being in a particular state - essentially predicting the expected future reward from that point. It helps guide the learning process step-by-step.\n", "\n", "In practice, when training language models, these concepts can overlap. Some systems use a single model that serves both purposes, while others keep them separate.\n", "\n", "Are you thinking about how this applies to the DeepSeekMath paper specifically, or are you curious about the broader distinction between these concepts?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "I dont understand still. In this specific paper, how do those differ? And can you provide more concrete example?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Looking at this specific paper, let me clarify with a concrete example:\n", "\n", "In traditional PPO training for math problems:\n", "- **Critic model**: Looks at a partial solution like \"To solve 2x + 3 = 7, I need to...\" and estimates \"this is heading toward a good solution, value = 0.8\"\n", "- **Reward model**: Looks at the complete final answer \"x = 2\" and judges \"this is correct, reward = +1\" or \"this is wrong, reward = -1\"\n", "\n", "But DeepSeekMath's GRPO approach eliminates the critic entirely. Instead of having a separate model estimate values, it:\n", "- Generates multiple complete solutions to the same problem\n", "- Uses the group's average performance as the baseline\n", "- Compares each solution directly to this group average\n", "\n", "So if 5 solutions to a math problem get rewards [1, 0, 1, 0, 1], the baseline becomes 0.6, and each solution is judged relative to that.\n", "\n", "This saves computational resources because you don't need to train and run a separate critic model.\n", "\n", "Does this distinction make more sense now? Would you like me to explain why this group-based approach might work well for math problems specifically?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "so critic model is same as value model?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Yes, exactly! In reinforcement learning terminology, a critic model and a value model are the same thing. \n", "\n", "The critic/value model estimates the \"value function\" - which predicts how good it is to be in a particular state (or how much total reward you can expect from that point forward).\n", "\n", "This is different from a policy model (which decides what action to take) and different from a reward model (which scores individual outputs or actions).\n", "\n", "Does this distinction make sense, or would you like me to clarify how these different components work together in the training process?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "What is a proof assistant like Isabelle? And is there anything to do with Lean?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "A proof assistant like Isabelle is a software tool that helps mathematicians and computer scientists write and verify formal mathematical proofs. Think of it as a very strict computer program that checks every single logical step in a proof to make sure it's completely correct.\n", "\n", "In regular mathematics, humans write proofs in natural language, and other humans check them. But proof assistants require you to write proofs in a precise formal language that the computer can understand and verify automatically.\n", "\n", "Yes, Lean is another proof assistant! It's actually quite popular and modern. Both Isabelle and Lean serve similar purposes but have different syntax and approaches. Lean has been getting a lot of attention lately, especially with projects like formalizing undergraduate mathematics.\n", "\n", "The DeepSeekMath paper mentions using Isabelle specifically for their \"informal-to-formal theorem proving task\" - essentially testing whether their AI can translate regular mathematical statements into the formal language that Isabelle understands.\n", "\n", "Are you interested in learning more about how these proof assistants work, or are you curious about how AI models like DeepSeekMath interact with them?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "why didnt they use Lean?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "That's a good question! The paper doesn't explicitly explain why they chose Isabelle over Lean, but there are a few possible reasons:\n", "\n", "1. **Dataset availability** - They might have had access to better or larger datasets of Isabelle proofs for training\n", "2. **Established benchmarks** - There may have been existing evaluation benchmarks using Isabelle that they wanted to compare against\n", "3. **Team expertise** - The researchers might have been more familiar with Isabelle's syntax and ecosystem\n", "\n", "Lean has been gaining popularity more recently, especially Lean 4, so the timing of when this research was done might also matter.\n", "\n", "Do you know much about the differences between Isabelle and Lean? Or are you curious about something else from the paper?\n", "\n"]}, {"cell_type": "code", "metadata": {}, "source": ["display_section(3)"], "outputs": [{"output_type": "display_data", "metadata": {}, "data": {"text/plain": ["<IPython.core.display.Markdown object>"], "text/markdown": ["##### **2. Math Pre-Training**\n", "\n", "**2.1. Data Collection and Decontamination**\n", "\n", "In this section, we will outline the process of constructing the DeepSeekMath Corpus from\n", "Common Crawl. As depicted in Figure 2, we present an iterative pipeline that demonstrates\n", "how to systematically gather a large-scale mathematical corpus from Common Crawl, starting\n", "with a seed corpus (e.g., a small but high-quality collection of math-related dataset). It\u2019s worth\n", "noting that this approach is also applicable to other domains, such as coding.\n", "\n", "First, we choose OpenWebMath (Paster et al., 2023), a collection of high-quality mathematical\n", "web texts, as our initial seed corpus. Using this corpus, we train a fastText model (Joulin et al.,\n", "2016) to recall more OpenWebMath-like mathematical web pages. Specifically, we randomly\n", "select 500,000 data points from the seed corpus as positive training examples and another\n", "500,000 web pages from Common Crawl as negative ones. We employ an open-source library [1]\n", "\n", "for training, configuring the vector dimension to 256, learning rate to 0.1, the maximum length\n", "\n", "1 `[https://fasttext.cc](https://fasttext.cc)`\n", "\n", "4\n", "\n", "Figure 2 | An iterative pipeline that collects mathematical web pages from Common Crawl.\n", "\n", "of word n-gram to 3, the minimum number of word occurrences to 3, and the number of\n", "training epochs to 3. To reduce the size of the original Common Crawl, we employ URL-based\n", "deduplication and near-deduplication techniques, resulting in 40B HTML web pages. We then\n", "recall mathematical web pages from deduplicated Common Crawl with the fastText model.\n", "To filter out low-quality mathematical content, we rank the collected pages according to their\n", "scores predicted by the fastText model, and only preserve the top-ranking ones. The volume\n", "of data preserved is assessed through pre-training experiments on the top 40B, 80B, 120B, and\n", "160B tokens. In the first iteration, we choose to keep the top 40B tokens.\n", "\n", "After the first iteration of data collection, numerous mathematical web pages remain uncollected, mainly because the fastText model is trained on a set of positive examples that lacks\n", "sufficient diversity. We therefore identify additional mathematical web sources to enrich the seed\n", "corpus, so that we can optimize the fastText model. Specifically, we first organize the entire Common Crawl into disjoint domains; a domain is defined as web pages sharing the same base URL.\n", "For each domain, we calculate the percentage of web pages that are collected in the first iteration.\n", "Domains where over 10% of the web pages have been collected are classified as math-related\n", "(e.g., `mathoverflow.net` ). Subsequently, we manually annotate the URLs associated with\n", "mathematical content within these identified domains (e.g., `mathoverflow.net/questions` ).\n", "Web pages linked to these URLs, yet uncollected, will be added to the seed corpus. This approach enables us to gather more positive examples, thereby training an improved fastText\n", "model capable of recalling more mathematical data in the subsequent iteration. After four\n", "iterations of data collection, we end up with 35.5M mathematical web pages, totaling 120B\n", "tokens. In the fourth iteration, we notice that nearly 98% of the data has already been collected\n", "in the third iteration, so we decide to cease data collection.\n", "\n", "To avoid benchmark contamination, we follow Guo et al. (2024) to filter out web pages\n", "containing questions or answers from English mathematical benchmarks such as GSM8K (Cobbe\n", "et al., 2021) and MATH (Hendrycks et al., 2021) and Chinese benchmarks such as CMATH\n", "(Wei et al., 2023) and AGIEval (Zhong et al., 2023). The filtering criteria are as follows: any\n", "text segment containing a 10-gram string that matches exactly with any sub-string from the\n", "evaluation benchmarks is removed from our math training corpus. For benchmark texts that\n", "are shorter than 10 grams but have at least 3 grams, we employ exact matching to filter out\n", "contaminated web pages.\n", "\n", "5\n", "\n", "**2.2. Validating the Quality of the DeepSeekMath Corpus**\n", "\n", "We run pre-training experiments to investigate how the DeepSeekMath Corpus is compared\n", "with the recently released math-training corpora:\n", "\n", " - **MathPile** (Wang et al., 2023c): a multi-source corpus (8.9B tokens) aggregated from\n", "textbooks, Wikipedia, ProofWiki, CommonCrawl, StackExchange, and arXiv, with the\n", "majority (over 85%) sourced from arXiv;\n", "\n", " - **OpenWebMath** (Paster et al., 2023): CommonCrawl data filtered for mathematical content,\n", "totaling 13.6B tokens;\n", "\n", " - **Proof-Pile-2** (Azerbayev et al., 2023): a mathematical corpus consisting of OpenWebMath, AlgebraicStack (10.3B tokens of mathematical code), and arXiv papers (28.0B tokens). When experimenting on Proof-Pile-2, we follow Azerbayev et al. (2023) to use an\n", "arXiv:Web:Code ratio of 2:4:1.\n", "\n", "_**2.2.1. Training Setting**_\n", "\n", "We apply math training to a general pre-trained language model with 1.3B parameters, which\n", "shares the same framework as the DeepSeek LLMs (DeepSeek-AI, 2024), denoted as DeepSeekLLM 1.3B. We separately train a model on each mathematical corpus for 150B tokens. All\n", "experiments are conducted using the efficient and light-weight HAI-LLM (High-flyer, 2023)\n", "training framework. Following the training practice of DeepSeek LLMs, we use the AdamW\n", "optimizer (Loshchilov and Hutter, 2017) with _\ud835\udefd_ 1 = 0.9, _\ud835\udefd_ 2 = 0.95, and weight_decay = 0.1, along\n", "with a multi-step learning rate schedule where the learning rate reaches the peak after 2,000\n", "warmup steps, decreases to its 31.6% after 80% of the training process, and further decreases to\n", "10.0% of the peak after 90% of the training process. We set the maximum value of learning rate\n", "to 5.3e-4, and use a batch size of 4M tokens with a 4K context length.\n", "\n", "English Benchmarks Chinese Benchmarks\n", "Math Corpus Size MMLU Gaokao Gaokao\n", "\n", "GSM8K MATH OCW SAT CMATH\n", "STEM MathCloze MathQA\n", "\n", "No Math Training N/A 2.9% 3.0% 2.9% 15.6% 19.5% 12.3% 0.8% 17.9%\n", "\n", "MathPile 8.9B 2.7% 3.3% 2.2% 12.5% 15.7% 1.2% 0.0% 2.8%\n", "OpenWebMath 13.6B 11.5% 8.9% 3.7% 31.3% 29.6% 16.8% 0.0% 14.2%\n", "Proof-Pile-2 51.9B 14.3% 11.2% 3.7% 43.8% 29.2% 19.9% 5.1% 11.7%\n", "\n", "DeepSeekMath Corpus **120.2B** **23.8%** **13.6%** **4.8% 56.3%** **33.1%** **41.5%** **5.9%** **23.6%**\n", "\n", "Table 1 | Performance of DeepSeek-LLM 1.3B trained on different mathematical corpora, evaluated using few-shot chain-of-thought prompting. Corpus sizes are calculated using our tokenizer\n", "with a vocabulary size of 100K.\n", "\n", "_**2.2.2. Evaluation Results**_\n", "\n", "**The DeepSeekMath Corpus is of high quality, covers multilingual mathematical content, and**\n", "**is the largest in size.**\n", "\n", " - **High-quality** : We evaluate downstream performance on 8 mathematical benchmarks using\n", "few-shot chain-of-thought prompting Wei et al. (2022). As shown in Table 1, there is a clear\n", "performance lead of the model trained on the DeepSeekMath Corpus. Figure 3 shows that\n", "the model trained on the DeepSeekMath Corpus demonstrates better performance than\n", "\n", "6\n", "\n", "Figure 3 | Benchmark curves of DeepSeek-LLM 1.3B trained on different mathematical corpora.\n", "\n", "Proof-Pile-2 at 50B tokens (1 full epoch of Proof-Pile-2), indicating the average quality of\n", "DeepSeekMath Corpus is higher.\n", "\n", " - **Multilingual** : The DeepSeekMath Corpus encompasses data in multiple languages, predominantly featuring English and Chinese as the two most represented languages. As\n", "shown in Table 1, training on the DeepSeekMath Corpus enhances mathematical reasoning\n", "performance in both English and Chinese. In contrast, existing mathematical corpora,\n", "which are primarily English-centric, show limited improvement and may even hinder\n", "performance in Chinese mathematical reasoning.\n", "\n", " - **Large-scale** : The DeepSeekMath Corpus is several times larger than existing mathematical\n", "corpora. As depicted in Figure 3, DeepSeek-LLM 1.3B, when trained on the DeepSeekMath Corpus, shows a steeper learning curve along with more lasting improvements. In\n", "contrast, the baseline corpora are much smaller, and have already been repeated multiple\n", "rounds during training, with the resulting model performance quickly reaching a plateau.\n", "\n", "**2.3. Training and Evaluating DeepSeekMath-Base 7B**\n", "\n", "In this section, we introduce DeepSeekMath-Base 7B, a base model with strong reasoning\n", "abilities, especially in mathematics. Our model is initialized with DeepSeek-Coder-Base-v1.5 7B\n", "\n", "7\n", "\n", "(Guo et al., 2024) and trained for 500B tokens. The distribution of the data is as follows: 56%\n", "is from the DeepSeekMath Corpus, 4% from AlgebraicStack, 10% from arXiv, 20% is Github\n", "code, and the remaining 10% is natural language data from Common Crawl in both English and\n", "Chinese. We mainly adopt the training setting specified in Section 2.2.1, except that we set the\n", "maximum value of the learning rate to 4.2e-4 and use a batch size of 10M tokens.\n", "\n", "We conduct a comprehensive assessment of the mathematical capabilities of DeepSeekMathBase 7B, focusing on its ability to produce self-contained mathematical solutions without relying\n", "on external tools, solve mathematical problems using tools, and conduct formal theorem proving.\n", "Beyond mathematics, we also provide a more general profile of the base model, including its\n", "performance of natural language understanding, reasoning, and programming skills.\n", "\n", "**Mathematical Problem Solving with Step-by-Step Reasoning** We evaluate DeepSeekMathBase\u2019s performance of solving mathematical problems using few-shot chain-of-thought prompting (Wei et al., 2022), across eight benchmarks in English and Chinese. These benchmarks encompass quantitative reasoning (e.g., GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021),\n", "and CMATH (Wei et al., 2023)) and multiple-choice problems (e.g., MMLU-STEM (Hendrycks\n", "et al., 2020) and Gaokao-MathQA (Zhong et al., 2023)), covering diverse fields of mathematics\n", "from elementary to college-level complexity.\n", "\n", "As shown in Table 2, DeepSeekMath-Base 7B leads in performance across all eight benchmarks among the open-source base models (including the widely-used general model Mistral\n", "7B (Jiang et al., 2023) and the recently released Llemma 34B (Azerbayev et al., 2023) which\n", "underwent math training on Proof-Pile-2 (Azerbayev et al., 2023)). Notably, on the competitionlevel MATH dataset, DeepSeekMath-Base surpasses existing open-source base models by over\n", "10% absolute, and outperforms Minerva 540B (Lewkowycz et al., 2022a), a closed-source base\n", "model 77 times larger which builds on PaLM (Lewkowycz et al., 2022b) and is further trained\n", "on mathematical texts.\n", "\n", "English Benchmarks Chinese Benchmarks\n", "Model Size\n", "\n", "MMLU Gaokao Gaokao\n", "GSM8K MATH OCW SAT CMATH\n", "STEM MathCloze MathQA\n", "\n", "Closed-Source Base Model\n", "\n", "Minerva 7B 16.2% 14.1% 7.7% - 35.6% - - \n", "Minerva 62B 52.4% 27.6% 12.0% - 53.9% - - \n", "Minerva 540B 58.8% 33.6% 17.6% - 63.9% - - \n", "Open-Source Base Model\n", "\n", "Mistral 7B 40.3% 14.3% 9.2% 71.9% 51.1% 44.9% 5.1% 23.4%\n", "\n", "Llemma 7B 37.4% 18.1% 6.3% 59.4% 43.1% 43.4% 11.9% 23.6%\n", "\n", "Llemma 34B 54.0% 25.3% 10.3% 71.9% 52.9% 56.1% 11.9% 26.2%\n", "\n", "DeepSeekMath-Base 7B **64.2%** **36.2% 15.4% 84.4%** **56.5%** **71.7%** **20.3%** **35.3%**\n", "\n", "Table 2 | Comparisons between DeepSeekMath-Base 7B and strong base models on English and\n", "Chinese mathematical benchmarks. Models are evaluated with chain-of-thought prompting.\n", "Minerva results are quoted from Lewkowycz et al. (2022a).\n", "\n", "8\n", "\n", "**Mathematical Problem Solving with Tool Use** We evaluate program-aided mathematical\n", "reasoning on GSM8K and MATH using few-shot program-of-thought prompting (Chen et al.,\n", "2022; Gao et al., 2023). Models are prompted to solve each problem by writing a Python program\n", "where libraries such as _math_ and _sympy_ can be utilized for intricate computations. The execution\n", "result of the program is evaluated as the answer. As shown in Table 3, DeepSeekMath-Base 7B\n", "outperforms the prior state-of-the-art Llemma 34B.\n", "\n", "Problem Solving w/ Tools Informal-to-Formal Proving\n", "Model Size\n", "\n", "GSM8K+Python MATH+Python miniF2F-valid miniF2F-test\n", "\n", "Mistral 7B 48.5% 18.2% 18.9% 18.0%\n", "\n", "CodeLlama 7B 27.1% 17.2% 16.3% 17.6%\n", "\n", "CodeLlama 34B 52.7% 23.5% 18.5% 18.0%\n", "\n", "Llemma 7B 41.0% 18.6% 20.6% 22.1%\n", "\n", "Llemma 34B 64.6% 26.3% 21.0% 21.3%\n", "\n", "DeepSeekMath-Base 7B **66.9%** **31.4%** **25.8%** **24.6%**\n", "\n", "Table 3 | Few-shot evaluation of base models\u2019 ability to solve mathematical problems using tools\n", "and the ability to conduct informal-to-formal theorem proving in Isabelle.\n", "\n", "**Formal Mathematics** Formal proof automation is beneficial to ensure the accuracy and reliability of mathematical proofs and enhance efficiency, with increasing attention in recent years.\n", "We evaluate DeepSeekMath-Base 7B on the task of informal-to-formal proving from (Jiang et al.,\n", "2022) which is to generate a formal proof based on an informal statement, a formal counterpart\n", "of the statement, and an informal proof. We evaluate on miniF2F (Zheng et al., 2021), a benchmark for formal Olympiad-level mathematics, and generate a formal proof in Isabelle for each\n", "problem with few-shot prompting. Following Jiang et al. (2022), we leverage models to generate\n", "proof sketches, and execute the off-the-shelf automated prover Sledgehammer (Paulson, 2010)\n", "to fill in the missing details. As shown in Table 3, DeepSeekMath-Base 7B demonstrates strong\n", "performance in proof autoformalization.\n", "\n", "Model Size MMLU BBH HumanEval (Pass@1) MBPP (Pass@1)\n", "\n", "Mistral 7B **62.4%** 55.7% 28.0% 41.4%\n", "\n", "DeepSeek-Coder-Base-v1.5 [\u2020] 7B 42.9% 42.9% 40.2% 52.6%\n", "DeepSeek-Coder-Base-v1.5 7B 49.1% 55.2% **43.2%** **60.4%**\n", "\n", "DeepSeekMath-Base 7B 54.9% **59.5%** 40.9% 52.6%\n", "\n", "Table 4 | Evaluation on natural language understanding, reasoning, and code benchmarks.\n", "DeepSeek-Coder-Base-v1.5 [\u2020] is the checkpoint right before learning rate decay, which is used to\n", "train DeepSeekMath-Base. On MMLU and BBH, we use few-shot chain-of-thought prompting.\n", "On HumanEval and MBPP, we evaluate model performance under the zero-shot setting and a\n", "few-shot setting, respectively.\n", "\n", "**Natural Language Understanding, Reasoning, and Code** We evaluate model performance of\n", "natural language understanding on MMLU (Hendrycks et al., 2020), reasoning on BBH (Suzgun\n", "et al., 2022), and coding capabilities on HumanEval (Chen et al., 2021) and MBPP (Austin et al.,\n", "\n", "9\n", "\n", "2021). As shown in Table 4, DeepSeekMath-Base 7B exhibits significant enhancements in performance on MMLU and BBH over its precursor, DeepSeek-Coder-Base-v1.5 (Guo et al., 2024),\n", "illustrating the positive impact of math training on language understanding and reasoning.\n", "Additionally, by including code tokens for continual training, DeepSeekMath-Base 7B effectively\n", "maintains the performance of DeepSeek-Coder-Base-v1.5 on the two coding benchmarks. Overall, DeepSeekMath-Base 7B significantly outperforms the general model Mistral 7B (Jiang et al.,\n", "2023) on the three reasoning and coding benchmarks."]}, "transient": {}}], "execution_count": null, "attachments": {}}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what are `URL-based deduplication and near-deduplication techniques`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "URL-based deduplication and near-deduplication are techniques used to remove duplicate or very similar web pages from large datasets like Common Crawl.\n", "\n", "**URL-based deduplication** removes exact duplicates by identifying web pages that have identical URLs. This catches cases where the same page appears multiple times in the dataset.\n", "\n", "**Near-deduplication** is more sophisticated - it identifies pages that are almost identical but not exactly the same. This might catch things like:\n", "- The same content with minor formatting differences\n", "- Pages with identical text but different timestamps or small updates\n", "- Mirror sites hosting the same content\n", "\n", "These techniques are important when building training datasets because you don't want your AI model to see the same content repeatedly, which could lead to overfitting or skewed learning.\n", "\n", "In the DeepSeekMath paper, they used these methods to reduce Common Crawl from its original size down to 40 billion HTML pages before applying their mathematical content filtering.\n", "\n", "Are you curious about how these deduplication algorithms actually work, or more interested in why they're important for training AI models?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what do they mean by `organize the entire Common Crawl into disjoint domains; a domain is defined as web pages sharing the same base URL.`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "They're grouping web pages by their base URL to organize them into separate categories.\n", "\n", "A **base URL** is the main part of a website address. For example:\n", "- `mathoverflow.net` is the base URL\n", "- `mathoverflow.net/questions/12345` and `mathoverflow.net/users/jane` would both belong to the same domain\n", "\n", "**Disjoint domains** means each web page belongs to exactly one group - no overlaps. So all pages from `mathoverflow.net` go in one bucket, all pages from `wikipedia.org` go in another bucket, etc.\n", "\n", "This helps them analyze patterns. They can say \"10% of all pages from mathoverflow.net were identified as math-related in our first pass, so this whole domain seems math-focused.\"\n", "\n", "Does this grouping strategy make sense? Are you wondering about the next step in their process?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what did the ydo next?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "After organizing Common Crawl into domains, they looked for domains where more than 10% of pages were already collected as math-related in their first iteration.\n", "\n", "When they found these math-heavy domains (like `mathoverflow.net`), they manually examined the URL patterns within those domains to identify which specific sections contained mathematical content.\n", "\n", "For example, they might notice that `mathoverflow.net/questions` has math content, but `mathoverflow.net/about` doesn't.\n", "\n", "Then they added the uncollected pages from these mathematical URL patterns to their seed corpus, which helped them train a better classifier for the next iteration.\n", "\n", "Does this step make sense? Are you curious about what happened in the subsequent iterations?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what happens in the subsequent iterations?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "In the subsequent iterations, they repeated the same process with their improved classifier:\n", "\n", "1. They retrained the fastText model using the expanded seed corpus (which now included more diverse math examples)\n", "2. This better classifier could find more mathematical web pages that were missed in earlier rounds\n", "3. They continued this cycle - each iteration finding more math content and improving the classifier\n", "\n", "After four iterations total, they ended up with 35.5 million mathematical web pages (120B tokens). By the fourth iteration, they found that 98% of the data had already been collected in the third iteration, so they stopped there.\n", "\n", "The key insight is that each iteration made their classifier better at recognizing different types of mathematical content, allowing them to cast a wider net each time.\n", "\n", "Would you like to know more about how they validated the quality of this final corpus, or are you curious about another aspect of their data collection process?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what do they mean by `For benchmark texts that are shorter than 10 grams but have at least 3 grams, we employ exact matching to filter out contaminated web pages`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "They're describing how they removed test questions from their training data to avoid \"cheating.\"\n", "\n", "An **n-gram** is a sequence of n consecutive words. So:\n", "- A 3-gram might be \"solve this equation\"\n", "- A 10-gram would be 10 words in a row\n", "\n", "Their filtering rule works like this:\n", "- If a benchmark question has 10+ words: they check if any 10-word sequence from the question appears exactly in their training data\n", "- If a benchmark question has 3-9 words: they check if the entire question appears exactly in their training data\n", "\n", "This prevents their model from just memorizing test answers during training, which would make evaluation results misleading.\n", "\n", "Does this contamination filtering approach make sense to you?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "I don't see how 10-gram differs from 3-9 words approach. Are they the same?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Good catch! You're right to question this - let me clarify the difference.\n", "\n", "For **10+ word questions**:\n", "- They don't check the entire question\n", "- They break it into sliding 10-word chunks and check if ANY of those chunks appear in training data\n", "- Example: A 15-word question gets checked as multiple overlapping 10-word sequences\n", "\n", "For **3-9 word questions**:\n", "- They check if the ENTIRE question appears exactly in training data\n", "- No chunking - it's an all-or-nothing match\n", "\n", "So a long question might be contaminated if even part of it appears in training data, while a short question is only contaminated if the whole thing appears.\n", "\n", "Does this distinction make more sense now?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what is `HAI-LLM (High-flyer, 2023) training framework`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "HAI-LLM appears to be a training framework developed by High-flyer in 2023 that the DeepSeekMath team used for their experiments. The paper describes it as \"efficient and light-weight.\"\n", "\n", "From the context, it seems to be a system designed to make training large language models more resource-efficient, but the paper doesn't provide detailed technical specifications about how it works internally.\n", "\n", "Are you curious about training frameworks in general, or wondering about other technical details from the DeepSeekMath training process?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what is chain-of-thought prompting?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Chain-of-thought prompting is a technique where you ask an AI model to show its reasoning steps when solving a problem, rather than just giving the final answer.\n", "\n", "For example, instead of just asking \"What's 15% of 80?\", you might prompt: \"What's 15% of 80? Let's think step by step.\"\n", "\n", "This encourages the model to write out something like:\n", "- First, convert 15% to decimal: 0.15\n", "- Then multiply: 0.15 \u00d7 80 = 12\n", "- So 15% of 80 is 12\n", "\n", "The \"chain\" refers to the linked sequence of reasoning steps that lead to the answer.\n", "\n", "Would you like to know why this technique is effective, or are you curious about how it was used specifically in the DeepSeekMath evaluation?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Is it the same thing as thinking mode?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Chain-of-thought prompting and \"thinking mode\" are related but not exactly the same thing.\n", "\n", "Chain-of-thought is a prompting technique where you explicitly ask the model to show its reasoning steps in the output that you see.\n", "\n", "\"Thinking mode\" (like in some recent AI systems) typically refers to an internal reasoning process where the model works through problems privately before giving you the final response - you might not see all the intermediate steps.\n", "\n", "Both aim to improve reasoning quality, but chain-of-thought makes the reasoning visible as part of the answer, while thinking mode keeps much of the reasoning hidden.\n", "\n", "Are you thinking about a specific \"thinking mode\" implementation, or curious about how these approaches compare in effectiveness?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "when did thinking mode come out?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "OpenAI introduced their \"o1\" models with reasoning/thinking capabilities in September 2024. These models show their reasoning process in a collapsed \"thinking\" section that you can expand to see.\n", "\n", "There have been other approaches to internal reasoning in AI systems before that, but the prominent \"thinking mode\" you're likely referring to is relatively recent.\n", "\n", "Are you interested in how this compares to the chain-of-thought approach used in the DeepSeekMath paper, or curious about other aspects of these reasoning techniques?\n", "\n"]}, {"cell_type": "code", "metadata": {}, "source": ["display_section(4)"], "outputs": [{"output_type": "display_data", "metadata": {}, "data": {"text/plain": ["<IPython.core.display.Markdown object>"], "text/markdown": ["##### **3. Supervised Fine-Tuning**\n", "\n", "**3.1. SFT Data Curation**\n", "\n", "We construct a mathematical instruction-tuning dataset covering English and Chinese problems\n", "from different mathematical fields and of varying complexity levels: problems are paired with\n", "solutions in chain-of-thought (CoT) (Wei et al., 2022), program-of-thought (PoT) (Chen et al.,\n", "2022; Gao et al., 2023), and tool-integrated reasoning format (Gou et al., 2023). The total number\n", "of training examples is 776K.\n", "\n", " - **English mathematical datasets** : We annotate GSM8K and MATH problems with toolintegrated solutions, and adopt a subset of MathInstruct (Yue et al., 2023) along with the\n", "training set of Lila-OOD (Mishra et al., 2022) where problems are solved with CoT or\n", "PoT. Our English collection covers diverse fields of mathematics, e.g., algebra, probability,\n", "number theory, calculus, and geometry.\n", "\n", " - **Chinese mathematical datasets** : We collect Chinese K-12 mathematical problems spanning\n", "76 sub-topics such as linear equations, with solutions annotated in both CoT and toolintegrated reasoning format.\n", "\n", "**3.2. Training and Evaluating DeepSeekMath-Instruct 7B**\n", "\n", "In this section, we introduce DeepSeekMath-Instruct 7B which undergoes mathematical instruction tuning based on DeepSeekMath-Base. Training examples are randomly concatenated until\n", "reaching a maximum context length of 4K tokens. We train the model for 500 steps with a batch\n", "size of 256 and a constant learning rate of 5e-5.\n", "\n", "We evaluate models\u2019 mathematical performance both without and with tool use, on 4\n", "quantitative reasoning benchmarks in English and Chinese. We benchmark our model against\n", "the leading models of the time:\n", "\n", " - **Closed-source models** include: (1) the GPT family among which GPT-4 (OpenAI, 2023)\n", "and GPT-4 Code Interpreter [2] are the most capable ones, (2) Gemini Ultra and Pro (Anil\n", "et al., 2023), (3) Inflection-2 (Inflection AI, 2023), (4) Grok-1 [3], as well as models recently\n", "released by Chinese companies including (5) Baichuan-3 [4], (6) the latest GLM-4 [5] from the\n", "GLM family (Du et al., 2022). These models are for general purposes, most of which have\n", "undergone a series of alignment procedures.\n", "\n", " - **Open-source models** include: general models like (1) DeepSeek-LLM-Chat 67B (DeepSeekAI, 2024), (2) Qwen 72B (Bai et al., 2023), (3) SeaLLM-v2 7B (Nguyen et al., 2023), and (4)\n", "\n", "2 `[https://openai.com/blog/chatgpt-plugins#code-interpreter](https://openai.com/blog/chatgpt-plugins##code-interpreter)`\n", "3 `[https://x.ai/model-card](https://x.ai/model-card)`\n", "4 `[https://www.baichuan-ai.com](https://www.baichuan-ai.com)`\n", "5 `[https://open.bigmodel.cn/dev/api#glm-4](https://open.bigmodel.cn/dev/api#glm-4)`\n", "\n", "10\n", "\n", "ChatGLM3 6B (ChatGLM3 Team, 2023), as well as models with enhancements in mathematics including (5) InternLM2-Math 20B [6] which builds on InternLM2 and underwent math\n", "training followed by instruction tuning, (6) Math-Shepherd-Mistral 7B which applys PPO\n", "training (Schulman et al., 2017) to Mistral 7B (Jiang et al., 2023) with a process-supervised\n", "reward model, (7) the WizardMath series (Luo et al., 2023) which improves mathematical\n", "reasoning in Mistral 7B and Llama-2 70B (Touvron et al., 2023) using evolve-instruct (i.e.,\n", "a version of instruction tuning that uses AI-evolved instructions) and PPO training with\n", "training problems primarily sourced from GSM8K and MATH, (8) MetaMath 70B (Yu et al.,\n", "2023) which is Llama-2 70B fine-tuned on an augmented version of GSM8K and MATH,\n", "(9) ToRA 34B Gou et al. (2023) which is CodeLlama 34B fine-tuned to do tool-integrated\n", "mathematical reasoning, (10) MAmmoTH 70B (Yue et al., 2023) which is Llama-2 70B\n", "instruction-tuned on MathInstruct.\n", "\n", "As shown in Table 5, under the evaluation setting where tool use is disallowed, DeepSeekMathInstruct 7B demonstrates strong performance of step-by-step reasoning. Notably, on the\n", "competition-level MATH dataset, our model surpasses all open-source models and the majority of proprietary models (e.g., Inflection-2 and Gemini Pro) by at least 9% absolute. This\n", "is true even for models that are substantially larger (e.g., Qwen 72B) or have been specifically enhanced through math-focused reinforcement learning (e.g., WizardMath-v1.1 7B). While\n", "DeepSeekMath-Instruct rivals the Chinese proprietary models GLM-4 and Baichuan-3 on MATH,\n", "it still underperforms GPT-4 and Gemini Ultra.\n", "\n", "Under the evaluation setting where models are allowed to integrate natural language reasoning and program-based tool use for problem solving, DeepSeekMath-Instruct 7B approaches\n", "an accuracy of 60% on MATH, surpassing all existing open-source models. On the other benchmarks, our model is competitive with DeepSeek-LLM-Chat 67B, the prior state-of-the-art that is\n", "10 times larger."]}, "transient": {}}], "execution_count": null, "attachments": {}}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "What do they mean `annotate` in `We annotate GSM8K and MATH problems with toolintegrated solutions, and adopt a subset of MathInstruct (Yue et al., 2023) along with the training set of Lila-OOD (Mishra et al., 2022) where problems are solved with CoT or PoT`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "\"Annotate\" here means they manually added new solution formats to existing problems.\n", "\n", "The GSM8K and MATH datasets originally came with basic solutions, but the DeepSeekMath team wrote new \"tool-integrated solutions\" for these problems. Tool-integrated solutions show how to solve the problem by combining reasoning steps with code/computational tools.\n", "\n", "So they took existing math problems and created new, more detailed solution examples that demonstrate using tools like Python calculations alongside written explanations.\n", "\n", "This gave them training data that teaches the model how to blend natural language reasoning with computational tools.\n", "\n", "Does this distinction between the original solutions and their new annotated versions make sense?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what does it mean tool-integrated solutions?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Tool-integrated solutions combine written reasoning with computational tools to solve problems.\n", "\n", "For example, a regular solution might say: \"To find 15% of 240, we calculate 0.15 \u00d7 240 = 36\"\n", "\n", "A tool-integrated solution would show:\n", "- Written explanation: \"I need to find 15% of 240\"\n", "- Code: `result = 0.15 * 240`\n", "- Written conclusion: \"So the answer is 36\"\n", "\n", "The model learns to seamlessly switch between explaining the approach in natural language and using Python code for calculations.\n", "\n", "Does this help clarify what they mean by integrating tools into the solution process?\n", "\n"]}, {"cell_type": "code", "metadata": {}, "source": ["display_section(5)"], "outputs": [{"output_type": "display_data", "metadata": {}, "data": {"text/plain": ["<IPython.core.display.Markdown object>"], "text/markdown": ["##### **4. Reinforcement Learning**\n", "\n", "**4.1. Group Relative Policy Optimization**\n", "\n", "Reinforcement learning (RL) has been proven to be effective in further improving the mathematical reasoning ability of LLMs after the Supervised Fine-Tuning (SFT) stage (Luo et al., 2023;\n", "Wang et al., 2023b). In this section, we introduce our efficient and effective RL algorithm, Group\n", "Relative Policy Optimization (GRPO).\n", "\n", "_**4.1.1. From PPO to GRPO**_\n", "\n", "Proximal Policy Optimization (PPO) (Schulman et al., 2017) is an actor-critic RL algorithm that is\n", "widely used in the RL fine-tuning stage of LLMs (Ouyang et al., 2022). In particular, it optimizes\n", "LLMs by maximizing the following surrogate objective:\n", "\n", "\n", "_\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _<\ud835\udc61_ ) _\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _<\ud835\udc61_ )\n", "\n", "_\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ ( _\ud835\udc5c_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _<\ud835\udc61_ ) _[\ud835\udc34]_ _[\ud835\udc61]_ [, clip] \ufffd _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ ( _\ud835\udc5c_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _<\ud835\udc61_\n", "\n", "\n", "_\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _<\ud835\udc61_ )\n", "\n", "_\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ ( _\ud835\udc5c_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _<\ud835\udc61_ ) [, 1][ \u2212] _[\ud835\udf00]_ [, 1][ +] _[ \ud835\udf00]_ \ufffd\n", "\n", "\n", "J _\ud835\udc43\ud835\udc43\ud835\udc42_ ( _\ud835\udf03_ ) = **E** [ _\ud835\udc5e_ \u223c _\ud835\udc43_ ( _\ud835\udc44_ ), _\ud835\udc5c_ \u223c _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ ( _\ud835\udc42_ | _\ud835\udc5e_ )] | [1] _\ud835\udc5c_ |\n", "\n", "\n", "| _\ud835\udc5c_ |\n", "\u2211\ufe01\n", "\n", "\n", "_\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _<\ud835\udc61_ )\n", "min\n", "\n", "\u2211\ufe01 _\ud835\udc61_ =1 \ufffd _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ ( _\ud835\udc5c_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _<\ud835\udc61_\n", "\n", "\n", "_\ud835\udc34_ _\ud835\udc61_\n", "\n", "\n", ", (1)\n", "\ufffd\n", "\n", "\n", "where _\ud835\udf0b_ _\ud835\udf03_ and _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ are the current and old policy models, and _\ud835\udc5e_, _\ud835\udc5c_ are questions and outputs\n", "sampled from the question dataset and the old policy _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_, respectively. _\ud835\udf00_ is a clipping-related\n", "hyper-parameter introduced in PPO for stabilizing training. _\ud835\udc34_ _\ud835\udc61_ is the advantage, which is\n", "computed by applying Generalized Advantage Estimation (GAE) (Schulman et al., 2015), based\n", "\n", "6 `[https://github.com/InternLM/InternLM-Math](https://github.com/InternLM/InternLM-Math)`\n", "\n", "11\n", "\n", "Model Size [En][g][lish Benchmarks Chinese Benchmarks]\n", "\n", "GSM8K MATH MGSM-zh CMATH\n", "\n", "**Chain-of-Thought Reasoning**\n", "\n", "Closed-Source Model\n", "\n", "Gemini Ultra     - 94.4% 53.2%     -     \n", "GPT-4     - 92.0% 52.9%     - 86.0%\n", "Inflection-2      - 81.4% 34.8%      -      GPT-3.5     - 80.8% 34.1%     - 73.8%\n", "\n", "Gemini Pro     - 86.5% 32.6%     -     \n", "Grok-1     - 62.9% 23.9%     -     \n", "Baichuan-3     - 88.2% 49.2%     -     \n", "GLM-4    - 87.6% 47.9%    -    \n", "Open-Source Model\n", "\n", "InternLM2-Math 20B 82.6% 37.7%     -     \n", "Qwen 72B 78.9% 35.2%    -    Math-Shepherd-Mistral 7B 84.1% 33.0%     -     WizardMath-v1.1 7B 83.2% 33.0%     -     DeepSeek-LLM-Chat 67B 84.1% 32.6% 74.0% 80.3%\n", "MetaMath 70B 82.3% 26.6% 66.4% 70.9%\n", "\n", "SeaLLM-v2 7B 78.2% 27.5% 64.8%     \n", "ChatGLM3 6B 72.3% 25.7%    -    \n", "WizardMath-v1.0 70B 81.6% 22.7% 64.8% 65.4%\n", "\n", "**DeepSeekMath-Instruct** 7B 82.9% 46.8% 73.2% 84.6%\n", "**DeepSeekMath-RL** 7B **88.2%** **51.7%** **79.6%** **88.8%**\n", "\n", "**Tool-Integrated Reasoning**\n", "\n", "Closed-Source Model\n", "\n", "GPT-4 Code Interpreter     - 97.0% 69.7%     -     \n", "Open-Source Model\n", "\n", "InternLM2-Math 20B 80.7% 54.3%     -     DeepSeek-LLM-Chat 67B 86.7% 51.1% 76.4% 85.4%\n", "ToRA 34B 80.7% 50.8% 41.2% 53.4%\n", "\n", "MAmmoTH 70B 76.9% 41.8%    -    \n", "**DeepSeekMath-Instruct** 7B 83.7% 57.4% 72.0% 84.3%\n", "**DeepSeekMath-RL** 7B **86.7%** **58.8%** **78.4%** **87.6%**\n", "\n", "Table 5 | Performance of Open- and Closed-Source models with both Chain-of-Thought and\n", "Tool-Integrated Reasoning on English and Chinese Benchmarks. Scores in gray denote majority\n", "votes with 32 candidates; The others are Top1 scores. DeepSeekMath-RL 7B beats all opensource models from 7B to 70B, as well as the majority of closed-source models. Although\n", "DeepSeekMath-RL 7B is only further trained on chain-of-thought-format instruction tuning data\n", "of GSM8K and MATH, it improves over DeepSeekMath-Instruct 7B on all benchmarks.\n", "\n", "12\n", "\n", "Figure 4 | Demonstration of PPO and our GRPO. GRPO foregoes the value model, instead\n", "estimating the baseline from group scores, significantly reducing training resources.\n", "\n", "on the rewards { _\ud835\udc5f_ \u2265 _\ud835\udc61_ } and a learned value function _\ud835\udc49_ _\ud835\udf13_ . Thus, in PPO, a value function needs to\n", "be trained alongside the policy model and to mitigate over-optimization of the reward model,\n", "the standard approach is to add a per-token KL penalty from a reference model in the reward at\n", "each token (Ouyang et al., 2022), i.e.,\n", "\n", "_\ud835\udc5f_ _\ud835\udc61_ = _\ud835\udc5f_ _\ud835\udf11_ ( _\ud835\udc5e_, _\ud835\udc5c_ \u2264 _\ud835\udc61_ ) \u2212 _\ud835\udefd_ log _\ud835\udf0b_ _[\ud835\udf0b]_ _\ud835\udc5f\ud835\udc52\ud835\udc53_ _[\ud835\udf03]_ [(] ( _[\ud835\udc5c]_ _\ud835\udc5c_ _[\ud835\udc61]_ _\ud835\udc61_ [|] _[\ud835\udc5e]_ | _\ud835\udc5e_ [,] _[ \ud835\udc5c]_, _\ud835\udc5c_ _[<\ud835\udc61]_ _<\ud835\udc61_ [)] ) [,] (2)\n", "\n", "where _\ud835\udc5f_ _\ud835\udf11_ is the reward model, _\ud835\udf0b_ _\ud835\udc5f\ud835\udc52\ud835\udc53_ is the reference model, which is usually the initial SFT model,\n", "and _\ud835\udefd_ is the coefficient of the KL penalty.\n", "\n", "As the value function employed in PPO is typically another model of comparable size as\n", "the policy model, it brings a substantial memory and computational burden. Additionally,\n", "during RL training, the value function is treated as a baseline in the calculation of the advantage\n", "for variance reduction. While in the LLM context, usually only the last token is assigned a\n", "reward score by the reward model, which may complicate the training of a value function that is\n", "accurate at each token. To address this, as shown in Figure 4, we propose Group Relative Policy\n", "Optimization (GRPO), which obviates the need for additional value function approximation as\n", "in PPO, and instead uses the average reward of multiple sampled outputs, produced in response\n", "to the same question, as the baseline. More specifically, for each question _\ud835\udc5e_, GRPO samples a\n", "group of outputs { _\ud835\udc5c_ 1, _\ud835\udc5c_ 2, - \u00b7 \u00b7, _\ud835\udc5c_ _\ud835\udc3a_ } from the old policy _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ and then optimizes the policy model\n", "by maximizing the following objective:\n", "\n", "J _\ud835\udc3a\ud835\udc45\ud835\udc43\ud835\udc42_ ( _\ud835\udf03_ ) = **E** [ _\ud835\udc5e_ \u223c _\ud835\udc43_ ( _\ud835\udc44_ ), { _\ud835\udc5c_ _\ud835\udc56_ } _[\ud835\udc3a]_ _\ud835\udc56_ =1 [\u223c] _[\ud835\udf0b]_ _[\ud835\udf03]_ _[\ud835\udc5c\ud835\udc59\ud835\udc51]_ [(] _[\ud835\udc42]_ [|] _[\ud835\udc5e]_ [)]]\n", "\n", "\n", "_\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_, _<\ud835\udc61_ ) _\ud835\udc34_ \u02c6 _\ud835\udc56_, _\ud835\udc61_\n", "\n", "_\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_, _<\ud835\udc61_ ) [, 1][ \u2212] _[\ud835\udf00]_ [, 1][ +] _[ \ud835\udf00]_ \ufffd\n", "\n", "\n", "(3)\n", "\u2212 _\ud835\udefd_ **D** _\ud835\udc3e\ud835\udc3f_ \ufffd _\ud835\udf0b_ _\ud835\udf03_ || _\ud835\udf0b_ _\ud835\udc5f\ud835\udc52\ud835\udc53_ \ufffd [\ufffd],\n", "\ufffd\n", "\n", "\n", "1\n", "\n", "_\ud835\udc3a_\n", "\n", "\n", "_\ud835\udc3a_\n", "\u2211\ufe01\n", "\n", "_\ud835\udc56_ =1\n", "\n", "\n", "1\n", "\n", "| _\ud835\udc5c_ _\ud835\udc56_ |\n", "\n", "\n", "| _\ud835\udc5c_ _\ud835\udc56_ |\n", "\u2211\ufe01\n", "\n", "_\ud835\udc61_ =1\n", "\n", "\n", "min _\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_, _<\ud835\udc61_ )\n", "\ufffd \ufffd _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ ( _\ud835\udc5c_ _\ud835\udc56_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_ _<\ud835\udc61_\n", "\n", "\n", "_\ud835\udf0b\ud835\udf0b_ _\ud835\udf03\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ ( _\ud835\udc5c_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc56\ud835\udc61_, | _\ud835\udc61_ _\ud835\udc5e_ | _\ud835\udc5e_, _\ud835\udc5c_, _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc56<\ud835\udc61_, _<\ud835\udc61_ )) _\ud835\udc34_ \u02c6 _\ud835\udc56_, _\ud835\udc61_, clip \ufffd _\ud835\udf0b\ud835\udf0b_ _\ud835\udf03\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ ( _\ud835\udc5c_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc56\ud835\udc61_, | _\ud835\udc61_ _\ud835\udc5e_ | _\ud835\udc5e_, _\ud835\udc5c_, _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc56<\ud835\udc61_, _<\ud835\udc61_ )\n", "\n", "\n", "where _\ud835\udf00_ and _\ud835\udefd_ are hyper-parameters, and _\ud835\udc34_ [\u02c6] _\ud835\udc56_, _\ud835\udc61_ is the advantage calculated based on relative\n", "rewards of the outputs inside each group only, which will be detailed in the following subsections. The group relative way that GRPO leverages to calculate the advantages, aligns well with\n", "the comparative nature of rewards models, as reward models are typically trained on datasets\n", "of comparisons between outputs on the same question. Also note that, instead of adding KL\n", "penalty in the reward, GRPO regularizes by directly adding the KL divergence between the\n", "trained policy and the reference policy to the loss, avoiding complicating the calculation of _\ud835\udc34_ [\u02c6] _\ud835\udc56_, _\ud835\udc61_ .\n", "\n", "13\n", "\n", "**Algorithm 1** Iterative Group Relative Policy Optimization\n", "**Input** initial policy model _\ud835\udf0b_ _\ud835\udf03_ init ; reward models _\ud835\udc5f_ _\ud835\udf11_ ; task prompts D; hyperparameters _\ud835\udf00_, _\ud835\udefd_, _\ud835\udf07_\n", "\n", "1: policy model _\ud835\udf0b_ _\ud835\udf03_ \u2190 _\ud835\udf0b_ _\ud835\udf03_ init\n", "2: **for** iteration = 1, ..., I **do**\n", "3: reference model _\ud835\udf0b_ _\ud835\udc5f\ud835\udc52\ud835\udc53_ \u2190 _\ud835\udf0b_ _\ud835\udf03_\n", "4: **for** step = 1, ..., M **do**\n", "5: Sample a batch D _\ud835\udc4f_ from D\n", "6: Update the old policy model _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ \u2190 _\ud835\udf0b_ _\ud835\udf03_\n", "7: Sample _\ud835\udc3a_ outputs { _\ud835\udc5c_ _\ud835\udc56_ } _[\ud835\udc3a]_ _\ud835\udc56_ =1 [\u223c] _[\ud835\udf0b]_ _[\ud835\udf03]_ _[\ud835\udc5c\ud835\udc59\ud835\udc51]_ [(\u00b7 |] _[ \ud835\udc5e]_ [)][ for each question] _[ \ud835\udc5e]_ [\u2208D] _[\ud835\udc4f]_\n", "8: Compute rewards { _\ud835\udc5f_ _\ud835\udc56_ } _[\ud835\udc3a]_ _\ud835\udc56_ =1 [for each sampled output] _[ \ud835\udc5c]_ _[\ud835\udc56]_ [by running] _[ \ud835\udc5f]_ _[\ud835\udf11]_\n", "9: Compute _\ud835\udc34_ [\u02c6] _\ud835\udc56_, _\ud835\udc61_ for the _\ud835\udc61_ -th token of _\ud835\udc5c_ _\ud835\udc56_ through group relative advantage estimation.\n", "10: **for** GRPO iteration = 1, ..., _\ud835\udf07_ **do**\n", "11: Update the policy model _\ud835\udf0b_ _\ud835\udf03_ by maximizing the GRPO objective (Equation 21)\n", "\n", "12: Update _\ud835\udc5f_ _\ud835\udf11_ through continuous training using a replay mechanism.\n", "\n", "**Output** _\ud835\udf0b_ _\ud835\udf03_\n", "\n", "And different from the KL penalty term used in (2), we estimate the KL divergence with the\n", "following unbiased estimator (Schulman, 2020):\n", "\n", "\n", "_\ud835\udf0b_ _\ud835\udc5f\ud835\udc52\ud835\udc53_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_, _<\ud835\udc61_ )\n", "**D** _\ud835\udc3e\ud835\udc3f_ \ufffd _\ud835\udf0b_ _\ud835\udf03_ || _\ud835\udf0b_ _\ud835\udc5f\ud835\udc52\ud835\udc53_ \ufffd =\n", "\n", "\n", ",, (4)\n", "\n", "_\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_, _<\ud835\udc61_ ) [\u2212] [1,]\n", "\n", "\n", "_\ud835\udc5f\ud835\udc52\ud835\udc53_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_, _<\ud835\udc61_ ) _\ud835\udf0b_ _\ud835\udc5f\ud835\udc52\ud835\udc53_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_, _<\ud835\udc61_ )\n", "\n", "_\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_, _<\ud835\udc61_ ) [\u2212] [log] _\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc56_, _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _\ud835\udc56_, _<\ud835\udc61_\n", "\n", "\n", "which is guaranteed to be positive.\n", "\n", "_**4.1.2. Outcome Supervision RL with GRPO**_\n", "\n", "Formally, for each question _\ud835\udc5e_, a group of outputs { _\ud835\udc5c_ 1, _\ud835\udc5c_ 2, - \u00b7 \u00b7, _\ud835\udc5c_ _\ud835\udc3a_ } are sampled from the old\n", "policy model _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc5c\ud835\udc59\ud835\udc51_ . A reward model is then used to score the outputs, yielding _\ud835\udc3a_ rewards\n", "**r** = { _\ud835\udc5f_ 1, _\ud835\udc5f_ 2, - \u00b7 \u00b7, _\ud835\udc5f_ _\ud835\udc3a_ } correspondingly. Subsequently, these rewards are normalized by subtracting\n", "the group average and dividing by the group standard deviation. Outcome supervision provides\n", "the normalized reward at the end of each output _\ud835\udc5c_ _\ud835\udc56_ and sets the advantages _\ud835\udc34_ [\u02c6] _\ud835\udc56_, _\ud835\udc61_ of all tokens in\n", "the output as the normalized reward, i.e., _\ud835\udc34_ [\u02c6] _\ud835\udc56_, _\ud835\udc61_ = \ufffd _\ud835\udc5f_ _\ud835\udc56_ = _[\ud835\udc5f]_ _[\ud835\udc56]_ [\u2212] std [mean] ( **r** ) [(] **[r]** [)], and then optimizes the policy by\n", "\n", "maximizing the objective defined in equation (3).\n", "\n", "_**4.1.3. Process Supervision RL with GRPO**_\n", "\n", "Outcome supervision only provides a reward at the end of each output, which may not be\n", "sufficient and efficient to supervise the policy in complex mathematical tasks. Following Wang\n", "et al. (2023b), we also explore process supervision, which provides a reward at the end of\n", "each reasoning step. Formally, given the question _\ud835\udc5e_ and _\ud835\udc3a_ sampled outputs { _\ud835\udc5c_ 1, _\ud835\udc5c_ 2, - \u00b7 \u00b7, _\ud835\udc5c_ _\ud835\udc3a_ }, a\n", "process reward model is used to score each step of the outputs, yielding corresponding rewards:\n", "**R** = {{ _\ud835\udc5f_ 1 _[\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc52\ud835\udc65]_ [(][1][)], - \u00b7 \u00b7, _\ud835\udc5f_ 1 _[\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc52\ud835\udc65]_ [(] _[\ud835\udc3e]_ [1] [)] }, - \u00b7 \u00b7, { _\ud835\udc5f_ _\ud835\udc3a_ _[\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc52\ud835\udc65]_ [(][1][)], - \u00b7 \u00b7, _\ud835\udc5f_ _\ud835\udc3a_ _[\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc52\ud835\udc65]_ [(] _[\ud835\udc3e]_ _[\ud835\udc3a]_ [)] }}, where _\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc52\ud835\udc65_ ( _\ud835\udc57_ ) is the end token index\n", "of the _\ud835\udc57_ -th step, and _\ud835\udc3e_ _\ud835\udc56_ is the total number of steps in the _\ud835\udc56_ -th output. We also normalize these\n", "\n", "\ufffd _\ud835\udc5f_ _[\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc52\ud835\udc65]_ [(] _[ \ud835\udc57]_ [)] = _\ud835\udc5f_ _\ud835\udc56_ _[\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc52\ud835\udc65]_ [(] _[ \ud835\udc57]_ [)] \u2212mean( **R** )\n", "rewards with the average and the standard deviation, i.e., _\ud835\udc56_ std( **R** ) . Subsequently,\n", "the process supervision calculates the advantage of each token as the sum of the normalized\n", "rewards from the following steps, i.e., _\ud835\udc34_ [\u02c6] _\ud835\udc56_, _\ud835\udc61_ = [\ufffd] _\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc52\ud835\udc65_ ( _\ud835\udc57_ )\u2265 _\ud835\udc61_ [\ufffd] _[\ud835\udc5f]_ _\ud835\udc56_ _[\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc52\ud835\udc65]_ [(] _[ \ud835\udc57]_ [)], and then optimizes the policy by\n", "maximizing the objective defined in equation (3).\n", "\n", "14\n", "\n", "_**4.1.4. Iterative RL with GRPO**_\n", "\n", "As the reinforcement learning training process progresses, the old reward model may not be\n", "sufficient to supervise the current policy model. Therefore, we also explore the iterative RL\n", "with GRPO. As shown in Algorithm 1, in iterative GRPO, we generate new training sets for the\n", "reward model based on the sampling results from the policy model and continually train the\n", "old reward model using a replay mechanism that incorporates 10% of historical data. Then, we\n", "set the reference model as the policy model, and continually train the policy model with the\n", "new reward model.\n", "\n", "**4.2. Training and Evaluating DeepSeekMath-RL**\n", "\n", "We conduct RL based on DeepSeekMath-Instruct 7B. The training data of RL are chain-ofthought-format questions related to GSM8K and MATH from the SFT data, which consists\n", "of around 144K questions. We exclude other SFT questions to investigate the impact of RL\n", "on benchmarks that lack data throughout the RL phase. We construct the training set of\n", "reward models following (Wang et al., 2023b). We train our initial reward model based on the\n", "DeepSeekMath-Base 7B with a learning rate of 2e-5. For GRPO, we set the learning rate of the\n", "policy model as 1e-6. The KL coefficient is 0.04. For each question, we sample 64 outputs. The\n", "max length is set to 1024, and the training batch size is 1024. The policy model only has a single\n", "update following each exploration stage. We evaluate DeepSeekMath-RL 7B on benchmarks\n", "following DeepSeekMath-Instruct 7B. For DeepSeekMath-RL 7B, GSM8K and MATH with\n", "chain-of-thought reasoning can be regarded as in-domain tasks and all the other benchmarks\n", "can be regarded as out-of-domain tasks.\n", "\n", "Table 5 demonstrates the performance of open- and closed-source models with both chainof-thought and tool-integrated reasoning on English and Chinese benchmarks. We find that:\n", "1) DeepSeekMath-RL 7B attains accuracies of 88.2% and 51.7% on GSM8K and MATH, respectively, utilizing chain-of-thought reasoning. This performance surpasses that of all open-source\n", "models in the 7B to 70B range, as well as the majority of closed-source models. 2) Crucially,\n", "DeepSeekMath-RL 7B is only trained on chain-of-thought-format instruction tuning data of\n", "GSM8K and MATH, starting from DeepSeekMath-Instruct 7B. Despite the constrained scope\n", "of its training data, it outperforms DeepSeekMath-Instruct 7B across all evaluation metrics,\n", "showcasing the effectiveness of reinforcement learning."]}, "transient": {}}], "execution_count": null, "attachments": {}}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Can you explain more about `While in the LLM context, usually only the last token is assigned a reward score by the reward model, which may complicate the training of a value function that is accurate at each token`? What do they mean bu accurate?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Great question! This gets to a key challenge in training language models with reinforcement learning.\n", "\n", "In typical RL problems, you get feedback at many steps. But when training language models to solve math problems, the reward model usually only gives a score at the very end - like \"correct answer = +1, wrong answer = 0.\"\n", "\n", "The **value function** (critic model) is supposed to predict \"how good is it to be at this point in the solution?\" But if it only sees rewards at the final token, it has to somehow learn to estimate values for all the intermediate tokens based on very sparse feedback.\n", "\n", "By \"accurate,\" they mean the value function struggles to give reliable estimates for early tokens in a solution, since it rarely gets direct training signal about whether those early steps are actually helpful.\n", "\n", "Does this help explain why they moved away from needing a separate value function in their GRPO approach?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "how is KL divergence in equation (4) guaranteed to be positive?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Looking at equation (4), the KL divergence formula they use is:\n", "\n", "**D**_KL(\u03c0_\u03b8 || \u03c0_ref) = \u03c0_\u03b8(o_i,t | q, o_i,<t) - 1 - \u03c0_ref(o_i,t | q, o_i,<t) + \u03c0_ref(o_i,t | q, o_i,<t) - log \u03c0_\u03b8(o_i,t | q, o_i,<t)\n", "\n", "This might look confusing, but it's actually a clever mathematical identity. The key insight is that for any positive number x, the expression (x - 1 - log x) is always non-negative, with equality only when x = 1.\n", "\n", "Here, x = \u03c0_\u03b8/\u03c0_ref (the ratio of the two probabilities).\n", "\n", "Can you see why x - 1 - log x \u2265 0 for any positive x? Would you like me to explain this mathematical property, or does this make sense?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Ok explain Algorithm 1 to me."]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Algorithm 1 shows the iterative GRPO training process. Let me break it down step by step:\n", "\n", "**Setup (lines 1-2)**: Start with an initial policy model and begin the main training loop.\n", "\n", "**Each iteration (lines 3-4)**: Set the current policy as the reference model, then start the inner training steps.\n", "\n", "**For each training step (lines 5-7)**: Sample a batch of questions, update the \"old policy\" to be the current one, then generate multiple outputs for each question using this old policy.\n", "\n", "The key idea is generating a group of different solutions to the same problem, which is where the \"Group\" in GRPO comes from.\n", "\n", "Would you like me to continue with the next part (computing rewards and advantages), or do you have questions about this first section?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "What is the point of `update the \"old policy\" to be the current one`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Good question! This is about maintaining stability during training.\n", "\n", "The \"old policy\" is used to generate the training examples (the multiple outputs for each question), while the \"current policy\" is what's actually being updated/improved.\n", "\n", "By keeping the old policy fixed during each training step, they ensure that all the outputs in a group were generated under the same conditions. This makes the comparisons between outputs fair and meaningful.\n", "\n", "If they used a constantly changing policy to generate examples, the group comparisons would become inconsistent as the policy shifts mid-training.\n", "\n", "Does this distinction between the policy that generates examples vs. the policy being trained make sense?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "What do MMLU, BBH, HumanEval, and MBPP test for?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "These are different types of benchmarks that test various AI capabilities:\n", "\n", "**MMLU (Massive Multitask Language Understanding)**: Tests knowledge across 57 subjects like history, science, math, law, etc. It's multiple choice questions covering diverse academic topics.\n", "\n", "**BBH (BIG-Bench Hard)**: Tests complex reasoning with 23 challenging tasks that require multi-step thinking to solve.\n", "\n", "**HumanEval**: Tests coding ability by asking the model to complete Python functions based on descriptions.\n", "\n", "**MBPP (Mostly Basic Python Problems)**: Another coding benchmark with Python programming tasks.\n", "\n", "Are you curious about why the DeepSeekMath paper included these non-math benchmarks, or would you like to know more about any specific one?\n", "\n"]}, {"cell_type": "code", "metadata": {}, "source": ["display_section(6)"], "outputs": [{"output_type": "display_data", "metadata": {}, "data": {"text/plain": ["<IPython.core.display.Markdown object>"], "text/markdown": ["##### **5. Discussion**\n", "\n", "In this section, we will share our findings in pre-training and RL experiments.\n", "\n", "**5.1. Lessons Learnt in Pre-Training**\n", "\n", "We first share our experience in pre-training. Unless otherwise specified, we will adhere to\n", "the training settings outlined in Section 2.2.1. It is worth noting that, when referring to the\n", "DeepSeekMath Corpus in this section, we use an 89B-token dataset from the second iteration of\n", "the data collection process.\n", "\n", "_**5.1.1. Code Training Benefits Mathematical Reasoning**_\n", "\n", "A popular yet unverified hypothesis suggests that code training improves reasoning. We attempt\n", "to offer a partial response to this, particularly within the mathematical domain: code training\n", "\n", "15\n", "\n", "Training Tokens w/o Tool Use w/ Tool Use\n", "Training Setting\n", "\n", "General Code Math GSM8K MATH CMATH GSM8K+Python MATH+Python\n", "\n", "No Continual Training \u2013 \u2013 \u2013 2.9% 3.0% 12.3% 2.7% 2.3%\n", "\n", "Two-Stage Training\n", "\n", "Stage 1: General Training 400B \u2013 \u2013 2.9% 3.2% 14.8% 3.3% 2.3%\n", "Stage 2: Math Training \u2013 \u2013 150B 19.1% 14.4% 37.2% 14.3% 6.7%\n", "\n", "Stage 1: Code Training \u2013 400B \u2013 5.9% 3.6% 19.9% 12.4% 10.0%\n", "Stage 2: Math Training \u2013 \u2013 150B **21.9%** **15.3%** **39.7%** 17.4% 9.4%\n", "\n", "One-Stage Training\n", "\n", "Math Training \u2013 \u2013 150B 20.5% 13.1% 37.6% 11.4% 6.5%\n", "\n", "Code & Math Mixed Training \u2013 400B 150B 17.6% 12.1% 36.3% **19.7%** **13.5%**\n", "\n", "Table 6 | Investigation of how code affects mathematical reasoning under different training\n", "settings. We experiment with DeepSeek-LLM 1.3B, and evaluate its mathematical reasoning\n", "performance without and with tool use via few-shot chain-of-thought prompting and few-shot\n", "program-of-thought prompting, respectively.\n", "\n", "improves models\u2019 ability to do mathematical reasoning both with and without tool use.\n", "\n", "To study how code training affects mathematical reasoning, we experimented with the\n", "following two-stage training and one-stage training settings:\n", "\n", "**Two-Stage Training**\n", "\n", " - **Code Training for 400B Tokens** \u2192 **Math Training for 150B Tokens** : We train DeepSeekLLM 1.3B for 400B code tokens followed by 150B math tokens;\n", "\n", " - **General Training for 400B Tokens** \u2192 **Math Training for 150B Tokens** : As a control\n", "experiment, we also experiment with general tokens (sampled from a large-scale general\n", "corpus created by DeepSeek-AI) instead of code tokens in the first stage of training, in an\n", "attempt to investigate the advantages of code tokens over general tokens in improving\n", "mathematical reasoning.\n", "\n", "**One-Stage Training**\n", "\n", " - **Math Training for 150B Tokens** : We train DeepSeek-LLM 1.3B for 150B math tokens;\n", "\n", " - **Training on a mixture of 400B Code Tokens and 150B Math Tokens** : Math training following code training degrades coding performance. We investigate whether code tokens,\n", "when mixed with math tokens for one-stage training, would still improve mathematical\n", "reasoning and also alleviate the problem of catastrophic forgetting.\n", "\n", "**Results** Table 6 and Table 7 demonstrate the downstream performance under different training\n", "settings.\n", "\n", "Code training benefits program-aided mathematical reasoning, both under the two-stage\n", "training and one-stage training settings. As shown in Table 6, under the two-stage training\n", "setting, code training alone already significantly enhances the ability to solve GSM8K and\n", "MATH problems using Python. Math training in the second stage yields further improvements.\n", "Interestingly, under the one-stage training setting, mixing code tokens and math tokens effectively mitigates the issue of catastrophic forgetting that arises from two-stage training, and also\n", "synergizes coding (Table 7) and program-aided mathematical reasoning (Table 6).\n", "\n", "16\n", "\n", "Training Tokens\n", "Training Setting MMLU BBH HumanEval (Pass@1) MBPP (Pass@1)\n", "General Code Math\n", "\n", "No Continual Training \u2013 \u2013 \u2013 24.5% 28.1% 12.2% 13.0%\n", "\n", "Two-Stage Training\n", "\n", "Stage 1: General Training 400B \u2013 \u2013 25.9% 27.7% 15.2% 13.6%\n", "Stage 2: Math Training \u2013 \u2013 150B 33.1% 32.7% 12.8% 13.2%\n", "\n", "Stage 1: Code Training \u2013 400B \u2013 25.0% 31.5% 25.0% **40.0%**\n", "Stage 2: Math Training \u2013 \u2013 150B **36.2%** 35.3% 12.2% 17.0%\n", "\n", "One-Stage Training\n", "\n", "Math Training \u2013 \u2013 150B 32.3% 32.5% 11.6% 13.2%\n", "\n", "Code & Math Mixed Training \u2013 400B 150B 33.5% **35.6%** **29.3%** 39.4%\n", "\n", "Table 7 | Investigation of how different settings of code and math training affect model performance of language understanding, reasoning, and coding. We experiment with DeepSeek-LLM\n", "1.3B. We evaluate the models on MMLU and BBH using few-shot chain-of-thought prompting.\n", "On HumanEval and MBPP, we conduct zero-shot and few-shot evaluations, respectively.\n", "\n", "English Benchmarks Chinese Benchmarks\n", "Model Size ArXiv Corpus MMLU Gaokao Gaokao\n", "\n", "GSM8K MATH OCW SAT CMATH\n", "STEM MathCloze MathQA\n", "\n", "No Math Training 2.9% 3.0% 2.9% 15.6% 19.5% 12.3% 0.8% 17.9%\n", "DeepSeek-LLM 1.3B MathPile 2.7% 3.3% 2.2% 12.5% 15.7% 1.2% 0.0% 2.8%\n", "\n", "ArXiv-RedPajama 3.3% 3.4% 4.0% 9.4% 9.0% 7.4% 0.8% 2.3%\n", "\n", "No Math Training 29.0% 12.5% 6.6% 40.6% 38.1% 45.9% 5.9% 21.1%\n", "DeepSeek-Coder-Base-v1.5 7B MathPile 23.6% 11.5% 7.0% 46.9% 35.8% 37.9% 4.2% 25.6%\n", "\n", "ArXiv-RedPajama 28.1% 11.1% 7.7% 50.0% 35.2% 42.6% 7.6% 24.8%\n", "\n", "Table 8 | Effect of math training on different arXiv datasets. Model performance is evaluated\n", "with few-shot chain-of-thought prompting.\n", "\n", "ArXiv Corpus miniF2F-valid miniF2F-test\n", "\n", "No Math Training 20.1% 21.7%\n", "\n", "MathPile 16.8% 16.4%\n", "ArXiv-RedPajama 14.8% 11.9%\n", "\n", "Table 9 | Effect of math training on different arXiv corpora, the base model being DeepSeekCoder-Base-v1.5 7B. We evaluate informal-to-formal proving in Isabelle.\n", "\n", "Code training also improves mathematical reasoning without tool use. Under the two-stage\n", "training setting, the initial stage of code training already results in moderate enhancements.\n", "It also boosts the efficiency of the subsequent math training, eventually leading to the best\n", "performance. However, combining code tokens and math tokens for one-stage training compromises mathematical reasoning without tool use. One conjecture is that DeepSeek-LLM 1.3B,\n", "due to its limited scale, lacks the capacity to fully assimilate both code and mathematical data\n", "simultaneously.\n", "\n", "_**5.1.2. ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning**_\n", "\n", "ArXiv papers are commonly included as a component of math pre-training data (Azerbayev\n", "et al., 2023; Lewkowycz et al., 2022a; Polu and Sutskever, 2020; Wang et al., 2023c). However,\n", "\n", "17\n", "\n", "detailed analysis regarding their impact on mathematical reasoning has not been extensively\n", "conducted. Perhaps counter-intuitively, according to our experiments, arXiv papers seem\n", "ineffective in improving mathematical reasoning. We experiment with models of different sizes,\n", "including DeepSeek-LLM 1.3B and DeepSeek-Coder-Base-v1.5 7B (Guo et al., 2024), using arXiv\n", "corpora that underwent varied processing pipelines:\n", "\n", " - **MathPile** (Wang et al., 2023c): an 8.9B-token corpus developed with cleaning and filtering\n", "heuristic rules, over 85% of which are scientific arXiv papers;\n", "\n", " - **ArXiv-RedPajama** (Computer, 2023): the entirety of arXiv LaTeX files with preambles,\n", "comments, macros, and bibliographies removed, totaling 28.0B tokens.\n", "\n", "In our experiments, we separately train DeepSeek-LLM 1.3B for 150B tokens and DeepSeekCoder-Base-v1.5 7B for 40B tokens on each arXiv corpus. It seems that arXiv papers are ineffective\n", "in improving mathematical reasoning. When trained on a arXiv-only corpus, both models display no notable improvements or even deterioration across various mathematical benchmarks of\n", "different complexities employed in this study. These benchmarks include quantitative reasoning\n", "datasets like GSM8K and MATH (Table 8), multiple-choice challenges like MMLU-STEM (Table\n", "8), and formal mathematics like miniF2F (Table 9).\n", "\n", "However, this conclusion has its limitations and should be taken with a grain of salt. We\n", "have not yet studied:\n", "\n", " - The impact of arXiv tokens on specific math-related tasks not included in this research,\n", "such as informalization of theorems which is to convert formal statements or proofs to\n", "their informal versions;\n", "\n", "  - The effect of arXiv tokens when combined with other types of data;\n", "\n", "  - Whether the benefits of arXiv papers would manifest themselves at a larger model scale.\n", "\n", "Thus, further exploration is required, which we leave for future studies.\n", "\n", "**5.2. Insights of Reinforcement Learning**\n", "\n", "_**5.2.1. Towards to a Unified Paradigm**_\n", "\n", "In this section, we provide a unified paradigm to analyze different training methods, such as\n", "SFT, RFT, DPO, PPO, GRPO, and further conduct experiments to explore the factors of the\n", "unified paradigm. Generally, the gradient with respect to the parameter _\ud835\udf03_ of a training method\n", "can be written as:\n", "\n", "\n", "\u2207 _\ud835\udf03_ log _\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc5c_ _\ud835\udc61_ | _\ud835\udc5e_, _\ud835\udc5c_ _<\ud835\udc61_ )\n", "\ufffd\ufffd\ufffd\ufffd\n", "\ufffd\n", "\n", "\n", "| _\ud835\udc5c_ |\n", "\u2211\ufe01 _\ud835\udc3a\ud835\udc36_ A ( _\ud835\udc5e_, _\ud835\udc5c_, _\ud835\udc61_, _\ud835\udf0b_ _\ud835\udc5f\ud835\udc53_ )\n", "\n", "_\ud835\udc61_ =1 \ufffd **\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd** \ufffd\ufffd **\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd** \ufffd\n", "\n", "_\ud835\udc3a\ud835\udc5f\ud835\udc4e\ud835\udc51\ud835\udc56\ud835\udc52\ud835\udc5b\ud835\udc61\ud835\udc36\ud835\udc5c\ud835\udc52\ud835\udc53\ud835\udc53\ud835\udc56\ud835\udc50\ud835\udc56\ud835\udc52\ud835\udc5b\ud835\udc61_\n", "\n", "\n", "\u2207 _\ud835\udf03_ J A ( _\ud835\udf03_ ) = **E** [( _\ud835\udc5e_, _\ud835\udc5c_ ) \u223c D\n", "\n", "\ufffd **\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd** \ufffd\ufffd **\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd** \ufffd\n", "\n", "_\ud835\udc37\ud835\udc4e\ud835\udc61\ud835\udc4e\ud835\udc46\ud835\udc5c\ud835\udc62\ud835\udc5f\ud835\udc50\ud835\udc52_\n", "\n", "\n", "]\n", "\ufffd\ufffd\ufffd\ufffd\n", "\ufffd\n", "\n", "\n", "1\n", "\n", "| _\ud835\udc5c_ |\n", "\n", "\n", ". (5)\n", "\n", "\n", "There exist three key components: 1) _Data Source_ D, which determines the training data; 2)\n", "_Reward Function_ _\ud835\udf0b_ _\ud835\udc5f\ud835\udc53_, which is the source of the training reward signal; 3) _Algorithm_ A : which\n", "processes the training data and the reward signal to the gradient coefficient _\ud835\udc3a\ud835\udc36_ that determines\n", "the magnitude of the penalty or reinforcement for the data. We analyze several representative\n", "methods based on such a unified paradigm:\n", "\n", " - **Supervised Fine-tuning (SFT)** : SFT fine-tunes pretrained model on human selected SFT\n", "data.\n", "\n", "18\n", "\n", "**Methods** **Data Source** **Reward Function** **Gradient Coefficient**\n", "\n", "SFT _\ud835\udc5e_, _\ud835\udc5c_ \u223c _\ud835\udc43_ _\ud835\udc60\ud835\udc53\ud835\udc61_ ( _\ud835\udc44_, _\ud835\udc42_ )    - 1\n", "\n", "RFT _\ud835\udc5e_ \u223c _\ud835\udc43_ _\ud835\udc60\ud835\udc53\ud835\udc61_ ( _\ud835\udc44_ ), _\ud835\udc5c_ \u223c _\ud835\udf0b_ _\ud835\udc60\ud835\udc53\ud835\udc61_ ( _\ud835\udc42_ | _\ud835\udc5e_ ) Rule Equation 10\n", "DPO _\ud835\udc5e_ \u223c _\ud835\udc43_ _\ud835\udc60\ud835\udc53\ud835\udc61_ ( _\ud835\udc44_ ), _\ud835\udc5c_ [+], _\ud835\udc5c_ [\u2212] \u223c _\ud835\udf0b_ _\ud835\udc60\ud835\udc53\ud835\udc61_ ( _\ud835\udc42_ | _\ud835\udc5e_ ) Rule Equation 14\n", "\n", "Online RFT _\ud835\udc5e_ \u223c _\ud835\udc43_ _\ud835\udc60\ud835\udc53\ud835\udc61_ ( _\ud835\udc44_ ), _\ud835\udc5c_ \u223c _\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc42_ | _\ud835\udc5e_ ) Rule Equation 10\n", "PPO _\ud835\udc5e_ \u223c _\ud835\udc43_ _\ud835\udc60\ud835\udc53\ud835\udc61_ ( _\ud835\udc44_ ), _\ud835\udc5c_ \u223c _\ud835\udf0b_ _\ud835\udf03_ ( _\ud835\udc42_ | _\ud835\udc5e_ ) Model Equation 18\n", "GRPO _\ud835\udc5e_ \u223c _\ud835\udc43_ _\ud835\udc60\ud835\udc53\ud835\udc61_ ( _\ud835\udc44_ ), { _\ud835\udc5c_ _\ud835\udc56_ } _[\ud835\udc3a]_ _\ud835\udc56_ =1 [\u223c] _[\ud835\udf0b]_ _[\ud835\udf03]_ [(] _[\ud835\udc42]_ [|] _[\ud835\udc5e]_ [)] Model Equation 21\n", "\n", "Table 10 | The data source and gradient coefficient of different methods. _\ud835\udc43_ _\ud835\udc60\ud835\udc53\ud835\udc61_ denotes the data\n", "distribution of supervised fine-tuning datasets. _\ud835\udf0b_ _\ud835\udf03_ _\ud835\udc60\ud835\udc53\ud835\udc61_ and _\ud835\udf0b_ _\ud835\udf03_ denote the supervised fine-tuned\n", "model and the real-time policy model during the online training process, respectively.\n", "\n", "\n", "\n", "\n", "\n", "\n", "Figure 5 | Performance of the DeepSeekMath-Instruct 1.3B model, which was further trained\n", "using various methods, on two benchmarks.\n", "\n", " - **Rejection Sampling Fine-tuning (RFT)** : RFT further fine-tunes the SFT model on the\n", "filtered outputs sampled from the SFT model based on SFT questions. RFT filters the\n", "outputs based on the correctness of their answers.\n", "\n", " - **Direct Preference Optimization (DPO)** : DPO further refines the SFT model by fine-tuning\n", "it on augmented outputs sampled from the SFT model, using pair-wise DPO loss.\n", "\n", " - **Online Rejection Sampling Fine-tuning (Online RFT)** : Different from RFT, Online RFT\n", "initiates the policy model using the SFT model and refines it by fine-tuning with the\n", "augmented outputs sampled from the real-time policy model.\n", "\n", " - **PPO/GRPO** : PPO/GRPO initializes the policy model using the SFT model and reinforces\n", "it with the outputs sampled from the real-time policy model.\n", "\n", "We summarize the components of these methods in Table 10. Please refer to Appendix A.1 for a\n", "more detailed derivation process.\n", "\n", "**Observation about Data Source** We divide the data source into two categories, online sampling, and offline sampling. Online sampling denotes that the training data is from the exploration results of the real-time training policy model, while offline sampling denotes that the\n", "\n", "19\n", "\n", "|Col1|Col2|Col3|\n", "|---|---|---|\n", "||||\n", "||||\n", "||||\n", "||||\n", "||||\n", "||||\n", "||||\n", "|0 1300 2300|0 1300 2300|3300 4300 5300|\n", "\n", "|Col1|Col2|Col3|Col4|\n", "|---|---|---|---|\n", "|||||\n", "|||||\n", "|||||\n", "|||||\n", "|||||\n", "|||||\n", "|0 1300 2|0 1300 2|300 3300 4300 5300|300 3300 4300 5300|\n", "\n", "\n", "Figure 6 | Performance of iterative reinforcement learning with DeepSeekMath-Instruct 7B on\n", "two benchmarks.\n", "\n", "training data is from the sampling results of the initial SFT model. RFT and DPO follow the\n", "offline style, while Online RFT and GRPO follow the online style.\n", "\n", "As shown in Figure 5, we find that the Online RFT significantly outperforms RFT on two\n", "benchmarks. Specifically, Online RFT is comparable to RFT in the early stage of training but\n", "gains an absolute advantage in the later stage, demonstrating the superiority of online training.\n", "This is intuitive, as in the initial stage, the actor and the SFT model exhibit close resemblance,\n", "with the sampled data revealing only minor differences. In the later stage, however, the data\n", "sampled from the actor will exhibit more significant differences, and real-time data sampling\n", "will offer greater advantages.\n", "\n", "**Observation about Gradient Coefficient** The algorithm processes the reward signal to the\n", "gradient coefficient to update the model parameter. We divide the reward function as \u2018Rule\u2019\n", "and \u2018Model\u2019 in our experiments. Rule refers to judging the quality of a response based on\n", "the correctness of the answer, and Model denotes that we train a reward model to score each\n", "response. The training data of the reward model is based on the rule judgment. Equations 10\n", "and 21 highlight a key difference between GRPO and Online RFT: GRPO uniquely adjusts its\n", "gradient coefficient based on the reward value provided by the reward model. This allows for\n", "differential reinforcement and penalization of responses according to their varying magnitudes.\n", "In contrast, Online RFT lacks this feature; it does not penalize incorrect responses and uniformly\n", "reinforces all responses with correct answers at the same level of intensity.\n", "\n", "As demonstrated in Figure 5, GRPO surpasses online RFT, thereby highlighting the efficiency\n", "of altering positive and negative gradient coefficients. In addition, GRPO+PS shows superior\n", "performance compared to GRPO+OS, indicating the benefits of using fine-grained, step-aware\n", "gradient coefficients. Furthermore, we explore the iterative RL, in our experiments, we conduct\n", "two rounds of iteration. As shown in Figure 6, we notice that the iterative RL significantly\n", "improves the performance, especially at the first iteration.\n", "\n", "20\n", "\n", "|Col1|Col2|Col3|Col4|Col5|Col6|Col7|\n", "|---|---|---|---|---|---|---|\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "|1|4 8 1|4 8 1|4 8 1|6 32 64|6 32 64|6 32 64|\n", "\n", "|Col1|Col2|Col3|Col4|Col5|Col6|Col7|\n", "|---|---|---|---|---|---|---|\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "||||||||\n", "|1|4 8 1|4 8 1|4 8 1|6 32 64|6 32 64|6 32 64|\n", "\n", "\n", "Figure 7 | The Maj@K and Pass@K of SFT and RL DeepSeekMath 7B on GSM8K and MATH\n", "(temperature 0.7). It was noted that RL enhances Maj@K but not Pass@K.\n", "\n", "_**5.2.2. Why RL Works?**_\n", "\n", "In this paper, we conduct reinforcement learning based on a subset of instruction tuning\n", "data, and it achieves significant performance enhancement upon the instruction tuning model.\n", "To further explain why reinforcement learning works. We evaluate the Pass@K and Maj@K\n", "accuracy of the Instruct and RL models on two benchmarks. As shown in Figure 7, RL enhances\n", "Maj@K\u2019s performance but not Pass@K. These findings indicate that RL enhances the model\u2019s\n", "overall performance by rendering the output distribution more robust, in other words, **it seems**\n", "**that the improvement is attributed to boosting the correct response from TopK rather than**\n", "**the enhancement of fundamental capabilities.** Similarly, (Wang et al., 2023a) identified a\n", "**misalignment problem** in reasoning tasks within the SFT model, showing that the reasoning\n", "performance of SFT models can be improved through a series of preference alignment strategies\n", "(Song et al., 2023; Wang et al., 2023a; Yuan et al., 2023b).\n", "\n", "_**5.2.3. How to Achieve More Effective RL?**_\n", "\n", "We demonstrate RL works pretty well in mathematical reasoning tasks. We also provide a unified\n", "paradigm to understand different representative training methods. Within this paradigm, all\n", "methods are conceptualized as either direct or simplified RL techniques. As summarized in\n", "Equation 5, there exist three key components: Data Source, Algorithm, and Reward Function.\n", "We provide some potential future directions about the three components.\n", "\n", "**Data Source** Data source is the raw material of all training methods. In the context of RL, we\n", "specifically refer to the data source as the unlabeled questions with the outputs sampled from\n", "the policy model. In this paper, we only use the questions from the instruction tuning stage and\n", "a naive nucleus sampling to sample outputs. We think this is a potential reason that our RL\n", "pipeline only improves the Maj@K performance. In the future, we will explore our RL pipeline\n", "on out-of-distribution question prompts, in conjunction with **advanced sampling (decoding)**\n", "**strategies**, like those based on tree-search methods (Yao et al., 2023). Also, the **efficient inference**\n", "**techniques** (Kwon et al., 2023; Leviathan et al., 2023; Xia et al., 2023, 2024), which determines\n", "\n", "21\n", "\n", "the exploration efficiency of policy models, also play an exceedingly important role.\n", "\n", "**Algorithms** Algorithms process the data and reward signal to the gradient coefficient to update\n", "the model parameter. Based on Equation 5, to some extent, all methods now fully **TRUST** the\n", "signal of the reward function to increase or decrease the conditional probability of a certain\n", "token. However, it is impossible to ensure the reward signal is always reliable, especially in\n", "extremely complex tasks. For example, even the PRM800K datasets (Lightman et al., 2023),\n", "which have been carefully annotated by well-trained annotators, still contain approximately 20%\n", "of incorrectly annotations [7] . To this end, we will explore the reinforcement learning algorithm\n", "that is robust against noisy reward signals. We believe such **WEAK-TO-STRONG** (Burns et al.,\n", "2023) alignment methods will bring a fundamental change to the learning algorithms.\n", "\n", "**Reward Function** Reward function is the source of the training signal. In RL, the reward\n", "function is usually the neural reward model. We think there exist three important directions for\n", "reward models: 1) **How to enhance the generalization ability of the reward model.** The reward\n", "model must be effectively generalized to handle out-of-distribution questions and advanced\n", "decoding outputs; otherwise, reinforcement learning may merely stabilize the distribution of\n", "LLMs rather than improve their fundamental capabilities; 2) **How to reflect the uncertainty**\n", "**of reward model.** The uncertainty could potentially act as a linking bridge between the weak\n", "reward model and the weak-to-strong learning algorithms; 3) **How to efficiently build high-**\n", "**quality process reward models** that can provide fine-grained training signals for the reasoning\n", "process (Lightman et al., 2023; Wang et al., 2023b)."]}, "transient": {}}], "execution_count": null, "attachments": {}}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "What is the difference between GRPO+PS and GRPO+OS?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Looking at the paper, GRPO+PS and GRPO+OS refer to different supervision approaches within GRPO:\n", "\n", "**GRPO+OS (Outcome Supervision)**: Provides a reward only at the end of each complete solution. All tokens in the solution get the same advantage value based on whether the final answer is correct or incorrect.\n", "\n", "**GRPO+PS (Process Supervision)**: Provides rewards at the end of each reasoning step throughout the solution. Each token's advantage is calculated based on the cumulative rewards from all the following steps.\n", "\n", "The paper shows that GRPO+PS performs better than GRPO+OS, suggesting that step-by-step feedback during the reasoning process is more effective than just judging the final outcome.\n", "\n", "Does this help clarify the distinction between these two approaches?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "What are `Pass@K` and `Maj@K`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "**Pass@K** measures whether at least one correct answer appears in the top K generated responses. If you generate K different solutions and any one of them is correct, Pass@K = 1.\n", "\n", "**Maj@K** (Majority@K) takes the most common answer among the top K responses and checks if that majority answer is correct.\n", "\n", "For example, if K=5 and you get answers [2, 3, 2, 2, 4], then:\n", "- Pass@5 = 1 (assuming 2 is correct)  \n", "- Maj@5 = 1 (majority answer is 2)\n", "\n", "The key insight from the paper is that RL improved Maj@K but not Pass@K, suggesting it made correct answers more likely to be the majority choice rather than fundamentally improving the model's capability to find correct solutions.\n", "\n", "Does this distinction make sense to you?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "How are K responses generated?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "K responses are generated by sampling multiple times from the same model using the same question, typically with some randomness (like temperature > 0) so you get different outputs each time.\n", "\n", "For example, you might ask the model \"What is 2x + 3 = 7?\" five times with temperature 0.7, and get slightly different solution approaches or intermediate steps, even though they might arrive at the same final answer.\n", "\n", "Are you curious about the specific sampling techniques they used, or how the temperature parameter affects the diversity of responses?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "What's the K value they used in the paper?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Looking at the paper, they used different K values in different contexts:\n", "\n", "For the Pass@K and Maj@K evaluation shown in Figure 7, they tested with K values of 1, 4, 8, 16, 32, and 64.\n", "\n", "For their GRPO training process, they mention sampling 64 outputs per question during reinforcement learning.\n", "\n", "Are you interested in understanding why they chose these specific values, or curious about another aspect of their methodology?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what is a misalignment problem?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "A misalignment problem occurs when a model's training objective doesn't perfectly match what we actually want it to do.\n", "\n", "In the context of this paper, the SFT (supervised fine-tuning) model learns to mimic the style and format of good mathematical reasoning, but it doesn't necessarily learn to consistently produce correct answers.\n", "\n", "So the model might write very convincing-looking step-by-step solutions that follow the right format, but still arrive at wrong conclusions.\n", "\n", "Does this help explain why they needed reinforcement learning to better align the model's outputs with actually getting the right answers?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what is `advanced sampling (decoding) strategies`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Advanced sampling strategies are more sophisticated ways to generate text from language models, beyond just picking the most likely next word.\n", "\n", "Basic sampling might use temperature or nucleus sampling (picking from the top-p most likely tokens). Advanced strategies include:\n", "\n", "- **Tree search methods**: Exploring multiple possible paths through a solution and selecting the best one\n", "- **Beam search**: Keeping track of several promising sequences simultaneously\n", "- **Monte Carlo Tree Search**: Using simulation to evaluate different reasoning paths\n", "\n", "These help the model explore more diverse and potentially better solutions rather than just following one greedy path.\n", "\n", "Are you curious about how these might improve mathematical reasoning specifically, or would you like to know more about any particular strategy?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "can you elaborate more on: `How to reflect the uncertainty of reward model. The uncertainty could potentially act as a linking bridge between the weak reward model and the weak-to-strong learning algorithms`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "This is about making reward models \"honest\" about when they're unsure, rather than always giving confident scores.\n", "\n", "A reward model might confidently score a math solution as \"good\" even when it's actually wrong or the model isn't sure. But if the reward model could express uncertainty - like \"I think this is good, but I'm only 60% confident\" - then the learning algorithm could adjust accordingly.\n", "\n", "The \"weak-to-strong\" connection refers to using a weaker model (the reward model) to train a stronger one (the policy model). If the reward model knows its own limitations and communicates uncertainty, it can avoid misleading the stronger model during training.\n", "\n", "Would you like me to explain more about why this uncertainty is important, or are you curious about how it might work in practice?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## From PPO to GRPO"], "attachments": {}}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "In PPO, what is the advantage term calculated by Generalized Advantage Estimation (GAE)?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "The advantage term in PPO represents \"how much better is this action compared to what we expected?\"\n", "\n", "GAE calculates this by comparing:\n", "- The actual rewards received from a particular action\n", "- What the value function predicted we would get from that state\n", "\n", "So if the value function predicted a reward of 0.5 but we actually got 0.8, the advantage would be positive (around +0.3), meaning this action was better than expected.\n", "\n", "Would you like me to explain how GAE specifically computes this comparison, or are you more interested in understanding why advantages are useful for training?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "How does GAE work?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "GAE works by combining multiple time horizons to estimate advantages more accurately.\n", "\n", "Instead of just looking at immediate rewards, GAE considers:\n", "- What happened in the next step\n", "- What happened in the next few steps  \n", "- What the value function predicts for longer-term rewards\n", "\n", "It uses a parameter \u03bb (lambda) to weight these different time horizons - higher \u03bb means looking further into the future, lower \u03bb focuses more on immediate rewards.\n", "\n", "Would you like me to explain the mathematical formula GAE uses to combine these different time perspectives, or are you more interested in understanding why this multi-horizon approach works better than simpler methods?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "mathematical formula GAE uses to combine these different time perspectives"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "The GAE formula combines different time horizons using this approach:\n", "\n", "**A^GAE(t) = \u03a3 (\u03b3\u03bb)^l \u00d7 \u03b4(t+l)**\n", "\n", "Where:\n", "- \u03b4(t) = r(t) + \u03b3V(t+1) - V(t) is the temporal difference error\n", "- \u03b3 is the discount factor (how much future rewards matter)\n", "- \u03bb controls the bias-variance tradeoff (0 = only immediate, 1 = full future)\n", "\n", "The key insight is that it exponentially weights different time horizons - nearby steps get more weight than distant ones.\n", "\n", "Would you like me to explain what the temporal difference error \u03b4 represents, or are you curious about how the \u03b3\u03bb weighting works in practice?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "why this multi-horizon approach works better than simpler methods?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "The multi-horizon approach works better because it balances two competing problems:\n", "\n", "**High bias vs. High variance**\n", "- Looking only at immediate rewards (low \u03bb) gives stable but potentially wrong estimates\n", "- Looking far into the future (high \u03bb) gives more accurate long-term estimates but with more noise\n", "\n", "GAE finds a sweet spot by blending these perspectives - it gets the stability of short-term estimates while still capturing important long-term patterns.\n", "\n", "This is especially valuable in complex tasks like math reasoning where early steps might not show their value until much later in the solution.\n", "\n", "Would you like me to explain more about the bias-variance tradeoff, or are you curious about how this applies specifically to training language models?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "GAE takes rewards and value as inputs, but where do different time horizons come from?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Great question! The different time horizons come from how GAE calculates the temporal difference (TD) errors at each step.\n", "\n", "When you have a sequence of states and rewards over time, GAE looks at:\n", "- 1-step TD error: immediate reward + next state's value - current state's value\n", "- 2-step TD error: reward + discounted reward from next step + value two steps ahead - current value\n", "- 3-step TD error: extends even further...\n", "\n", "Each of these gives a different \"horizon\" or time perspective on how good the current action was.\n", "\n", "Does this help clarify where the multiple time horizons originate from in the calculation?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "I thought reward model only returned reward after the final token. How could there by immediate reward and so on?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "You're absolutely right! This is exactly the issue the DeepSeekMath paper highlights.\n", "\n", "In language model training, the reward model typically only gives a score at the very end (like \"correct answer = +1, wrong = 0\"). There usually aren't intermediate rewards during the reasoning process.\n", "\n", "This is actually one of the key problems that motivated GRPO - traditional PPO with GAE expects rewards at multiple timesteps, but math problems only get scored at the end. This makes it hard for the value function to learn good estimates for intermediate steps.\n", "\n", "Does this help explain why they moved away from needing a value function in their GRPO approach?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Does value model provide value for each token then?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Yes, exactly! In traditional PPO, the value model is supposed to provide a value estimate for each token position in the sequence.\n", "\n", "But this creates a problem: if you only get a reward signal at the final token (like \"correct answer\"), how can the value model learn to accurately predict values for all the earlier tokens in the reasoning chain?\n", "\n", "This is why the paper says it \"may complicate the training of a value function that is accurate at each token\" - the value model has to somehow learn from very sparse feedback.\n", "\n", "Does this help clarify why GRPO's approach of using group averages instead of a separate value model is appealing?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "So how did original PPO do with immediate reward part? Did they just use reward from the overall sequence?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Yes, that's exactly the approach! In the original PPO for language models, they typically:\n", "\n", "1. Only get a reward at the final token (like +1 for correct, 0 for incorrect)\n", "2. Set all intermediate token rewards to 0\n", "3. Let the value function try to \"back-propagate\" the final reward signal to estimate how valuable each earlier token position was\n", "\n", "But this is quite challenging - the value function has to learn complex patterns about which early reasoning steps tend to lead to correct final answers, with very limited direct feedback.\n", "\n", "Does this help explain why the DeepSeekMath team found this approach problematic for mathematical reasoning tasks?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "So, DeepSeekMath team used GRPO, which uses the average reward of multiple sampled outputs as the baseline. But then how is this used to calculate advantage?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Good question! In GRPO, the advantage for each token is calculated as:\n", "\n", "**Advantage = (individual output's reward - group average reward) / group standard deviation**\n", "\n", "So if you have 5 outputs with rewards [1, 0, 1, 0, 1]:\n", "- Group average = 0.6\n", "- An output with reward 1 gets advantage = (1 - 0.6) / std = positive\n", "- An output with reward 0 gets advantage = (0 - 0.6) / std = negative\n", "\n", "This means tokens in better-performing outputs get positive reinforcement, while tokens in worse-performing outputs get negative reinforcement.\n", "\n", "Does this approach make sense to you? Would you like me to clarify how this differs from the traditional value function approach?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "For calculating Advantage, how do tokens come into play? Because the Advatage term, A\u02c6i,t, has i and t. I know i is for the number of groups and t is for tokens. But there is only one reward for each group. Right?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "You're absolutely right to be confused! This is a key insight.\n", "\n", "In **outcome supervision** (which is simpler), there's indeed only one reward per output. So all tokens in output i get the same advantage: \u00c2i,t = (ri - group_average) / group_std for every token t.\n", "\n", "But in **process supervision**, they actually get multiple rewards per output - one reward at the end of each reasoning step. So different tokens within the same output can have different advantages based on which reasoning steps they're part of.\n", "\n", "Does this distinction between outcome vs. process supervision help clarify why the advantage has both i and t indices?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Why did they use different KL divergence penalty with unbiased estimator?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "They used a different KL divergence approach for two main reasons:\n", "\n", "**1. Simplicity**: Instead of adding KL penalty to each token's reward (which complicates advantage calculations), they add it directly to the loss function.\n", "\n", "**2. Mathematical correctness**: The unbiased estimator they use is guaranteed to be positive, which is a required property for KL divergence.\n", "\n", "The traditional approach of modifying rewards with KL penalties would make their group-based advantage calculation more complex, since they'd need to account for the penalty when computing group averages.\n", "\n", "Would you like me to explain more about why KL divergence needs to be positive, or are you curious about how this affects the training process?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Can you elaborate more on process supervision? How do they produce multiple rewards on a token level?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "In process supervision, they don't actually give rewards at the token level - they give rewards at the **reasoning step** level.\n", "\n", "For example, in a math solution like:\n", "- Step 1: \"First, I'll isolate the variable...\"\n", "- Step 2: \"Subtracting 3 from both sides gives...\"  \n", "- Step 3: \"Dividing by 2 gives x = 2\"\n", "\n", "Each step gets its own reward score from a process reward model that judges whether that reasoning step is correct or helpful.\n", "\n", "Then, for advantage calculation, each token gets the sum of all the step rewards that come after it in the solution.\n", "\n", "Does this help clarify how they break down the reasoning process? Would you like me to explain how they identify where each \"step\" ends?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "How does advantage differ from reward? How can baseline be useful?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Great question! The key difference is:\n", "\n", "**Reward** tells you \"how good was this action in absolute terms\"\n", "**Advantage** tells you \"how much better was this action compared to what we expected\"\n", "\n", "For example, if all your math solutions get rewards of 0.7, 0.8, and 0.9, the raw rewards suggest they're all pretty good. But the advantages would be negative, neutral, and positive respectively, showing which ones were better than average.\n", "\n", "The baseline (expected value) is crucial because it provides the comparison point. Without it, the model might reinforce mediocre solutions just because they got positive rewards.\n", "\n", "Does this distinction make sense? Would you like me to explain why this relative comparison is more effective for training?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Can you provide an analogy between those two?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "Sure! Think of it like grading on a curve:\n", "\n", "**Reward** is like your raw test score - you got 85/100 points.\n", "\n", "**Advantage** is like your curve-adjusted grade - if the class average was 90, then your 85 becomes \"below average\" even though 85 is normally a good score.\n", "\n", "The baseline (class average) helps you understand whether your performance was actually good or bad relative to what's typical.\n", "\n", "Does this analogy help clarify why using advantages rather than raw rewards leads to better learning?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Can you explain how they calculated the advantage for Process Supervision RL: `Subsequently,  the process supervision calculates the advantage of each token as the sum of the normalized rewards from the following steps, i.e., A\u02c6i,t = I\u0301  index ( j) \u2265t re  index ( j)  i`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "This formula means each token's advantage equals the sum of all step rewards that come **after** that token in the sequence.\n", "\n", "For example, if you have 3 reasoning steps with rewards [0.2, -0.1, 0.3]:\n", "- Tokens in step 1 get advantage = 0.2 + (-0.1) + 0.3 = 0.4\n", "- Tokens in step 2 get advantage = (-0.1) + 0.3 = 0.2  \n", "- Tokens in step 3 get advantage = 0.3\n", "\n", "The intuition is: tokens that lead to more good future steps should be reinforced more strongly.\n", "\n", "Does this \"future reward sum\" approach make sense to you?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "In algorithm, what does old policy model do?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "The old policy model serves as a **stable reference point** during training.\n", "\n", "It generates the training examples (the multiple outputs for each question) at the beginning of each training step, then stays frozen while the current policy gets updated.\n", "\n", "This prevents the training data from shifting mid-step, which would make the group comparisons inconsistent.\n", "\n", "Would you like me to explain why this stability is important for the learning process, or are you curious about another part of the algorithm?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "What is happening in `Update the policy model \u03c0\u03b8 by maximizing the GRPO objective (Equation 21)`?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "In this step, the policy model's parameters are being adjusted to increase the probability of generating tokens that had positive advantages and decrease the probability of tokens that had negative advantages.\n", "\n", "The GRPO objective function combines:\n", "- The advantage-weighted probability changes (reinforce good tokens, penalize bad ones)\n", "- A KL divergence penalty to prevent the model from changing too drastically\n", "\n", "This is where the actual learning happens - the model becomes more likely to produce better solutions.\n", "\n", "Would you like me to explain more about how the advantage weighting works, or are you curious about the KL penalty's role?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "what does `\u03c0re f (oi,t |q, oi,<t)` mean?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "`\u03c0ref(oi,t |q, oi,<t)` represents the probability that the reference model assigns to generating token `oi,t` given:\n", "\n", "- `q`: the question\n", "- `oi,<t`: all the previous tokens in the output sequence\n", "\n", "So it's asking: \"What's the likelihood the reference model would produce this specific token at this position?\"\n", "\n", "The reference model is typically the model before RL training started, used as a baseline for comparison.\n", "\n", "Does this notation make sense? Would you like me to explain why they compare against this reference model?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Does it return the whole distribution of all the probabilities or just one for oi, t?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "It returns just one probability - specifically the probability for the token `oi,t` that actually appeared in that position.\n", "\n", "The reference model has a full probability distribution over all possible tokens at that position, but this notation extracts only the probability assigned to the specific token that was generated.\n", "\n", "Does this distinction between the full distribution vs. one specific probability make sense?\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Prompt**\n", "\n", "Why do they divide the current policy by old one?"]}, {"cell_type": "markdown", "metadata": {}, "source": ["**AI Response**\n", "\n", "They divide the current policy by the old policy to measure how much the model has changed during training.\n", "\n", "This ratio tells you:\n", "- If ratio > 1: the current model assigns higher probability to this token than before\n", "- If ratio < 1: the current model assigns lower probability to this token than before\n", "- If ratio = 1: no change\n", "\n", "This ratio is then multiplied by the advantage to create the training signal - tokens with positive advantages get reinforced proportionally to how much the model changed.\n", "\n", "Does this help explain why the ratio is useful for controlling the training updates?\n", "\n"]}], "metadata": {"kernelspec": {"display_name": "python3", "language": "python", "name": "python3"}}, "nbformat": 4, "nbformat_minor": 4}