Skip to content

Instantly share code, notes, and snippets.

@StarDuster

StarDuster/LMDeploy 的量化和部署 笔记.md

Last active February 1, 2024 08:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save StarDuster/3301b7e9805e738b9f084011eff69eea to your computer and use it in GitHub Desktop.
Save StarDuster/3301b7e9805e738b9f084011eff69eea to your computer and use it in GitHub Desktop.
Download ZIP
Raw
LMDeploy 的量化和部署 笔记.md
image

很喜欢这次的课程，这次课程加入了一些我比较想了解的技术内容，比如量化部署，之前一直对量化有误解，今天才学习到原来计算时候是要反量化的。

image

批处理，课程介绍了turbomind的持续批处理优化（感觉有点类似 CPU 的指令流水线，CPU 也是访存压力大于计算压力，从需求和解决方式上都有点像）

https://github.com/InternLM/tutorial/blob/main/lmdeploy/img/6.png![image](https://gist.github.com/assets/7328584/3278911e-9c79-428f-ac2c-b506c8888507)

之前就听说过的张量并行，这几个图放的很有助于理解。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment