How to run StarCoderBase 1B SFT on a MacBook Pro with Apple Silicon
These are notes on how I managed to get StarCoderBase-1B-SFT model compiled into a quantized version such that it can run locally on my MBP M1 Pro and be queryable through an OpenAI API-compatible server. [StarCoderBase][1] is a model trained/tuned for programming tasks. The [1B parameters SFT model][2] I am using in this article is a version of the model that has had supervised fine tuning applied to it. I am just going to call this "StarCoder" in the rest of this article for the sake of simplicity. Number of parameters that a model has is going to impact resource usage, so a smaller version of the model makes it more