Deploy Dolly v2.0 to SageMaker

deploy_dolly_v2.ipynb

Thanks for the example. We have deployed many conversational models on Sagemaker. The challenge is that this way the endpoint does not stream the response and a lot of times for longer responses it times out.

You can try another conversational pattern for your server/client like a websocket

Has anyone used an inference config for the code as seen above so that the model can handle embeddings ?

Thanks for sharing, this is helping me a lot in trying to figure this topic out.
One question - why is there a mismatch between the transformers version in the requirements.txt file and in the Sagmaker model creation command? What is the difference, and how does it make sense that they will be different?