sandeshrajbhandari

Open Interpreter will require approval before running code.                                                        

Use interpreter -y to bypass this.                                                                                 

Press CTRL-C to exit.                                                                                              

> hi

We were unable to determine the context window of this model. Defaulting to 3000.                                  

sandeshrajbhandari

sandeshrajbhandari

sandeshrajbhandari

sandeshrajbhandari

sandeshrajbhandari

#!/usr/bin/env python
# coding=utf-8
# Copyright 2023 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#

sandeshrajbhandari

sandeshrajbhandari

sandeshrajbhandari

sandeshrajbhandari

## elaborate text from https://lightning.ai/pages/community/tutorial/lora-llm/ using bing chat


*understanding lora for PEFT and LLM*

```
Why would we do this? For now, this alternative formulation serves a pedagogical goal to illustrate LoRA, but we will come back to it. So, when we train fully connected (i.e., “dense”) layers in a neural network, as shown above, the weight matrices usually have full rank, which is a technical term meaning that a matrix does not have any linearly dependent (i.e., “redundant”) rows or columns. In contrast, to full rank, low rank means that the matrix has redundant rows or columns. So, while the weights of a pretrained model have full rank on the pretrained tasks, the LoRA authors point out that pretrained large language models have a low “intrinsic dimension” when they are adapted to a new task, according to Aghajanyan et al. (2020). A low intrinsic dimension means the data can be effectively represented or approximated by a lower-dimensional space while retaining most of i