Table of Links
- Proposed Method: Quantized DyLoRA
- Experiments and Evaluation
- On the semi-sorted behavior of QDyLoRA
- Conclusion, Limitations, and References
A. Supplementary Material
4 On the semi-sorted behavior of QDyLoRA
As shown in Table 2, QDyLoRA reveals a semisorted performance across ranks. We justify this behavior by pointing out the limited finetuning budget. In a limited budget assumption, QDyLoRA updates its lower ranks more frequently than its higher ranks. That is because of the fact that lower ranks are also updated when higher ranks are selected. In other words, lower ranks have more chance to get updated than higher ranks. Hence, lower ranks are more tuned than higher ranks.
Authors:
(1) Hossein Rajabzadeh, University of Waterloo and Huawei Noah’s Ark Lab ([email protected]);
(2) Mojtaba Valipour, University of Waterloo ([email protected]);
(3) Tianshu Zhu, Huawei Noah’s Ark Lab ([email protected]);
(4) Marzieh Tahaei, Huawei Noah’s Ark Lab ([email protected]);
(5) Hyock Ju Kwon, ([email protected]);
(6) Ali Ghodsi, ([email protected]);
(7) Boxing Chen, Huawei Noah’s Ark Lab ([email protected]);
(8) Mehdi Rezagholizadeh, Huawei Noah’s Ark Lab ([email protected]).
This paper is
