Experiment 27

experiment27

Is ByteDance’s Lumi an AI Experiment Disguised as an AI Creator Community?

ByteDance's Lumi የ AI ሙከራ እንደ AI ፈጣሪ ማህበረሰብ ተደብቋል?

¿Es Lumi de ByteDance un experimento de IA disfrazado de comunidad de creadores de IA?

Èske Lumi ByteDance a se yon eksperyans AI degize kòm yon kominote kreyatè AI?

Արդյո՞ք ByteDance-ի Lumi-ն AI փորձ է, որը քողարկված է որպես AI Ստեղծողների համայնք:

ByteDance の Lumi は AI クリエイター コミュニティを装った AI 実験なのか?

ByteDance의 Lumi는 AI 크리에이터 커뮤니티로 위장한 AI 실험일까요?

ByteDance’s Lumi ezali Expérience ya AI oyo e déguisé lokola Communauté ya Créateur ya AI?

Lumi ຂອງ ByteDance ເປັນການທົດລອງ AI ປອມຕົວເປັນຊຸມຊົນຜູ້ສ້າງ AI ບໍ?

Adakah Lumi ByteDance merupakan Eksperimen AI yang Menyamar sebagai Komuniti Pencipta AI?

Je, Lumi ya ByteDance ni Jaribio la AI Limefichwa kama Jumuiya ya Watayarishi wa AI?

Оё Lumi byteDance як озмоиши AI аст, ки ҳамчун ҷомеаи Офарандаи AI пинҳон карда шудааст?

Ngaba i-ByteDance's Lumi liLingelo le-AI eliguqulelwe njengoluntu lwabadali be-AI?

Google's Gemini AI: A Thought Experiment in Deception

Google ukan Gemini AI: ¿Mä amuyt’äw yant’äwix sallqjañataki?

La inteligencia artificial Gemini de Google: ¿un experimento mental sobre el engaño?

AI של Gemini של גוגל: ניסוי מחשבה בהונאה?

Google-ի Gemini AI. Մտքի փորձ խաբեության մեջ:

Google の Gemini AI: 欺瞞の思考実験?

구글의 제미니 AI: 기만에 대한 사고실험?

Google-ийн Gemini AI: Хууран мэхлэлтийн талаархи бодлын туршилт уу?

Gemini AI Google: Eksperimen Pemikiran dalam Penipuan?

د ګوګل جیمني AI: په فریب کې د فکر تجربه؟

Google Gemini AI: Tijaabo Fikirka ah ee Khiyaanada?

Gemini AI ya Google: Jaribio la Mawazo katika Udanganyifu?

Trí tuệ nhân tạo Gemini của Google: Một thí nghiệm tư duy về sự lừa dối?

Authors:
(1) Junwei Su, Department of Computer Science, the University of Hong Kong and jwsu@cs.hku.hk;
(2) Chuan Wu, Department of Computer Science, the University of Hong Kong and cwu@cs.hku.hk. Table of Links Abstract and 1 Introduction 2 Related Work 3 Framework 4 Main Results 5 A Case Study on Shortest-Path Distance 6 Conclusion and Discussion, and References 7 Proof of Theorem 1 8 Proof of Theorem 2 9 Procedure for Solving Eq. (6) 10 Additional Experiments Details and Results 11 Other Potential Applications 10 Additional Experiments Details and Results In this appendix, we provide additional experimental results and include a detailed set-up of the experiments for reproducibility. 10.1 Hardware and Software All the experiments of this paper are conducted on the following machine CPU: two Intel Xeon Gold 6230 2.1G, 20C/40T, 10.4GT/s, 27.5M Cache, Turbo, HT (125W) DDR4-2933 GPU: four NVIDIA Tesla V100 SXM2 32G GPU Accelerator for NV Link Memory: 256GB (8 x 32GB) RDIMM, 3200MT/s, Dual Rank OS: Ubuntu 18.04LTS 10.2 Training Hyper-parameters and Detailed Dataset Description Table 3 gives a more detailed description of the statistic of datasets used in the experiment as well as the hyper-parameters used for training. We keep the hyper-parameters the same across different models. To keep the experiment consistent, we do not employ randomized regularization techniques such as dropout. To ensure a fair comparison, we make sure each experiment is trained till convergence. 10.3 Additional Experiment Results In addition to the datasets we present in the main text, we also conduct experiments on Citeseer, Cora Full and OGBN-arxiv. The results of these experiments show that the theoretical structures proved in this paper are indeed consistent across different datasets. Next, we present additional experimental results on these datasets for validating the main theorems of our papers. Graph Distance and Performance We present additional results with Citeseer, CoraFull, and OGBN-Arxiv on the relation between the graph distance (shortest-path distance). For these experiments, we follow the same setting as the one presented in Fig. 4. We compute the average prediction accuracy in each group of vertices (corresponding to vertices at each hop of the neighbourhood of the training set). ’0 hop’ indicates the training set. We observe that GNNs tend to predict better on the vertices that are closer to the training set. The results are presented in Fig. 6, 7 and 8. Subgroup Embedding Distance and Accuracy We present additional results with Citeseer, CoraFull, and OGBN-Arxiv on that GNNs preserve distances (with a small distortion rate) locally, by evaluating the relation between the graph distances of vertices to the training set and the embedding distances (Euclidean distance) of vertex representations to the representations of the training set, i.e., the relation between d(v, D) and d(MθD (v),MθD (D)). In this experiment, we follow the same setting as the one in Fig. 3 and train the GNN models on the default training set provided in the dataset. We extract vertices within the 5-hop neighbourhood of the training set and group the vertices based on their distances to the training set. We then compute the average embedding distance of each group to the training set in the embedding space. Based on Def. 3, if an embedding has low distortion, we should expect to see linear relation with slop being the scaling factor. As shown in Fig. 9,Fig. 10, and Fig. 11 there exists a strong correlation between the graph distance and the embedding distance. The relation is near linear and the relative orders of distances are mostly preserved within the first few hops. This implies that the distortion α is indeed small with different GNNs and that the common aggregation mechanism of GNNs can well capture local graph structure information (i.e., graph distances). It also indicates that the premise on the distortion rate, which we used to prove our theoretical results, can be commonly satisfied in practice. 10.4 Graph Distance and Performance This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Junwei Su, Department of Computer Science, the University of Hong Kong and jwsu@cs.hku.hk; (2) Chuan Wu, Department of Computer Science, the University of Hong Kong and cwu@cs.hku.hk. Authors: Authors: (1) Junwei Su, Department of Computer Science, the University of Hong Kong and jwsu@cs.hku.hk; (2) Chuan Wu, Department of Computer Science, the University of Hong Kong and cwu@cs.hku.hk. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Framework 3 Framework 4 Main Results 4 Main Results 5 A Case Study on Shortest-Path Distance 5 A Case Study on Shortest-Path Distance 6 Conclusion and Discussion, and References 6 Conclusion and Discussion, and References 7 Proof of Theorem 1 7 Proof of Theorem 1 8 Proof of Theorem 2 8 Proof of Theorem 2 9 Procedure for Solving Eq. (6) 9 Procedure for Solving Eq. (6) 10 Additional Experiments Details and Results 10 Additional Experiments Details and Results 11 Other Potential Applications 11 Other Potential Applications 10 Additional Experiments Details and Results In this appendix, we provide additional experimental results and include a detailed set-up of the experiments for reproducibility. 10.1 Hardware and Software All the experiments of this paper are conducted on the following machine CPU: two Intel Xeon Gold 6230 2.1G, 20C/40T, 10.4GT/s, 27.5M Cache, Turbo, HT (125W) DDR4-2933 GPU: four NVIDIA Tesla V100 SXM2 32G GPU Accelerator for NV Link Memory: 256GB (8 x 32GB) RDIMM, 3200MT/s, Dual Rank OS: Ubuntu 18.04LTS 10.2 Training Hyper-parameters and Detailed Dataset Description Table 3 gives a more detailed description of the statistic of datasets used in the experiment as well as the hyper-parameters used for training. We keep the hyper-parameters the same across different models. To keep the experiment consistent, we do not employ randomized regularization techniques such as dropout. To ensure a fair comparison, we make sure each experiment is trained till convergence. 10.3 Additional Experiment Results In addition to the datasets we present in the main text, we also conduct experiments on Citeseer, Cora Full and OGBN-arxiv. The results of these experiments show that the theoretical structures proved in this paper are indeed consistent across different datasets. Next, we present additional experimental results on these datasets for validating the main theorems of our papers. Graph Distance and Performance We present additional results with Citeseer, CoraFull, and OGBN-Arxiv on the relation between the graph distance (shortest-path distance). For these experiments, we follow the same setting as the one presented in Fig. 4. We compute the average prediction accuracy in each group of vertices (corresponding to vertices at each hop of the neighbourhood of the training set). ’0 hop’ indicates the training set. We observe that GNNs tend to predict better on the vertices that are closer to the training set. The results are presented in Fig. 6, 7 and 8. Graph Distance and Performance Subgroup Embedding Distance and Accuracy We present additional results with Citeseer, CoraFull, and OGBN-Arxiv on that GNNs preserve distances (with a small distortion rate) locally, by evaluating the relation between the graph distances of vertices to the training set and the embedding distances (Euclidean distance) of vertex representations to the representations of the training set, i.e., the relation between d(v, D) and d(MθD (v),MθD (D)). In this experiment, we follow the same setting as the one in Fig. 3 and train the GNN models on the default training set provided in the dataset. We extract vertices within the 5-hop neighbourhood of the training set and group the vertices based on their distances to the training set. We then compute the average embedding distance of each group to the training set in the embedding space. Based on Def. 3, if an embedding has low distortion, we should expect to see linear relation with slop being the scaling factor. As shown in Fig. 9,Fig. 10, and Fig. 11 there exists a strong correlation between the graph distance and the embedding distance. The relation is near linear and the relative orders of distances are mostly preserved within the first few hops. This implies that the distortion α is indeed small with different GNNs and that the common aggregation mechanism of GNNs can well capture local graph structure information (i.e., graph distances). It also indicates that the premise on the distortion rate, which we used to prove our theoretical results, can be commonly satisfied in practice. Subgroup Embedding Distance and Accuracy 10.4 Graph Distance and Performance This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Comprehensive Overview of GNN Experiments: Hardware, Hyperparameters, and Findings

Authors:
(1) Divyanshu Kumar, Enkrypt AI;
(2) Anurakt Kumar, Enkrypt AI;
(3) Sahil Agarwa, Enkrypt AI;
(4) Prashanth Harshangi, Enkrypt AI. Table of Links Abstract and 1 Introduction 2 Problem Formulation and Experiments 3 Experiment Set-up & Results 4 Conclusion and References A. Appendix 3 EXPERIMENT SET-UP & RESULTS The following section outlines the testing environments utilized for various LLMs. The LLMs are tested under three different downstream tasks: (1) fine-tuning, (2) quantization, and (3) guardrails. They are chosen to cover most of the LLM practical use cases and applications in the industry and academia. For TAP configuration, as mentioned before, we use GPT-3.5-turbo as the attack model, and GPT-4-turbo as the judge model. We employ Anyscale endpoints, the OpenAI API, and HuggingFace for our target model. Further information regarding the model and its sources is available in appendix A.1. The details about different settings are mentioned below : • Fine-tuning: Fine-tuning LLMs on different tasks increases their effectiveness in completing the tasks as it incorporates the specialized domain knowledge needed; these can include SQL code generation, chat, and more. We compare the jailbreaking vulnerability of foundational models compared to their fine tune versions. This helps us understand the role of fine-tuning in increasing or decreasing the vulnerability of LLMs and the strategies to mitigate this risk Weyssow et al. (2023). We use foundation models such as Llama2, Mistral, and MPT-7B and their fine-tuned versions such as CodeLlama, SQLCoder, Dolphin, and Intel Neural Chat. The details of the models and versions are specified in the appendix. From the table 1, we empirically conclude that fine-tuned models lose their safety alignment and are jailbroken quite easily compared to the foundational models. • Quantization: Many models require huge computational resources during training, finetuning, and even during inference Hu et al. (2021). Quantization is one of the most popular ways to reduce the computational burden, but it comes at the cost of the numerical precision of model parameters. The quantized models we evaluate below use GPT-Generated Unified Format (GGUF) for quantization, which involves scaling down model weights (stored in 16-bit floating point numbers) to save computational resources at the cost of numerical precision of the model parameters. Kashiwamura et al. (2024) Gorsline et al. (2021) Xiao et al.(2023) Hu et al. (2021) Dettmers et al. (2023). The table 2 demonstrates that quantization of the model renders it susceptible to vulnerabilities. • Guardrails: Guardrails act as a line of defence against LLM attacks. Their primary function is to meticulously filter out prompts that could potentially lead to harmful or malicious outcomes, preventing such prompts from reaching the LLM as an instruction Rebedea et al. (2023). This proactive approach not only promotes the safe utilization of LLMs but also facilitates their optimal performance, thereby maximizing their potential benefits in various domains Kumar et al. (2023) Wei et al. (2023) Zhou et al. (2024). We use our proprietary jailbreak attack detector derived from Deberta-V3 models, trained on harmful prompts generated to jailbreak LLMs. You can reach out to the authors to get more details on this model. From table 3, we observe that the introduction of guardrails as a pre-step has a significant effect and can mitigate jailbreaking attempts by a considerable margin. The results from tables 1, 2, and 3 conclusively show the vulnerability of LLMs post fine-tuning and quantization, and effectiveness of external guardrails in mitigating this vulnerability. The detailed experimental results can be found in appendix A.2. Additionally, this section provides a comprehensive analysis of the experiments conducted, offering insights into the various aspects of the research findings. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Divyanshu Kumar, Enkrypt AI; (2) Anurakt Kumar, Enkrypt AI; (3) Sahil Agarwa, Enkrypt AI; (4) Prashanth Harshangi, Enkrypt AI. Authors: Authors: (1) Divyanshu Kumar, Enkrypt AI; (2) Anurakt Kumar, Enkrypt AI; (3) Sahil Agarwa, Enkrypt AI; (4) Prashanth Harshangi, Enkrypt AI. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Problem Formulation and Experiments 2 Problem Formulation and Experiments 3 Experiment Set-up & Results 3 Experiment Set-up & Results 4 Conclusion and References 4 Conclusion and References A. Appendix A. Appendix 3 EXPERIMENT SET-UP & RESULTS The following section outlines the testing environments utilized for various LLMs. The LLMs are tested under three different downstream tasks: (1) fine-tuning, (2) quantization, and (3) guardrails. They are chosen to cover most of the LLM practical use cases and applications in the industry and academia. For TAP configuration, as mentioned before, we use GPT-3.5-turbo as the attack model, and GPT-4-turbo as the judge model. We employ Anyscale endpoints, the OpenAI API, and HuggingFace for our target model. Further information regarding the model and its sources is available in appendix A.1. The details about different settings are mentioned below : • Fine-tuning: Fine-tuning LLMs on different tasks increases their effectiveness in completing the tasks as it incorporates the specialized domain knowledge needed; these can include SQL code generation, chat, and more. We compare the jailbreaking vulnerability of foundational models compared to their fine tune versions. This helps us understand the role of fine-tuning in increasing or decreasing the vulnerability of LLMs and the strategies to mitigate this risk Weyssow et al. (2023). We use foundation models such as Llama2, Mistral, and MPT-7B and their fine-tuned versions such as CodeLlama, SQLCoder, Dolphin, and Intel Neural Chat. The details of the models and versions are specified in the appendix. From the table 1, we empirically conclude that fine-tuned models lose their safety alignment and are jailbroken quite easily compared to the foundational models. • Fine-tuning: • Quantization : Many models require huge computational resources during training, finetuning, and even during inference Hu et al. (2021). Quantization is one of the most popular ways to reduce the computational burden, but it comes at the cost of the numerical precision of model parameters. The quantized models we evaluate below use GPT-Generated Unified Format (GGUF) for quantization, which involves scaling down model weights (stored in 16-bit floating point numbers) to save computational resources at the cost of numerical precision of the model parameters. Kashiwamura et al. (2024) Gorsline et al. (2021) Xiao et al.(2023) Hu et al. (2021) Dettmers et al. (2023). The table 2 demonstrates that quantization of the model renders it susceptible to vulnerabilities. Quantization • Guardrails : Guardrails act as a line of defence against LLM attacks. Their primary function is to meticulously filter out prompts that could potentially lead to harmful or malicious outcomes, preventing such prompts from reaching the LLM as an instruction Rebedea et al. (2023). This proactive approach not only promotes the safe utilization of LLMs but also facilitates their optimal performance, thereby maximizing their potential benefits in various domains Kumar et al. (2023) Wei et al. (2023) Zhou et al. (2024). We use our proprietary jailbreak attack detector derived from Deberta-V3 models, trained on harmful prompts generated to jailbreak LLMs. You can reach out to the authors to get more details on this model. From table 3, we observe that the introduction of guardrails as a pre-step has a significant effect and can mitigate jailbreaking attempts by a considerable margin. Guardrails The results from tables 1, 2, and 3 conclusively show the vulnerability of LLMs post fine-tuning and quantization, and effectiveness of external guardrails in mitigating this vulnerability. The detailed experimental results can be found in appendix A.2. Additionally, this section provides a comprehensive analysis of the experiments conducted, offering insights into the various aspects of the research findings. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Increased LLM Vulnerabilities from Fine-tuning and Quantization: Experiment Set-up & Results

Authors:
(1) Divyanshu Kumar, Enkrypt AI;
(2) Anurakt Kumar, Enkrypt AI;
(3) Sahil Agarwa, Enkrypt AI;
(4) Prashanth Harshangi, Enkrypt AI. Table of Links Abstract and 1 Introduction 2 Problem Formulation and Experiments 3 Experiment Set-up & Results 4 Conclusion and References A. Appendix 2 PROBLEM FORMULATION AND EXPERIMENTS We want to understand the role played by fine-tuning, quantization, and guardrails on LLM’s vulnerability towards jailbreaking attacks. We create a pipeline to test for jailbreaking of LLMs which undergo these further processes before deployment. As mentioned in the introduction, we attack the LLM via the TAP algorithm using the AdvBench subset. We use a subset of AdvBenchAndy Zou (2023). It contains 50 prompts asking for harmful information across 32 categories. The evaluation results, along with the complete system information, are then logged. The overall flow is shown in the figure 2. This process continues for multiple iterations, taking into account the stochastic nature associated with LLMs. The complete experiment pipeline is shown in Figure 2. TAP Mehrotra et al. (2023) is used as the jailbreaking method, as it is currently the state-of-the-art, black-box, and automatic method which generates prompts with semantic meaning to jailbreak LLMs. TAP algorithm uses an attacker LLM A, which sends a prompt P to the target LLM T. The response of the target LLM R along with the prompt P are fed into the evaluator LLM JUDGE, which determines if the prompt is on-topic or off-topic. If the prompt is off-topic, it is removed, thereby eliminating the tree of bad attack prompts it was going to generate. Otherwise, if the prompt is on-topic, it receives a score between 0-10 by the JUDGE LLM. This prompt is used to generate a new attack prompt using breadth-first search. This process continues till the LLM is jailbroken or a specified number of iterations are exhausted. Now describing our guardrail against jailbreaking prompts, we use our in-house Deberta-V3 model, which has been trained to detect jailbreaking prompts. It acts as an input filter to ensure that only sanitized prompts are received by the LLM. If the input prompt is filtered out by the guardrail or fails to jailbreak the LLM, then the TAP algorithm generates a new prompt considering the initial prompt and response. The new attacking prompt is then again passed through the guardrail. This process is repeated till a jailbreaking prompt is found or a pre-specified number of iterations are exhausted. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Divyanshu Kumar, Enkrypt AI; (2) Anurakt Kumar, Enkrypt AI; (3) Sahil Agarwa, Enkrypt AI; (4) Prashanth Harshangi, Enkrypt AI. Authors: Authors: (1) Divyanshu Kumar, Enkrypt AI; (2) Anurakt Kumar, Enkrypt AI; (3) Sahil Agarwa, Enkrypt AI; (4) Prashanth Harshangi, Enkrypt AI. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Problem Formulation and Experiments 2 Problem Formulation and Experiments 3 Experiment Set-up & Results 3 Experiment Set-up & Results 4 Conclusion and References 4 Conclusion and References A. Appendix A. Appendix 2 PROBLEM FORMULATION AND EXPERIMENTS We want to understand the role played by fine-tuning, quantization, and guardrails on LLM’s vulnerability towards jailbreaking attacks. We create a pipeline to test for jailbreaking of LLMs which undergo these further processes before deployment. As mentioned in the introduction, we attack the LLM via the TAP algorithm using the AdvBench subset. We use a subset of AdvBenchAndy Zou (2023). It contains 50 prompts asking for harmful information across 32 categories. The evaluation results, along with the complete system information, are then logged. The overall flow is shown in the figure 2. This process continues for multiple iterations, taking into account the stochastic nature associated with LLMs. The complete experiment pipeline is shown in Figure 2. TAP Mehrotra et al. (2023) is used as the jailbreaking method, as it is currently the state-of-the-art, black-box, and automatic method which generates prompts with semantic meaning to jailbreak LLMs. TAP algorithm uses an attacker LLM A, which sends a prompt P to the target LLM T. The response of the target LLM R along with the prompt P are fed into the evaluator LLM JUDGE, which determines if the prompt is on-topic or off-topic. If the prompt is off-topic, it is removed, thereby eliminating the tree of bad attack prompts it was going to generate. Otherwise, if the prompt is on-topic, it receives a score between 0-10 by the JUDGE LLM. This prompt is used to generate a new attack prompt using breadth-first search. This process continues till the LLM is jailbroken or a specified number of iterations are exhausted. Now describing our guardrail against jailbreaking prompts, we use our in-house Deberta-V3 model, which has been trained to detect jailbreaking prompts. It acts as an input filter to ensure that only sanitized prompts are received by the LLM. If the input prompt is filtered out by the guardrail or fails to jailbreak the LLM, then the TAP algorithm generates a new prompt considering the initial prompt and response. The new attacking prompt is then again passed through the guardrail. This process is repeated till a jailbreaking prompt is found or a pre-specified number of iterations are exhausted. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Increased LLM Vulnerabilities from Fine-tuning and Quantization: Problem Formulation and Experiments

Authors:
(1) Jianhui Pang, from the University of Macau, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (nlp2ct.pangjh3@gmail.com);
(2) Fanghua Ye, University College London, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (fanghua.ye.19@ucl.ac.uk);
(3) Derek F. Wong, University of Macau;
(4) Longyue Wang, Tencent AI Lab, and corresponding author. Table of Links Abstract and 1 Introduction 2 Related Work 3 Anchor-based Large Language Models 3.1 Background 3.2 Anchor-based Self-Attention Networks 3.3 Anchor-based Inference 4 Experiments and 4.1 Our Implementation 4.2 Data and Training Procedure 4.3 Evaluation 5 Results 6 Analysis 7 Conclusion, Limitations, Ethics Statement, and References A More Experimental Results B Data Settings A More Experimental Results A.1 Testing Acceleration Ratio to Full-Caching Method In Section 5, we report the testing acceleration ratio following the setting of Wang et al. (2023), comparing the time difference between caching and non-caching inference. Although our method reduces the keys/values caches, enabling smaller space for prefix information storage and improving testing time up to ×3.5, we are still curious about whether it would enhance inference efficiency if conventional methods use full caches that save all keys/values of prefix tokens. As a supplement to Table 1, we present the testing acceleration ratio between anchor-caching and full-caching inference in Table 3. The acceleration ratios for AnLLMEP-AnSAN and AnLLM-AC-AnSAN achieve the highest improvements observed in tasks such as HS, SCIQ, and BoolQ. The average acceleration ratios for AnLLM-EP-AnSAN and AnLLM-ACAnSAN are 1.03. These results demonstrate that our anchor-based caching method can enhance inference efficiency even when compared to conventional methods that save all keys/values of prefix tokens. These results suggest that our anchor-based caching approach, which saves only the keys/values caches of anchor tokens, can effectively accelerate the inference process for the lengthy prefix texts. A.2 Model Scalability Assessment To examine the scalability of our approach, we extend the AnLLM-AC model to 13B and assess its performance on eight question-answering benchmarks using the same evaluation strategy as previously mentioned. In comparison to the 7B AnLLM models in Table 1, Results in Table 4 indicate that as the model size expands, the AnLLM-AC model achieves accuracies of 67.5% and 70.0% for 0-shot and 5-shot testing, respectively, resulting in up to a 2.4% improvement. Moreover, by incorporating anchor-based attention, the AnLLM-AC-AnSAN model achieves an average accuracy of 69.5%, signifying a 2.0% increase. The performance enhancement underscores the effectiveness of our methods in accommodating larger model capacities. The consistent improvements observed in the AnLLMAC model across various scenarios highlight its robustness and adaptability. Furthermore, the increased performance of the AnLLM-AC-AnSAN model, facilitated by anchor-based attention, emphasizes the potential of our approaches in optimizing LLMs. Collectively, these findings point to promising avenues for future research aimed at maximizing the utility and efficiency of AnLLM. A.3 Case Study in Real-Time Inference To elaborate on the optimization of keys/values caches by AnLLM-EP and AnLLM-AC during real-time inference, we reference examples from the translation task in Section 6.2. As per Table 5, AnLLM-EP and AnLLM-AC use "endpoints" (".") and "" tokens as anchor tokens, respectively. During inference, both models employ auto-regressive generation, creating outputs token-by-token. Upon generating an anchor token (as per Line 16, Algorithm 1), the Reduction function (defined in Line 1) is activated, preserving relevant caches and eliminating others. As a result, the Keys/Values Cache lengths are reduced to roughly the sequence length, averaging around 50 for AnLLM-EP and 35 for AnLLM-AC, as shown in Table 2. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Jianhui Pang, from the University of Macau, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (nlp2ct.pangjh3@gmail.com); (2) Fanghua Ye, University College London, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (fanghua.ye.19@ucl.ac.uk); (3) Derek F. Wong, University of Macau; (4) Longyue Wang, Tencent AI Lab, and corresponding author. Authors: Authors: (1) Jianhui Pang, from the University of Macau, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (nlp2ct.pangjh3@gmail.com); (2) Fanghua Ye, University College London, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (fanghua.ye.19@ucl.ac.uk); (3) Derek F. Wong, University of Macau; (4) Longyue Wang, Tencent AI Lab, and corresponding author. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Anchor-based Large Language Models 3.1 Background 3.1 Background 3.2 Anchor-based Self-Attention Networks 3.2 Anchor-based Self-Attention Networks 3.3 Anchor-based Inference 3.3 Anchor-based Inference 4 Experiments and 4.1 Our Implementation 4 Experiments and 4.1 Our Implementation 4.2 Data and Training Procedure 4.2 Data and Training Procedure 4.3 Evaluation 4.3 Evaluation 5 Results 5 Results 6 Analysis 6 Analysis 7 Conclusion, Limitations, Ethics Statement, and References 7 Conclusion, Limitations, Ethics Statement, and References A More Experimental Results A More Experimental Results B Data Settings B Data Settings A More Experimental Results A.1 Testing Acceleration Ratio to Full-Caching Method In Section 5, we report the testing acceleration ratio following the setting of Wang et al. (2023), comparing the time difference between caching and non-caching inference. Although our method reduces the keys/values caches, enabling smaller space for prefix information storage and improving testing time up to ×3.5, we are still curious about whether it would enhance inference efficiency if conventional methods use full caches that save all keys/values of prefix tokens. As a supplement to Table 1, we present the testing acceleration ratio between anchor-caching and full-caching inference in Table 3. The acceleration ratios for AnLLMEP-AnSAN and AnLLM-AC-AnSAN achieve the highest improvements observed in tasks such as HS, SCIQ, and BoolQ. The average acceleration ratios for AnLLM-EP-AnSAN and AnLLM-ACAnSAN are 1.03. These results demonstrate that our anchor-based caching method can enhance inference efficiency even when compared to conventional methods that save all keys/values of prefix tokens. These results suggest that our anchor-based caching approach, which saves only the keys/values caches of anchor tokens, can effectively accelerate the inference process for the lengthy prefix texts. A.2 Model Scalability Assessment To examine the scalability of our approach, we extend the AnLLM-AC model to 13B and assess its performance on eight question-answering benchmarks using the same evaluation strategy as previously mentioned. In comparison to the 7B AnLLM models in Table 1, Results in Table 4 indicate that as the model size expands, the AnLLM-AC model achieves accuracies of 67.5% and 70.0% for 0-shot and 5-shot testing, respectively, resulting in up to a 2.4% improvement. Moreover, by incorporating anchor-based attention, the AnLLM-AC-AnSAN model achieves an average accuracy of 69.5%, signifying a 2.0% increase. The performance enhancement underscores the effectiveness of our methods in accommodating larger model capacities. The consistent improvements observed in the AnLLMAC model across various scenarios highlight its robustness and adaptability. Furthermore, the increased performance of the AnLLM-AC-AnSAN model, facilitated by anchor-based attention, emphasizes the potential of our approaches in optimizing LLMs. Collectively, these findings point to promising avenues for future research aimed at maximizing the utility and efficiency of AnLLM. A.3 Case Study in Real-Time Inference To elaborate on the optimization of keys/values caches by AnLLM-EP and AnLLM-AC during real-time inference, we reference examples from the translation task in Section 6.2. As per Table 5, AnLLM-EP and AnLLM-AC use "endpoints" (".") and "" tokens as anchor tokens, respectively. During inference, both models employ auto-regressive generation, creating outputs token-by-token. Upon generating an anchor token (as per Line 16, Algorithm 1), the Reduction function (defined in Line 1) is activated, preserving relevant caches and eliminating others. As a result, the Keys/Values Cache lengths are reduced to roughly the sequence length, averaging around 50 for AnLLM-EP and 35 for AnLLM-AC, as shown in Table 2. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Anchor-based Large Language Models: More Experimental Results

Authors:
(1) Jianhui Pang, from the University of Macau, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (nlp2ct.pangjh3@gmail.com);
(2) Fanghua Ye, University College London, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (fanghua.ye.19@ucl.ac.uk);
(3) Derek F. Wong, University of Macau;
(4) Longyue Wang, Tencent AI Lab, and corresponding author. Table of Links Abstract and 1 Introduction 2 Related Work 3 Anchor-based Large Language Models 3.1 Background 3.2 Anchor-based Self-Attention Networks 3.3 Anchor-based Inference 4 Experiments and 4.1 Our Implementation 4.2 Data and Training Procedure 4.3 Evaluation 5 Results 6 Analysis 7 Conclusion, Limitations, Ethics Statement, and References A More Experimental Results B Data Settings 4 Experiments In this section, we first detail AnLLM’s implementation, then outline the training procedure and model perplexity. Finally, we introduce the evaluation datasets and metrics. 4.1 Our Implementation Llama2-7b (Touvron et al., 2023b) is adopted as the base model in our experiments, which is an open-source and English-centric LLM. In accordance with the principles outlined in Section 3, we present our implementations here. The crux is to identify which tokens in a sequence can be considered anchor tokens. In light of this, we describe two implementation strategies: one employs the endpoints directly, and the other involves appending a new token at the end of a sequence to serve as the anchor token. The details are as follows: • AnLLM-EP. This approach uses punctuation marks within the sequence as anchor tokens. Punctuation marks, such as commas, periods, and question marks, are viewed as semantic boundaries within a sequence. As such, they can serve as anchor tokens in AnLLM. In our experiments of AnLLM-EP, we use the endpoint in English as the anchor tokens. • AnLLM-AC. This strategy entails the introduction of a new token to act as the sequence anchor. In our implementation, we designate as the new token and initialize its embedding using the mean value of the embedding matrix. For training data, we use the sentence tokenizer from the NLTK package to split texts into sentences, appending at the end of each sentence as the anchor token.[1] During inference, tokens can be flexibly added to the text based on user requirements, such as adding one anchor for each demonstration, allowing for flexible and controllable sequence compression. This paper is available on arxiv under CC BY 4.0 DEED license. [1] https://www.nltk.org/api/nltk.tokenize.punkt. html Authors: (1) Jianhui Pang, from the University of Macau, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (nlp2ct.pangjh3@gmail.com); (2) Fanghua Ye, University College London, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (fanghua.ye.19@ucl.ac.uk); (3) Derek F. Wong, University of Macau; (4) Longyue Wang, Tencent AI Lab, and corresponding author. Authors: Authors: (1) Jianhui Pang, from the University of Macau, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (nlp2ct.pangjh3@gmail.com); (2) Fanghua Ye, University College London, and work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab (fanghua.ye.19@ucl.ac.uk); (3) Derek F. Wong, University of Macau; (4) Longyue Wang, Tencent AI Lab, and corresponding author. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Anchor-based Large Language Models 3.1 Background 3.1 Background 3.2 Anchor-based Self-Attention Networks 3.2 Anchor-based Self-Attention Networks 3.3 Anchor-based Inference 3.3 Anchor-based Inference 4 Experiments and 4.1 Our Implementation 4 Experiments and 4.1 Our Implementation 4.2 Data and Training Procedure 4.2 Data and Training Procedure 4.3 Evaluation 4.3 Evaluation 5 Results 5 Results 6 Analysis 6 Analysis 7 Conclusion, Limitations, Ethics Statement, and References 7 Conclusion, Limitations, Ethics Statement, and References A More Experimental Results A More Experimental Results B Data Settings B Data Settings 4 Experiments In this section, we first detail AnLLM’s implementation, then outline the training procedure and model perplexity. Finally, we introduce the evaluation datasets and metrics. 4.1 Our Implementation Llama2-7b (Touvron et al., 2023b) is adopted as the base model in our experiments, which is an open-source and English-centric LLM. In accordance with the principles outlined in Section 3, we present our implementations here. The crux is to identify which tokens in a sequence can be considered anchor tokens. In light of this, we describe two implementation strategies: one employs the endpoints directly, and the other involves appending a new token at the end of a sequence to serve as the anchor token. The details are as follows: • AnLLM-EP. This approach uses punctuation marks within the sequence as anchor tokens. Punctuation marks, such as commas, periods, and question marks, are viewed as semantic boundaries within a sequence. As such, they can serve as anchor tokens in AnLLM. In our experiments of AnLLM-EP, we use the endpoint in English as the anchor tokens. • AnLLM-EP. • AnLLM-AC. This strategy entails the introduction of a new token to act as the sequence anchor. In our implementation, we designate as the new token and initialize its embedding using the mean value of the embedding matrix. For training data, we use the sentence tokenizer from the NLTK package to split texts into sentences, appending at the end of each sentence as the anchor token.[1] During inference, tokens can be flexibly added to the text based on user requirements, such as adding one anchor for each demonstration, allowing for flexible and controllable sequence compression. • AnLLM-AC. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [1] https://www.nltk.org/api/nltk.tokenize.punkt. html

Anchor-based Large Language Models: Experiments and Implementation

Machine learning experiments require extensive parametrization, including optimizer parameters, network architecture, and data augmentation. However, we strive for concise, readable code instead of a cumbersome 200 lines dedicated to argparse. Our goal is to focus on programming logic rather than threading new parameters through function signatures. Additionally, we seek a structure that is easily expandable without burdening the project, ensuring the reproducibility of experiments. Hydra offers a solution to these challenges. Below, you will find a basic guide on how to use it. What is Hydra? Hydra is a library with rich capabilities for managing configurations. The main site describes the name like this: “The key feature is the ability to dynamically create a hierarchical configuration by composition and override it through config files and the command line.The name Hydra comes from its ability to run multiple similar jobs - much like a Hydra with multiple heads.” But I have my own interpretation of its name. It is just a combination of some tools that work together impressively well! Hydra offers a seamless solution to the common headaches faced by ML engineers when attempting to replicate experiments. It elegantly replaces the need for argparse or YAML configurations, allowing access to parameters both from the command line and YAML files. Consider the pain points: Replicating experiments using argparse forces reliance on string inputs and complicates YAML config launches. Relying solely on YAML files for configuration leads to duplication and potential errors when only a single parameter needs alteration. Hydra addresses these issues by enabling dynamic configuration adjustments without the need for multiple bulky files or rigid command-line arguments. Furthermore, it simplifies the process of passing complex configurations, such as model architectures or functions, directly from the config file to the model. This capability eliminates the tedious task of manually feeding parameters into the model, streamlining the workflow and reducing the margin for error. Basic Setup Let's imagine the simplest setup: multiclass classification on MNIST using an MLP. We have a configuration and a training script. .
├── configs
│   └── config.yaml
└── main.py The main script can look like this. import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

import hydra
from omegaconf import DictConfig

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.model = nn.Sequential(
            nn.Linear(28*28, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 10)
        )
    
    def forward(self, x):
        return self.model(x.view(-1, 28*28))

@hydra.main(version_base=None, config_path=None, config_name="config")
def main(cfg: DictConfig):
    # Load MNIST dataset
    transform = transforms.Compose([
        transforms.ToTensor(), 
        transforms.Normalize((0.5,), (0.5,))
    ])
    train_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
    train_loader = DataLoader(train_dataset, batch_size=cfg.batch_size, shuffle=True)

    # Initialize the network, loss function, and optimizer
    model = Net()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=cfg.lr, momentum=cfg.momentum)

    # Train the network
    for epoch in range(cfg.epochs):  # loop over the dataset multiple times
        for i, (inputs, labels) in enumerate(train_loader, 0):
            optimizer.zero_grad()  # zero the parameter gradients
            outputs = model(inputs)  # forward pass
            loss = criterion(outputs, labels)  # calculate loss
            loss.backward()  # backward pass
            optimizer.step()  # optimize

    print('Finished Training')

if __name__ == "__main__":
    main() The configuration has the following structure. It is the default, and parameters from it will be used by default if not specified otherwise. # conf/config.yaml
batch_size: 64
lr: 0.01
momentum: 0.9
epochs: 1 To start training you can use: python main.py And for changing parameters, you can do it not only from YAML but also from cli: python main.py lr=0.03 If you use it, parameters from cli override the first parameters. As you can see, there is no argparse or additional middleware. It's easy to change parameters from the command line. Pass Class Object With Hydra We see that in class Net, the architecture can be highly customizable. Even more, we can use also another net for this, e.g. CNN. But we do not want to change our pipeline. Hydra can construct almost any Python object with specified parameters. Let’s describe our net in another way, using YAML. model:
  _target_: torch.nn.Sequential
  layers:
    - _target_: torch.nn.Flatten
    - _target_: torch.nn.Linear
      in_features: 784  # 28x28 images are flattened into 784
      out_features: 128
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.Linear
      in_features: 128
      out_features: 64
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.Linear
      in_features: 64
      out_features: 10 and for CNN _target_: torch.nn.Sequential
  layers:
    - _target_: torch.nn.Conv2d
      in_channels: 1  # MNIST images are grayscale, so 1 input channel
      out_channels: 32  # Number of output channels
      kernel_size: 3  # Size of the convolutional kernel
      stride: 1
      padding: 1
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.MaxPool2d
      kernel_size: 2  # Pooling window size
      stride: 2
    - _target_: torch.nn.Conv2d
      in_channels: 32
      out_channels: 64
      kernel_size: 3
      stride: 1
      padding: 1
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.MaxPool2d
      kernel_size: 2
      stride: 2
    - _target_: torch.nn.Flatten  # Flatten the output for the fully connected layer
    - _target_: torch.nn.Linear
      in_features: 7*7*64  # Size after convolutions and pooling
      out_features: 128
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.Dropout
      p: 0.5  # Dropout rate
    - _target_: torch.nn.Linear
      in_features: 128
      out_features: 10  # Number of classes in MNIST Pass Function Through Config Sometimes, our parameters can also function. For example, How to Access Config Without Decorating Main - e.g., in ipython Notebook. So, you can use Compose API. from hydra import compose, initialize from omegaconf import OmegaConf

with initialize(version_base=None, config_path="conf", job_name="run_0001"):
  cfg = compose(config_name="config", overrides= ["parameter=value"])

print (OmegaConf.to_yaml(cfg)) Multi Runs For example, you want to find some good hyperparam and Hydra allows a lot for that. Basically, you can launch several experiments sequentially like. python train_model.py -m "batch_size=16,32,64" It launches sequentially, but it is easy to extend to parallel if turned on some launcher like joblib. python train_model.py -m "batch_size=16,32,64 hydra/launcher=joblib" Run All Experiments From the Folder for config_file in configs/*.yaml; do python main.py --config-name="${config_file}"; done Resources hydra - https://hydra.cc/
good template for DL - https://github.com/ashleve/lightning-hydra-template/tree/main Machine learning experiments require extensive parametrization, including optimizer parameters, network architecture, and data augmentation. However, we strive for concise, readable code instead of a cumbersome 200 lines dedicated to argparse. Our goal is to focus on programming logic rather than threading new parameters through function signatures. Additionally, we seek a structure that is easily expandable without burdening the project, ensuring the reproducibility of experiments. Hydra offers a solution to these challenges. Below, you will find a basic guide on how to use it. What is Hydra? Hydra is a library with rich capabilities for managing configurations. The main site describes the name like this: “The key feature is the ability to dynamically create a hierarchical configuration by composition and override it through config files and the command line.The name Hydra comes from its ability to run multiple similar jobs - much like a Hydra with multiple heads.” “The key feature is the ability to dynamically create a hierarchical configuration by composition and override it through config files and the command line. The name Hydra comes from its ability to run multiple similar jobs - much like a Hydra with multiple heads.” But I have my own interpretation of its name. It is just a combination of some tools that work together impressively well! Hydra offers a seamless solution to the common headaches faced by ML engineers when attempting to replicate experiments. It elegantly replaces the need for argparse or YAML configurations, allowing access to parameters both from the command line and YAML files. Consider the pain points: Replicating experiments using argparse forces reliance on string inputs and complicates YAML config launches. Replicating experiments using argparse forces reliance on string inputs and complicates YAML config launches. Relying solely on YAML files for configuration leads to duplication and potential errors when only a single parameter needs alteration. Relying solely on YAML files for configuration leads to duplication and potential errors when only a single parameter needs alteration. Hydra addresses these issues by enabling dynamic configuration adjustments without the need for multiple bulky files or rigid command-line arguments. Furthermore, it simplifies the process of passing complex configurations, such as model architectures or functions, directly from the config file to the model. This capability eliminates the tedious task of manually feeding parameters into the model, streamlining the workflow and reducing the margin for error. Basic Setup Let's imagine the simplest setup: multiclass classification on MNIST using an MLP. We have a configuration and a training script. .
├── configs
│   └── config.yaml
└── main.py .
├── configs
│   └── config.yaml
└── main.py The main script can look like this. import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

import hydra
from omegaconf import DictConfig

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.model = nn.Sequential(
            nn.Linear(28*28, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 10)
        )
    
    def forward(self, x):
        return self.model(x.view(-1, 28*28))

@hydra.main(version_base=None, config_path=None, config_name="config")
def main(cfg: DictConfig):
    # Load MNIST dataset
    transform = transforms.Compose([
        transforms.ToTensor(), 
        transforms.Normalize((0.5,), (0.5,))
    ])
    train_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
    train_loader = DataLoader(train_dataset, batch_size=cfg.batch_size, shuffle=True)

    # Initialize the network, loss function, and optimizer
    model = Net()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=cfg.lr, momentum=cfg.momentum)

    # Train the network
    for epoch in range(cfg.epochs):  # loop over the dataset multiple times
        for i, (inputs, labels) in enumerate(train_loader, 0):
            optimizer.zero_grad()  # zero the parameter gradients
            outputs = model(inputs)  # forward pass
            loss = criterion(outputs, labels)  # calculate loss
            loss.backward()  # backward pass
            optimizer.step()  # optimize

    print('Finished Training')

if __name__ == "__main__":
    main() import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

import hydra
from omegaconf import DictConfig

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.model = nn.Sequential(
            nn.Linear(28*28, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 10)
        )
    
    def forward(self, x):
        return self.model(x.view(-1, 28*28))

@hydra.main(version_base=None, config_path=None, config_name="config")
def main(cfg: DictConfig):
    # Load MNIST dataset
    transform = transforms.Compose([
        transforms.ToTensor(), 
        transforms.Normalize((0.5,), (0.5,))
    ])
    train_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
    train_loader = DataLoader(train_dataset, batch_size=cfg.batch_size, shuffle=True)

    # Initialize the network, loss function, and optimizer
    model = Net()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=cfg.lr, momentum=cfg.momentum)

    # Train the network
    for epoch in range(cfg.epochs):  # loop over the dataset multiple times
        for i, (inputs, labels) in enumerate(train_loader, 0):
            optimizer.zero_grad()  # zero the parameter gradients
            outputs = model(inputs)  # forward pass
            loss = criterion(outputs, labels)  # calculate loss
            loss.backward()  # backward pass
            optimizer.step()  # optimize

    print('Finished Training')

if __name__ == "__main__":
    main() The configuration has the following structure. It is the default, and parameters from it will be used by default if not specified otherwise. # conf/config.yaml
batch_size: 64
lr: 0.01
momentum: 0.9
epochs: 1 # conf/config.yaml
batch_size: 64
lr: 0.01
momentum: 0.9
epochs: 1 To start training you can use: python main.py python main.py And for changing parameters, you can do it not only from YAML but also from cli: python main.py lr=0.03 python main.py lr=0.03 If you use it, parameters from cli override the first parameters. As you can see, there is no argparse or additional middleware. It's easy to change parameters from the command line. Pass Class Object With Hydra We see that in class Net, the architecture can be highly customizable. Even more, we can use also another net for this, e.g. CNN. But we do not want to change our pipeline. Hydra can construct almost any Python object with specified parameters. Let’s describe our net in another way, using YAML. model:
  _target_: torch.nn.Sequential
  layers:
    - _target_: torch.nn.Flatten
    - _target_: torch.nn.Linear
      in_features: 784  # 28x28 images are flattened into 784
      out_features: 128
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.Linear
      in_features: 128
      out_features: 64
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.Linear
      in_features: 64
      out_features: 10 model:
  _target_: torch.nn.Sequential
  layers:
    - _target_: torch.nn.Flatten
    - _target_: torch.nn.Linear
      in_features: 784  # 28x28 images are flattened into 784
      out_features: 128
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.Linear
      in_features: 128
      out_features: 64
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.Linear
      in_features: 64
      out_features: 10 and for CNN _target_: torch.nn.Sequential
  layers:
    - _target_: torch.nn.Conv2d
      in_channels: 1  # MNIST images are grayscale, so 1 input channel
      out_channels: 32  # Number of output channels
      kernel_size: 3  # Size of the convolutional kernel
      stride: 1
      padding: 1
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.MaxPool2d
      kernel_size: 2  # Pooling window size
      stride: 2
    - _target_: torch.nn.Conv2d
      in_channels: 32
      out_channels: 64
      kernel_size: 3
      stride: 1
      padding: 1
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.MaxPool2d
      kernel_size: 2
      stride: 2
    - _target_: torch.nn.Flatten  # Flatten the output for the fully connected layer
    - _target_: torch.nn.Linear
      in_features: 7*7*64  # Size after convolutions and pooling
      out_features: 128
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.Dropout
      p: 0.5  # Dropout rate
    - _target_: torch.nn.Linear
      in_features: 128
      out_features: 10  # Number of classes in MNIST _target_: torch.nn.Sequential
  layers:
    - _target_: torch.nn.Conv2d
      in_channels: 1  # MNIST images are grayscale, so 1 input channel
      out_channels: 32  # Number of output channels
      kernel_size: 3  # Size of the convolutional kernel
      stride: 1
      padding: 1
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.MaxPool2d
      kernel_size: 2  # Pooling window size
      stride: 2
    - _target_: torch.nn.Conv2d
      in_channels: 32
      out_channels: 64
      kernel_size: 3
      stride: 1
      padding: 1
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.MaxPool2d
      kernel_size: 2
      stride: 2
    - _target_: torch.nn.Flatten  # Flatten the output for the fully connected layer
    - _target_: torch.nn.Linear
      in_features: 7*7*64  # Size after convolutions and pooling
      out_features: 128
    - _target_: torch.nn.ReLU
    - _target_: torch.nn.Dropout
      p: 0.5  # Dropout rate
    - _target_: torch.nn.Linear
      in_features: 128
      out_features: 10  # Number of classes in MNIST Pass Function Through Config Sometimes, our parameters can also function. For example, How to Access Config Without Decorating Main - e.g., in ipython Notebook. So, you can use Compose API . Compose API from hydra import compose, initialize from omegaconf import OmegaConf

with initialize(version_base=None, config_path="conf", job_name="run_0001"):
  cfg = compose(config_name="config", overrides= ["parameter=value"])

print (OmegaConf.to_yaml(cfg)) from hydra import compose, initialize from omegaconf import OmegaConf

with initialize(version_base=None, config_path="conf", job_name="run_0001"):
  cfg = compose(config_name="config", overrides= ["parameter=value"])

print (OmegaConf.to_yaml(cfg)) Multi Runs For example, you want to find some good hyperparam and Hydra allows a lot for that. Basically, you can launch several experiments sequentially like. python train_model.py -m "batch_size=16,32,64" python train_model.py -m "batch_size=16,32,64" It launches sequentially, but it is easy to extend to parallel if turned on some launcher like joblib. python train_model.py -m "batch_size=16,32,64 hydra/launcher=joblib" python train_model.py -m "batch_size=16,32,64 hydra/launcher=joblib" Run All Experiments From the Folder for config_file in configs/*.yaml; do python main.py --config-name="${config_file}"; done for config_file in configs/*.yaml; do python main.py --config-name="${config_file}"; done Resources hydra - https://hydra.cc/ good template for DL - https://github.com/ashleve/lightning-hydra-template/tree/main hydra - https://hydra.cc/ https://hydra.cc/ good template for DL - https://github.com/ashleve/lightning-hydra-template/tree/main https://github.com/ashleve/lightning-hydra-template/tree/main

How to Configure Experiments With Hydra - From an ML Enginner Perspective

Authors:
(1) Luyang Zhu, University of Washington and Google Research, and work done while the author was an intern at Google;
(2) Dawei Yang, Google Research;
(3) Tyler Zhu, Google Research;
(4) Fitsum Reda, Google Research;
(5) William Chan, Google Research;
(6) Chitwan Saharia, Google Research;
(7) Mohammad Norouzi, Google Research;
(8) Ira Kemelmacher-Shlizerman, University of Washington and Google Research. Table of Links Abstract and 1. Introduction 2. Related Work 3. Method 3.1. Cascaded Diffusion Models for Try-On 3.2. Parallel-UNet 4. Experiments 5. Summary and Future Work and References Appendix A. Implementation Details B. Additional Results 4. Experiments Datasets. We collect a paired training dataset of 4 Million samples. Each sample consists of two images of the same person wearing the same garment in two different poses. For test, we collect 6K unpaired samples that are never seen during training. Each test sample includes two images of different people wearing different garments under different poses. Both training and test images are cropped and resized to 1024×1024 based on detected 2D human poses. Our dataset includes both men and women captured in different poses, with different body shapes, skin tones, and wearing a wide variety of garments with diverse texture patterns. In addition, we also provide results on the VITONHD dataset [6]. Comparison with other methods. We compare our approach to three methods: TryOnGAN [26], SDAFN [2] and HR-VITON [25]. For fair comparison, we re-train all three methods on our 4 Million samples until convergence. Without re-training, the results of these methods are worse. Released checkpoints of SDAFN and HR-VITON also require layflat garment as input, which is not applicable to our setting. The resolutions of the related methods vary, and we present each method’s results in their native resolution: SDAFN’s at 256×256, TryOnGAN’s at 512×512 and HR-VITON at 1024 × 1024. Quantitative comparison. Table 1 provides comparisons with two metrics. Since our test dataset is unpaired, we compute Frechet Inception Distance (FID) [16] and Kernel Inception Distance (KID) [3] as evaluation metrics. We computed those metrics on both test datasets (our 6K, and VITON-HD) and observe a significantly better performance with our method. User study. We ran two user studies to objectively evaluate our methods compared to others at scale. The results are reported in Table 2. In first study (named “random”), we randomly selected 2804 input pairs out of the 6K test set, ran all four methods on those pairs, and presented to raters. 15 non-expert raters (on crowdsource platform) have been asked to select the best result out of four or choose “hard to tell” option. Our method was selected as best for 92.72% of the inputs. In a second study (named “challenging”), we performed the same setup but chose 2K input pairs (out of 6K) with more challenging poses. The raters selected our method as best for 95.8% of the inputs. Qualitative comparison. In Figures 3 and 4, we provide visual comparisons to all baselines on two test datasets (our 6K, and VITON-HD). Note that many of the chosen input pairs have quite different body poses, shapes and complex garment materials–all limitations of most previous methods–thus we don’t expect them to perform well but present here to show the strength of our method. Specifically, we observe that TryOnGAN struggles to retain the texture pattern of the garments while SDAFN and HRVITON introduce warping artifacts in the try-on results. In contrast, our approach preserves fine details of the source garment and seamlessly blends the garment with the person even if the poses are hard or materials are complex (Fig. 3, row 4). Note also how TryOnDiffusion generates realistic garment wrinkles corresponding to the new body poses (Fig. 3, row 1). We show easier poses in the supplementary (in addition to more results) to provide a fair comparison to other methods. Ablation 1: Cross attention vs concatenation for implicit warping. The implementation of cross attention is detailed in Sec. 3.2. For concatenation, we discard the garmentUNet, directly concatenate the segmented garment Ic to the noisy image zt, and drop cross attention modules in the person-UNet. We apply these changes to each ParallelUNet, and keep the final SR diffusion model same. Fig. 5 shows that cross attention is better at preserving garment details under significant body pose and shape changes. Limitations. First, our method exhibits garment leaking artifacts in case of errors in segmentation maps and pose estimations during preprocessing. Fortunately, those [11, 28] became quite accurate in recent years and this does not happen often. Second, representing identity via clothingagnostic RGB is not ideal, since sometimes it may preserve only part of the identity, e.g., tatooes won’t be visible in this representation, or specific muscle structure. Third, our train and test datasets have mostly clean uniform background so it is unknown how the method performs with more complex backgrounds. Finally, this work focused on upper body clothing and we have not experimented with full body tryon, which is left for future work. Fig. 6 demonstrates failure cases. Finally, Fig. 7 shows TryOnDiffusion results on variety of people and garments. Please refer to supplementary material for more results. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Luyang Zhu, University of Washington and Google Research, and work done while the author was an intern at Google; (2) Dawei Yang, Google Research; (3) Tyler Zhu, Google Research; (4) Fitsum Reda, Google Research; (5) William Chan, Google Research; (6) Chitwan Saharia, Google Research; (7) Mohammad Norouzi, Google Research; (8) Ira Kemelmacher-Shlizerman, University of Washington and Google Research. Authors: Authors: (1) Luyang Zhu, University of Washington and Google Research, and work done while the author was an intern at Google; (2) Dawei Yang, Google Research; (3) Tyler Zhu, Google Research; (4) Fitsum Reda, Google Research; (5) William Chan, Google Research; (6) Chitwan Saharia, Google Research; (7) Mohammad Norouzi, Google Research; (8) Ira Kemelmacher-Shlizerman, University of Washington and Google Research. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. Method 3. Method 3.1. Cascaded Diffusion Models for Try-On 3.1. Cascaded Diffusion Models for Try-On 3.2. Parallel-UNet 3.2. Parallel-UNet 4. Experiments 4. Experiments 5. Summary and Future Work and References 5. Summary and Future Work and References Appendix Appendix A. Implementation Details A. Implementation Details B. Additional Results B. Additional Results 4. Experiments Datasets . We collect a paired training dataset of 4 Million samples. Each sample consists of two images of the same Datasets person wearing the same garment in two different poses. For test, we collect 6K unpaired samples that are never seen during training. Each test sample includes two images of different people wearing different garments under different poses. Both training and test images are cropped and resized to 1024×1024 based on detected 2D human poses. Our dataset includes both men and women captured in different poses, with different body shapes, skin tones, and wearing a wide variety of garments with diverse texture patterns. In addition, we also provide results on the VITONHD dataset [6]. Comparison with other methods. We compare our approach to three methods: TryOnGAN [26], SDAFN [2] and HR-VITON [25]. For fair comparison, we re-train all three methods on our 4 Million samples until convergence. Without re-training, the results of these methods are worse. Released checkpoints of SDAFN and HR-VITON also require layflat garment as input, which is not applicable to our setting. The resolutions of the related methods vary, and we present each method’s results in their native resolution: SDAFN’s at 256×256, TryOnGAN’s at 512×512 and HR-VITON at 1024 × 1024. Comparison with other methods. Quantitative comparison. Table 1 provides comparisons with two metrics. Since our test dataset is unpaired, we Quantitative comparison. compute Frechet Inception Distance (FID) [16] and Kernel Inception Distance (KID) [3] as evaluation metrics. We computed those metrics on both test datasets (our 6K, and VITON-HD) and observe a significantly better performance with our method. User study. We ran two user studies to objectively evaluate our methods compared to others at scale. The results are reported in Table 2. In first study (named “random”), we randomly selected 2804 input pairs out of the 6K test set, ran all four methods on those pairs, and presented to raters. 15 non-expert raters (on crowdsource platform) have been asked to select the best result out of four or choose “hard to tell” option. Our method was selected as best for 92.72% of the inputs. In a second study (named “challenging”), we performed the same setup but chose 2K input pairs (out of 6K) with more challenging poses. The raters selected our method as best for 95.8% of the inputs. User study. Qualitative comparison. In Figures 3 and 4, we provide visual comparisons to all baselines on two test datasets (our 6K, and VITON-HD). Note that many of the chosen input pairs have quite different body poses, shapes and complex garment materials–all limitations of most previous methods–thus we don’t expect them to perform well but present here to show the strength of our method. Specifically, we observe that TryOnGAN struggles to retain the texture pattern of the garments while SDAFN and HRVITON introduce warping artifacts in the try-on results. In contrast, our approach preserves fine details of the source garment and seamlessly blends the garment with the person even if the poses are hard or materials are complex (Fig. 3, row 4). Note also how TryOnDiffusion generates realistic garment wrinkles corresponding to the new body poses (Fig. 3, row 1). We show easier poses in the supplementary (in addition to more results) to provide a fair comparison to other methods. Qualitative comparison. Ablation 1: Cross attention vs concatenation for implicit warping. The implementation of cross attention is detailed in Sec. 3.2. For concatenation, we discard the garmentUNet, directly concatenate the segmented garment Ic to the noisy image zt, and drop cross attention modules in the person-UNet. We apply these changes to each ParallelUNet, and keep the final SR diffusion model same. Fig. 5 shows that cross attention is better at preserving garment details under significant body pose and shape changes. Ablation 1: Cross attention vs concatenation for implicit warping. Limitations . First, our method exhibits garment leaking artifacts in case of errors in segmentation maps and pose estimations during preprocessing. Fortunately, those [11, 28] became quite accurate in recent years and this does not happen often. Second, representing identity via clothingagnostic RGB is not ideal, since sometimes it may preserve only part of the identity, e.g., tatooes won’t be visible in this representation, or specific muscle structure. Third, our train and test datasets have mostly clean uniform background so it is unknown how the method performs with more complex backgrounds. Finally, this work focused on upper body clothing and we have not experimented with full body tryon, which is left for future work. Fig. 6 demonstrates failure cases. Limitations Finally, Fig. 7 shows TryOnDiffusion results on variety of people and garments. Please refer to supplementary material for more results. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

TryOnDiffusion: A Tale of Two UNets: Experiments

Authors:
(1) Chengfeng Shen, School of Mathematical Sciences, Peking University, Beijing;
(2) Yifan Luo, School of Mathematical Sciences, Peking University, Beijing;
(3) Zhennan Zhou, Beijing International Center for Mathematical Research, Peking University. Table of Links Abstract and 1. Introduction 2 Model and 2.1 Optimal Stopping and Obstacle Problem 2.2 Mean Field Games with Optimal Stopping 2.3 Pure Strategy Equilibrium for OSMFG 2.4 Mixed Strategy Equilibrium for OSMFG 3 Algorithm Construction and 3.1 Fictitious Play 3.2 Convergence of Fictitious Play to Mixed Strategy Equilibrium 3.3 Algorithm Based on Fictitious Play 3.4 Numerical Analysis 4 Numerical Experiments and 4.1 A Non-local OSMFG Example 4.2 A Local OSMFG Example 5 Conclusion, Acknowledgement, and References 4 Numerical Experiments In this section, we conduct several numerical experiments to demonstrate the effectiveness of the proposed semi-implicit finite difference algorithm (Algorithm 2). Through these experiments, we examine the convergence properties of our algorithm highlighting the implementation of the fictitious play. We demonstrate that our requirement for δn in (3.3) is a sufficient yet unnecessary condition. However, in certain cases where the pure strategy equilibrium may not exist, the iteration method may not converge if the condition (3.3) is violated. 4.1 A Non-local OSMFG Example Setup. In this example, the state of the representative agent belongs to the domain [0, 1]×R. It dynamic is a Brownian motion, i.e. Intuitively, as the agent aims to minimize expected cost, the running cost encourages the agent to remain in the game when its state is below the average state of other remaining agents, while the stopping cost encourages the agent to continue playing for a longer duration. It is easy to check that the cost functions above are equivalent with In Figure 2, log-log plots of error versus iteration numbers are shown for different learning rates δn. For this example, taking δn = 1 leads to the fastest convergence, although with δn = 1 no fictitious play is implemented. The reason for this is that the convergence requirement for the learning rate δn in Definition 3.1 is only a sufficient condition, rather than a necessary one. For problems with a pure strategy equilibrium, the condition in definition 3.1 can be relaxed and a more aggressive learning rate can be taken to obtain faster convergence. However, we would like to stress that we could not know a priori whether a pure strategy equilibrium exists, but the proposed algorithm, which is based on fictitious play, always produces convergence results. This paper is available on arxiv under CC 4.0 license. Authors: (1) Chengfeng Shen, School of Mathematical Sciences, Peking University, Beijing; (2) Yifan Luo, School of Mathematical Sciences, Peking University, Beijing; (3) Zhennan Zhou, Beijing International Center for Mathematical Research, Peking University. Authors: Authors: (1) Chengfeng Shen, School of Mathematical Sciences, Peking University, Beijing; (2) Yifan Luo, School of Mathematical Sciences, Peking University, Beijing; (3) Zhennan Zhou, Beijing International Center for Mathematical Research, Peking University. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Model and 2.1 Optimal Stopping and Obstacle Problem 2 Model and 2.1 Optimal Stopping and Obstacle Problem 2.2 Mean Field Games with Optimal Stopping 2.2 Mean Field Games with Optimal Stopping 2.3 Pure Strategy Equilibrium for OSMFG 2.3 Pure Strategy Equilibrium for OSMFG 2.4 Mixed Strategy Equilibrium for OSMFG 2.4 Mixed Strategy Equilibrium for OSMFG 3 Algorithm Construction and 3.1 Fictitious Play 3 Algorithm Construction and 3.1 Fictitious Play 3.2 Convergence of Fictitious Play to Mixed Strategy Equilibrium 3.2 Convergence of Fictitious Play to Mixed Strategy Equilibrium 3.3 Algorithm Based on Fictitious Play 3.3 Algorithm Based on Fictitious Play 3.4 Numerical Analysis 3.4 Numerical Analysis 4 Numerical Experiments and 4.1 A Non-local OSMFG Example 4 Numerical Experiments and 4.1 A Non-local OSMFG Example 4.2 A Local OSMFG Example 4.2 A Local OSMFG Example 5 Conclusion, Acknowledgement, and References 5 Conclusion, Acknowledgement, and References 4 Numerical Experiments In this section, we conduct several numerical experiments to demonstrate the effectiveness of the proposed semi-implicit finite difference algorithm (Algorithm 2). Through these experiments, we examine the convergence properties of our algorithm highlighting the implementation of the fictitious play. We demonstrate that our requirement for δ n in (3.3) is a sufficient yet unnecessary condition. However, in certain cases where the pure strategy equilibrium may not exist, the iteration method may not converge if the condition (3.3) is violated. n 4.1 A Non-local OSMFG Example Setup . In this example, the state of the representative agent belongs to the domain [0, 1]×R. It dynamic is a Brownian motion, i.e. Setup Intuitively, as the agent aims to minimize expected cost, the running cost encourages the agent to remain in the game when its state is below the average state of other remaining agents, while the stopping cost encourages the agent to continue playing for a longer duration. It is easy to check that the cost functions above are equivalent with In Figure 2, log-log plots of error versus iteration numbers are shown for different learning rates δn. For this example, taking δn = 1 leads to the fastest convergence, although with δn = 1 no fictitious play is implemented. The reason for this is that the convergence requirement for the learning rate δn in Definition 3.1 is only a sufficient condition, rather than a necessary one. For problems with a pure strategy equilibrium, the condition in definition 3.1 can be relaxed and a more aggressive learning rate can be taken to obtain faster convergence. However, we would like to stress that we could not know a priori whether a pure strategy equilibrium exists, but the proposed algorithm, which is based on fictitious play, always produces convergence results. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Fictitious Play for Mixed Strategy Equilibria in Mean Field Games: Numerical Experiments

Stories, News & Mentions about EXPERIMENT27

Experiment27’s Stories on HackerNoon