The State had given the defence 31 recordings of phone conversations of some of the accused that they had recorded between October 2010 and mid 2012. The defence want access to all the recordings from the 18 months.

Thoshan Panday, co-accused urged to disclose transcripts of ‘tapped’ phone conversations

A sturdy aluminum spool and a digital gear design for $20. This is a great reel to start off hitting lakes and ponds. Daiwa reels are known and respected for being affordable, well made, reliable ...

Best Daiwa Reels of 2023

ThoughtSpot Product Innovation Powers New Wave of Growth With the strategic acquisition of Mode Analytics, ThoughtSpot has further diversified its position as a multi-product company, now serving a wider array of customer personas and delivering a comprehensible platform for all needs. In addition, in the last twelve months, ThoughtSpot:

ThoughtSpot Experiences Historic Year of Growth as Customers Around the World Realize Unprecedented Value from AI-Powered Analytics

On the heels of Underscore VC’s Core Summit, its largest community event of the year, the firm has some funding news on the horizon.

Underscore VC’s third fund in the works for early-stage startups

In addition, in the last twelve months, ThoughtSpot: "At Goldcast, our objective has been to empower our business users to make more data-driven decisions in their day-to-day. ThoughtSpot Sage ...

Stephanie Roulic founded Startup Boston Week as a way bridge connections between people from different industries and companies from all stages.

Boston Speaks Up with Stephanie Roulic, Startup Boston founder and CEO

Lindsay McGuire, who now works from her Fishers home for Boston-based marketing platform Goldcast, was laid off from Formstack in May. (IBJ photo/Eric Learned) After peaking early this year ...

Is the tech recession over? Maybe.

The AGM also saw other appointments to the executive committee, including Tomasz Orkiszewski of Goldcast Academy, Alison Foster of C Foster Plant Hire and Bryan Shilliday of Tam Shilliday Plant ...

New president for the Scottish Plant Owners Association

Panday and his co-accused – former KwaZulu-Natal police commissioner Mmamonnye Ngobeni; South African Police Service Colonel Navin Madhoe and former police captain Aswin Narainpershad – face charges of racketeering,

Durban businessman Thoshan Panday and former top cop involved in R47 million FIFA World Cup tender scandal edge closer to court trial

Goldcast

goldcast

Everything You Need to Know About Zero-Knowledge Proof (ZKP) Technology

How Decentralized Interaction Protocols Work: Analyzing VC, ZKP, and Āut’s Interaction Tree

Authors:
(1) Seokil Ham, KAIST;
(2) Jungwuk Park, KAIST;
(3) Dong-Jun Han, Purdue University;
(4) Jaekyun Moon, KAIST. Table of Links Abstract and 1. Introduction 2. Related Works 3. Proposed NEO-KD Algorithm and 3.1 Problem Setup: Adversarial Training in Multi-Exit Networks 3.2 Algorithm Description 4. Experiments and 4.1 Experimental Setup 4.2. Main Experimental Results 4.3. Ablation Studies and Discussions 5. Conclusion, Acknowledgement and References A. Experiment Details B. Clean Test Accuracy and C. Adversarial Training via Average Attack D. Hyperparameter Tuning E. Discussions on Performance Degradation at Later Exits F. Comparison with Recent Defense Methods for Single-Exit Networks G. Comparison with SKD and ARD and H. Implementations of Stronger Attacker Algorithms 5 Conclusion In this paper, we proposed a new knowledge distillation based adversarial training strategy for robust multi-exit networks. Our solution, NEO-KD, reduces adversarial transferability in the network while guiding the output of the adversarial examples to closely follow the ensemble outputs of the neighbor exits of the clean data, significantly improving the overall adversarial test accuracy. Extensive experimental results on both anytime and budgeted prediction setups using various datasets confirmed the effectiveness of our method, compared to baselines relying on existing adversarial training or knowledge distillation techniques for multi-exit networks. Acknowledgement This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2019R1I1A2A02061135), by the Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD230017TD), and by IITP funds from MSIT of Korea (No. 2020-0-00626). References [1] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII, pages 484–501. Springer, 2020. [2] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017. [3] Sihong Chen, Haojing Shen, Ran Wang, and Xizhao Wang. Towards improving fast adversarial training in multi-exit network. Neural Networks, 150:1–11, 2022. [4] Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning, pages 2196–2205. PMLR, 2020. [5] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020. [6] Junhao Dong, Seyed-Mohsen Moosavi-Dezfooli, Jianhuang Lai, and Xiaohua Xie. The enemy of my enemy is my friend: Exploring inverse adversaries for improving adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24678–24687, 2023. [7] Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. arXiv preprint arXiv:2106.01606, 2021. [8] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3996–4003, 2020. [9] Dong-Jun Han, Jungwuk Park, Seokil Ham, Namjin Lee, and Jaekyun Moon. Improving lowlatency predictions in multi-exit neural networks via block-dependent losses. IEEE Transactions on Neural Networks and Learning Systems, 2023. [10] Mirazul Haque, Anki Chauhan, Cong Liu, and Wei Yang. Ilfo: Adversarial attack on adaptive neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14264–14273, 2020. [11] Sanghyun Hong, Yigitcan Kaya, Ionut,-Vlad Modoranu, and Tudor Dumitras. A panda? no, it’s a sloth: Slowdown attacks on adaptive multi-exit neural network inference. In International Conference on Learning Representations, 2020. [12] Ting-Kuei Hu, Tianlong Chen, Haotao Wang, and Zhangyang Wang. Triple wins: Boosting accuracy, robustness and efficiency together by enabling input-adaptive inference. In International Conference on Learning Representations, 2020. [13] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844, 2017. [14] Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.01950, 2020. [15] Alexandros Kouris, Stylianos I Venieris, Stefanos Laskaridis, and Nicholas Lane. Multi-exit semantic segmentation networks. In European Conference on Computer Vision, pages 330–349. Springer, 2022. [16] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. [17] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. [18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [19] Sungyoon Lee, Hoki Kim, and Jaewook Lee. Graddiv: Adversarial robustness of randomized neural networks via gradient diversity regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. [20] Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang, and Gao Huang. Improved techniques for training adaptive deep networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1891–1900, 2019. [21] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. [22] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pages 4970–4979. PMLR, 2019. [23] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pages 582–597. IEEE, 2016. [24] Mary Phuong and Christoph H Lampert. Distillation-based training for multi-exit architectures. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1355– 1364, 2019. [25] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. [26] Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464–2469. IEEE, 2016. [27] Xinglu Wang and Yingming Li. Harmonized dense knowledge distillation training for multi-exit architectures. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10218–10226, 2021. [28] Qunliang Xing, Mai Xu, Tianyi Li, and Zhenyu Guan. Early exit or not: Resource-efficient blind quality enhancement for compressed images. In European Conference on Computer Vision, pages 275–292. Springer, 2020. [29] Huanrui Yang, Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. Dverge: diversifying vulnerabilities for enhanced robust generation of ensembles. Advances in Neural Information Processing Systems, 33:5505–5515, 2020. [30] Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Pan Zhou, Benjamin Rubinstein, Ce Zhang, and Bo Li. Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems, 34, 2021. [31] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482. PMLR, 2019. [32] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330–18341, 2020. [33] Jianing Zhu, Jiangchao Yao, Bo Han, Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. arXiv preprint arXiv:2106.04928, 2021. [34] Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang. Revisiting adversarial robustness distillation: Robust soft labels make student better. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16443–16452, 2021. This paper is available on arxiv under CC 4.0 license. Authors: (1) Seokil Ham, KAIST; (2) Jungwuk Park, KAIST; (3) Dong-Jun Han, Purdue University; (4) Jaekyun Moon, KAIST. Authors: Authors: (1) Seokil Ham, KAIST; (2) Jungwuk Park, KAIST; (3) Dong-Jun Han, Purdue University; (4) Jaekyun Moon, KAIST. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Works 2. Related Works 3. Proposed NEO-KD Algorithm and 3.1 Problem Setup: Adversarial Training in Multi-Exit Networks 3. Proposed NEO-KD Algorithm and 3.1 Problem Setup: Adversarial Training in Multi-Exit Networks 3.2 Algorithm Description 3.2 Algorithm Description 4. Experiments and 4.1 Experimental Setup 4. Experiments and 4.1 Experimental Setup 4.2. Main Experimental Results 4.2. Main Experimental Results 4.3. Ablation Studies and Discussions 4.3. Ablation Studies and Discussions 5. Conclusion, Acknowledgement and References 5. Conclusion, Acknowledgement and References A. Experiment Details A. Experiment Details B. Clean Test Accuracy and C. Adversarial Training via Average Attack B. Clean Test Accuracy and C. Adversarial Training via Average Attack D. Hyperparameter Tuning D. Hyperparameter Tuning E. Discussions on Performance Degradation at Later Exits E. Discussions on Performance Degradation at Later Exits F. Comparison with Recent Defense Methods for Single-Exit Networks F. Comparison with Recent Defense Methods for Single-Exit Networks G. Comparison with SKD and ARD and H. Implementations of Stronger Attacker Algorithms G. Comparison with SKD and ARD and H. Implementations of Stronger Attacker Algorithms 5 Conclusion In this paper, we proposed a new knowledge distillation based adversarial training strategy for robust multi-exit networks. Our solution, NEO-KD, reduces adversarial transferability in the network while guiding the output of the adversarial examples to closely follow the ensemble outputs of the neighbor exits of the clean data, significantly improving the overall adversarial test accuracy. Extensive experimental results on both anytime and budgeted prediction setups using various datasets confirmed the effectiveness of our method, compared to baselines relying on existing adversarial training or knowledge distillation techniques for multi-exit networks. Acknowledgement This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2019R1I1A2A02061135), by the Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD230017TD), and by IITP funds from MSIT of Korea (No. 2020-0-00626). References [1] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII, pages 484–501. Springer, 2020. [2] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017. [3] Sihong Chen, Haojing Shen, Ran Wang, and Xizhao Wang. Towards improving fast adversarial training in multi-exit network. Neural Networks, 150:1–11, 2022. [4] Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning, pages 2196–2205. PMLR, 2020. [5] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020. [6] Junhao Dong, Seyed-Mohsen Moosavi-Dezfooli, Jianhuang Lai, and Xiaohua Xie. The enemy of my enemy is my friend: Exploring inverse adversaries for improving adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24678–24687, 2023. [7] Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. arXiv preprint arXiv:2106.01606, 2021. [8] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3996–4003, 2020. [9] Dong-Jun Han, Jungwuk Park, Seokil Ham, Namjin Lee, and Jaekyun Moon. Improving lowlatency predictions in multi-exit neural networks via block-dependent losses. IEEE Transactions on Neural Networks and Learning Systems, 2023. [10] Mirazul Haque, Anki Chauhan, Cong Liu, and Wei Yang. Ilfo: Adversarial attack on adaptive neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14264–14273, 2020. [11] Sanghyun Hong, Yigitcan Kaya, Ionut,-Vlad Modoranu, and Tudor Dumitras. A panda? no, it’s a sloth: Slowdown attacks on adaptive multi-exit neural network inference. In International Conference on Learning Representations, 2020. [12] Ting-Kuei Hu, Tianlong Chen, Haotao Wang, and Zhangyang Wang. Triple wins: Boosting accuracy, robustness and efficiency together by enabling input-adaptive inference. In International Conference on Learning Representations, 2020. [13] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844, 2017. [14] Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.01950, 2020. [15] Alexandros Kouris, Stylianos I Venieris, Stefanos Laskaridis, and Nicholas Lane. Multi-exit semantic segmentation networks. In European Conference on Computer Vision, pages 330–349. Springer, 2022. [16] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. [17] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. [18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [19] Sungyoon Lee, Hoki Kim, and Jaewook Lee. Graddiv: Adversarial robustness of randomized neural networks via gradient diversity regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. [20] Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang, and Gao Huang. Improved techniques for training adaptive deep networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1891–1900, 2019. [21] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. [22] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pages 4970–4979. PMLR, 2019. [23] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pages 582–597. IEEE, 2016. [24] Mary Phuong and Christoph H Lampert. Distillation-based training for multi-exit architectures. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1355– 1364, 2019. [25] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. [26] Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464–2469. IEEE, 2016. [27] Xinglu Wang and Yingming Li. Harmonized dense knowledge distillation training for multi-exit architectures. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10218–10226, 2021. [28] Qunliang Xing, Mai Xu, Tianyi Li, and Zhenyu Guan. Early exit or not: Resource-efficient blind quality enhancement for compressed images. In European Conference on Computer Vision, pages 275–292. Springer, 2020. [29] Huanrui Yang, Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. Dverge: diversifying vulnerabilities for enhanced robust generation of ensembles. Advances in Neural Information Processing Systems, 33:5505–5515, 2020. [30] Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Pan Zhou, Benjamin Rubinstein, Ce Zhang, and Bo Li. Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems, 34, 2021. [31] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482. PMLR, 2019. [32] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330–18341, 2020. [33] Jianing Zhu, Jiangchao Yao, Bo Han, Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. arXiv preprint arXiv:2106.04928, 2021. [34] Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang. Revisiting adversarial robustness distillation: Robust soft labels make student better. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16443–16452, 2021. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

How NEO-KD Reduces Adversarial Transferability and Improves Accuracy

Optimizing Prompts with LLMs: Key Findings and Future Directions

Authors:
(1) Taha Laroussi, Laboratoire d’Hydrodynamique (LadHyX), CNRS, Ecole Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France;
(2) Mojtaba Jarrahi, Universit´e Paris-Saclay, CNRS, FAST, 91405 Orsay, France;
(3) Gabriel Amselem, Laboratoire d’Hydrodynamique (LadHyX), CNRS, Ecole Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France. Table of Links Abstract and Introduction Methods Results Model Discussion, Acknowledgements and References Supplementary Material V. DISCUSSION We have shown experimentally that the change in phototactic behavior of C. reinhardtii can be due to its recent history of illumination. When exposed to two closely spaced, identical stimuli, the algae act as if they were time-integrating the stimuli. This is visible when the stimulus is close to that delimiting positive and negative phototaxis. Then, the algae show two different behaviors in response to the two consecutive stimuli: first a backand-forth response, then a negative phototaxis. When the stimuli are sufficiently spaced apart, the algae have had time to relax to a basal state, and show the same qualitative response to the two stimuli, see Fig. 6. In our experiments, we find this time of complete relaxation to be approximately equal to 30 minutes. A simplified model of phototaxis explains this memory effect by the introduction of a time-scale of relaxation in the inner biochemistry of the cell. When the time-scale of relaxation is larger than the time between two consecutive stimuli, the alga integrates the signal and can display different behaviors in response to the two identical stimuli, see Fig. 8. The time needed to return to the basal state should be of order 3τ , which underlies our choice of τ = 10 min for the relaxation time constant. Our experimental data are consistent with an inner relaxation time τ in the range ≈ 5 to 20 minutes. What could be the biochemical species corresponding to the intracellular species S introduced in the model? It is very likely that our simplified model compounds the dynamics of multiple biochemical species, activated sequentially in a signalling cascade. Still, it is known that the photoreceptors for phototaxis in C. reinhardtii are channelrhodopsins, that become phosphorylated within milliseconds after the application of a light stimulus. A high level of phosphorylated channelrhodopsin-1 (ChR1) in the cell correlates with a change from positive to negative phototaxis [11], much like a high level of activated S corresponds to negative phototaxis in our model. Moreover, the time scale of dephosphorylation of ChR1 in C. reinhardtii has been shown to be of the order of ≈ 5 to 15 minutes [11], which is compatible with our value of τ = 10 min. A possibility is therefore to identify S with ChR1. A more physical model of phototaxis was recently introduced by Leptos et al. to describe the response of C. reinhardtii to light [22]. In their model, Leptos et al. take into account the rotation of the microalga around its axis while swimming, and the resulting oscillating light intensity seen by the eyespot. The consecutive step-up and step-down stimuli sensed by the eyespot during one body rotation lead to a realignment of the trajectory towards the light source. Our model averages out the body rotation of the microalgae and describes the behavior of the microorganism at a more coarse-grained time-scale. Another difference is that the model of Leptos et al. does not take into account the fact that phototaxis changes sign when the light intensity increases. This change of phototactic behavior requires the introduction of a threshold value in the model, below (resp. above) which phototaxis is positive (resp. negative). Such a threshold is incorporated in our simplified model. Note that the threshold in the model, introduced as a simple means to obtain a change in behavior between positive and negative phototaxis at different light intensities, leads to the back-and-forth behavior at intermediate light intensities. The existence of this threshold also enables to explain why the time scale of negative phototaxis is longer than that of positive phototaxis. Negative phototaxis occurs when the amount of active species s ⋆ overcomes a threshold sT ; this takes time, during which the alga exhibits positive phototaxis. Only when s ⋆ > sT do the algae show negative phototaxis, and accumulate at one boundary of the well. The small initial positive phototactic response predicted by the model is in agreement with experimental data. The model also predicts that it is possible to switch the phototactic behavior of algae from positive phototaxis to back-and-forth to negative phototaxis, using three consecutive identical stimuli, by adjusting the waiting times between the stimuli and choosing carefully the stimulus intensity, see Fig. 8d. While we were not able to find the experimental conditions corresponding to this theoretical prediction, we were able to experimentally observe the switch from back-and-forth to negative phototaxis, and from positive phototaxis to back-and-forth in a couple experiments, see Supp. Fig. 5. While the model successfully reproduces the phototactic behavior and the time scales at play, it fails to reproduce two experimental observations. First, in a couple experiments, multiple consecutive back-and-forth of the center of mass were observed at intermediate intensities, see Supp. Fig. 6. Second, in experiments with consecutive stimuli at intermediate light intensities, algae in the third experiment (after a waiting time of 30 minutes) display positive phototaxis instead of the expected backand-forth behavior observed during the first experiment, see Fig. 6b and Supp. Fig. 5. Both behaviors can be obtained in the model by supposing that the threshold sT increases with the time spent under a light stimulus, and so depends on the history of the microorganism. Such an increasing threshold would also enable to recover the traditional adaptative behavior at longer time scales [15]. This implies having a third equation describing the timeevolution of sT , thus introducing another time constant into the problem. In the lab, the memory effects reported here imply that it is not possible to repeat the same phototaxis experiments on a given batch of cells and expect similar outcomes. Such an effect was already known in the case of long-term adaptation [15]. The data reported here show that even in the case of short-term experiments, the behavior is not necessarily reproducible. To study phototaxis quantitatively, experimental repeats should be carried out with different batches of naive cells. Having identified this short time-scale, our experiments pave the way for the study of phototaxis in the presence of timevarying light stimuli, much as the algae are subject to in their natural environment. There, light is constantly fluctuating. The short-term memory could then be a beneficial way to integrate a light signal randomly interrupted by the shadow of other motile organisms or objects. ACKNOWLEDGMENTS We thank David Gonzalez-Rodriguez for insightful comments on an early version of the model, Maxime Simon for preliminary experiments, Salome Guttierez Ramos for help with microfabrication, and Caroline Frot for precious technical support. This work was supported by “Investissements d’Avenir” LabEx PALM (ANR-10- LABX-0039-PALM) and by PEPS “M´ecanique du futur”. References [1] G. J´ekely, Philosophical Transactions of the Royal Society B: Biological Sciences 364, 2795 (2009). [2] K. W. Foster and R. D. Smyth, Microbiological reviews 44, 572 (1980). [3] P. Hegemann and P. Berthold, in The Chlamydomonas Sourcebook (Elsevier, 2009) pp. 395–429. [4] U. R¨uffer and W. Nultsch, Cell motility and the cytoskeleton 18, 269 (1991). [5] M. E. H. Feinleib and G. M. Curry, Physiologia Plantarum 25, 346 (1971). [6] W. Nultsch, J. Pfau, and R. Dolle, Archives of microbiology 144, 393 (1986). [7] N. Morel-Laurens, Photochemistry and Photobiology 45, 119 (1987). [8] R. Dolle, J. Pfau, and W. Nultsch, Journal of plant physiology 126, 467 (1987). [9] T. Takahashi and M. Watanabe, FEBS Letters 336, 516 (1993). [10] K.-i. Wakabayashi, Y. Misawa, S. Mochiji, and R. Kamiya, Proceedings of the National Academy of Sciences 108, 11280 (2011). [11] M. B¨ohm, D. Boness, E. Fantisch, H. Erhard, J. Frauenholz, Z. Kowalzyk, N. Marcinkowski, S. Kateriya, P. Hegemann, and G. Kreimer, The Plant Cell 31, 886 (2019). [12] R. M. Macnab and D. E. Koshland Jr, Proceedings of the National Academy of Sciences 69, 2509 (1972). [13] H. C. Berg and P. Tedesco, Proceedings of the National Academy of Sciences 72, 3235 (1975). [14] P. Van Haastert, The Journal of cell biology 96, 1559 (1983). [15] A. Mayer, Nature 217, 875 (1968). [16] E. Harris, D. Stern, and G. Witman, The chlamydomonas sourcebook, vol. 1 san diego (2009). [17] Y. Xia and G. M. Whitesides, Annual review of materials science 28, 153 (1998). [18] N. Ueki, T. Ide, S. Mochiji, Y. Kobayashi, R. Tokutsu, N. Ohnishi, K. Yamaguchi, S. Shigenobu, K. Tanaka, J. Minagawa, et al., Proceedings of the National Academy of Sciences 113, 5299 (2016). [19] A. Ramamonjy, J. Dervaux, and P. Brunet, Physical Review Letters 128, 258101 (2022). [20] C. T. Kreis, M. Le Blay, C. Linne, M. M. Makowski, and O. B¨aumchen, Nature Physics 14, 45 (2018). [21] S. Till, F. Ebmeier, A. A. Fragkopoulos, M. G. Mazza, and O. B¨aumchen, Physical Review Research 4, L042046 (2022). [22] K. C. Leptos, M. Chioccioli, S. Furlan, A. I. Pesci, and R. E. Goldstein, Physical Review E 107 (2023). This paper is available on arxiv under CC 4.0 license. Authors: (1) Taha Laroussi, Laboratoire d’Hydrodynamique (LadHyX), CNRS, Ecole Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France; (2) Mojtaba Jarrahi, Universit´e Paris-Saclay, CNRS, FAST, 91405 Orsay, France; (3) Gabriel Amselem, Laboratoire d’Hydrodynamique (LadHyX), CNRS, Ecole Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France. Authors: Authors: (1) Taha Laroussi, Laboratoire d’Hydrodynamique (LadHyX), CNRS, Ecole Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France; (2) Mojtaba Jarrahi, Universit´e Paris-Saclay, CNRS, FAST, 91405 Orsay, France; (3) Gabriel Amselem, Laboratoire d’Hydrodynamique (LadHyX), CNRS, Ecole Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France. Table of Links Abstract and Introduction Abstract and Introduction Methods Methods Results Results Model Model Discussion, Acknowledgements and References Discussion, Acknowledgements and References Supplementary Material Supplementary Material V. DISCUSSION We have shown experimentally that the change in phototactic behavior of C. reinhardtii can be due to its recent history of illumination. When exposed to two closely spaced, identical stimuli, the algae act as if they were time-integrating the stimuli. This is visible when the stimulus is close to that delimiting positive and negative phototaxis. Then, the algae show two different behaviors in response to the two consecutive stimuli: first a backand-forth response, then a negative phototaxis. When the stimuli are sufficiently spaced apart, the algae have had time to relax to a basal state, and show the same qualitative response to the two stimuli, see Fig. 6. In our experiments, we find this time of complete relaxation to be approximately equal to 30 minutes. A simplified model of phototaxis explains this memory effect by the introduction of a time-scale of relaxation in the inner biochemistry of the cell. When the time-scale of relaxation is larger than the time between two consecutive stimuli, the alga integrates the signal and can display different behaviors in response to the two identical stimuli, see Fig. 8. The time needed to return to the basal state should be of order 3τ , which underlies our choice of τ = 10 min for the relaxation time constant. Our experimental data are consistent with an inner relaxation time τ in the range ≈ 5 to 20 minutes. What could be the biochemical species corresponding to the intracellular species S introduced in the model? It is very likely that our simplified model compounds the dynamics of multiple biochemical species, activated sequentially in a signalling cascade. Still, it is known that the photoreceptors for phototaxis in C. reinhardtii are channelrhodopsins, that become phosphorylated within milliseconds after the application of a light stimulus. A high level of phosphorylated channelrhodopsin-1 (ChR1) in the cell correlates with a change from positive to negative phototaxis [11], much like a high level of activated S corresponds to negative phototaxis in our model. Moreover, the time scale of dephosphorylation of ChR1 in C. reinhardtii has been shown to be of the order of ≈ 5 to 15 minutes [11], which is compatible with our value of τ = 10 min. A possibility is therefore to identify S with ChR1. A more physical model of phototaxis was recently introduced by Leptos et al. to describe the response of C. reinhardtii to light [22]. In their model, Leptos et al. take into account the rotation of the microalga around its axis while swimming, and the resulting oscillating light intensity seen by the eyespot. The consecutive step-up and step-down stimuli sensed by the eyespot during one body rotation lead to a realignment of the trajectory towards the light source. Our model averages out the body rotation of the microalgae and describes the behavior of the microorganism at a more coarse-grained time-scale. Another difference is that the model of Leptos et al. does not take into account the fact that phototaxis changes sign when the light intensity increases. This change of phototactic behavior requires the introduction of a threshold value in the model, below (resp. above) which phototaxis is positive (resp. negative). Such a threshold is incorporated in our simplified model. Note that the threshold in the model, introduced as a simple means to obtain a change in behavior between positive and negative phototaxis at different light intensities, leads to the back-and-forth behavior at intermediate light intensities. The existence of this threshold also enables to explain why the time scale of negative phototaxis is longer than that of positive phototaxis. Negative phototaxis occurs when the amount of active species s ⋆ overcomes a threshold sT ; this takes time, during which the alga exhibits positive phototaxis. Only when s ⋆ > sT do the algae show negative phototaxis, and accumulate at one boundary of the well. The small initial positive phototactic response predicted by the model is in agreement with experimental data. The model also predicts that it is possible to switch the phototactic behavior of algae from positive phototaxis to back-and-forth to negative phototaxis, using three consecutive identical stimuli, by adjusting the waiting times between the stimuli and choosing carefully the stimulus intensity, see Fig. 8d. While we were not able to find the experimental conditions corresponding to this theoretical prediction, we were able to experimentally observe the switch from back-and-forth to negative phototaxis, and from positive phototaxis to back-and-forth in a couple experiments, see Supp. Fig. 5. While the model successfully reproduces the phototactic behavior and the time scales at play, it fails to reproduce two experimental observations. First, in a couple experiments, multiple consecutive back-and-forth of the center of mass were observed at intermediate intensities, see Supp. Fig. 6. Second, in experiments with consecutive stimuli at intermediate light intensities, algae in the third experiment (after a waiting time of 30 minutes) display positive phototaxis instead of the expected backand-forth behavior observed during the first experiment, see Fig. 6b and Supp. Fig. 5. Both behaviors can be obtained in the model by supposing that the threshold sT increases with the time spent under a light stimulus, and so depends on the history of the microorganism. Such an increasing threshold would also enable to recover the traditional adaptative behavior at longer time scales [15]. This implies having a third equation describing the timeevolution of sT , thus introducing another time constant into the problem. In the lab, the memory effects reported here imply that it is not possible to repeat the same phototaxis experiments on a given batch of cells and expect similar outcomes. Such an effect was already known in the case of long-term adaptation [15]. The data reported here show that even in the case of short-term experiments, the behavior is not necessarily reproducible. To study phototaxis quantitatively, experimental repeats should be carried out with different batches of naive cells. Having identified this short time-scale, our experiments pave the way for the study of phototaxis in the presence of timevarying light stimuli, much as the algae are subject to in their natural environment. There, light is constantly fluctuating. The short-term memory could then be a beneficial way to integrate a light signal randomly interrupted by the shadow of other motile organisms or objects. ACKNOWLEDGMENTS We thank David Gonzalez-Rodriguez for insightful comments on an early version of the model, Maxime Simon for preliminary experiments, Salome Guttierez Ramos for help with microfabrication, and Caroline Frot for precious technical support. This work was supported by “Investissements d’Avenir” LabEx PALM (ANR-10- LABX-0039-PALM) and by PEPS “M´ecanique du futur”. References [1] G. J´ekely, Philosophical Transactions of the Royal Society B: Biological Sciences 364, 2795 (2009). [2] K. W. Foster and R. D. Smyth, Microbiological reviews 44, 572 (1980). [3] P. Hegemann and P. Berthold, in The Chlamydomonas Sourcebook (Elsevier, 2009) pp. 395–429. [4] U. R¨uffer and W. Nultsch, Cell motility and the cytoskeleton 18, 269 (1991). [5] M. E. H. Feinleib and G. M. Curry, Physiologia Plantarum 25, 346 (1971). [6] W. Nultsch, J. Pfau, and R. Dolle, Archives of microbiology 144, 393 (1986). [7] N. Morel-Laurens, Photochemistry and Photobiology 45, 119 (1987). [8] R. Dolle, J. Pfau, and W. Nultsch, Journal of plant physiology 126, 467 (1987). [9] T. Takahashi and M. Watanabe, FEBS Letters 336, 516 (1993). [10] K.-i. Wakabayashi, Y. Misawa, S. Mochiji, and R. Kamiya, Proceedings of the National Academy of Sciences 108, 11280 (2011). [11] M. B¨ohm, D. Boness, E. Fantisch, H. Erhard, J. Frauenholz, Z. Kowalzyk, N. Marcinkowski, S. Kateriya, P. Hegemann, and G. Kreimer, The Plant Cell 31, 886 (2019). [12] R. M. Macnab and D. E. Koshland Jr, Proceedings of the National Academy of Sciences 69, 2509 (1972). [13] H. C. Berg and P. Tedesco, Proceedings of the National Academy of Sciences 72, 3235 (1975). [14] P. Van Haastert, The Journal of cell biology 96, 1559 (1983). [15] A. Mayer, Nature 217, 875 (1968). [16] E. Harris, D. Stern, and G. Witman, The chlamydomonas sourcebook, vol. 1 san diego (2009). [17] Y. Xia and G. M. Whitesides, Annual review of materials science 28, 153 (1998). [18] N. Ueki, T. Ide, S. Mochiji, Y. Kobayashi, R. Tokutsu, N. Ohnishi, K. Yamaguchi, S. Shigenobu, K. Tanaka, J. Minagawa, et al., Proceedings of the National Academy of Sciences 113, 5299 (2016). [19] A. Ramamonjy, J. Dervaux, and P. Brunet, Physical Review Letters 128, 258101 (2022). [20] C. T. Kreis, M. Le Blay, C. Linne, M. M. Makowski, and O. B¨aumchen, Nature Physics 14, 45 (2018). [21] S. Till, F. Ebmeier, A. A. Fragkopoulos, M. G. Mazza, and O. B¨aumchen, Physical Review Research 4, L042046 (2022). [22] K. C. Leptos, M. Chioccioli, S. Furlan, A. I. Pesci, and R. E. Goldstein, Physical Review E 107 (2023). This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Short-term Memory Effects in the Phototactic Behavior of Microalgae: Discussion and References

FaceStudio: Put Your Face Everywhere in Seconds: Conclusion and References

Load Balancing For High Performance Computing
Using Quantum Annealing: Conclusions and Outlook

Exploring Drone Dynamics: Stability and Force Generation in Multimedia Design

Authors:
(1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.);
(2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Related Work Preliminaries Simplifying Transformer Blocks Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts C Additional Experiments D Implementation Details D IMPLEMENTATION DETAILS In this section we add remaining implementation details that were not discussed in the main paper. We break down our implementation details into two subsections, one for the next-token prediction task on CodeParrot and one for our Crammed BERT (Geiping & Goldstein, 2023) masked language modelling experiments pretrained on the Pile dataset (Gao et al., 2020) and fine-tuned to downstream GLUE benchmark (Wang et al., 2019). To avoid repetition, any details that are mentioned in one subsection but not the other are shared between both subsections. All runtime results on CodeParrot were run on a single A5000 GPU. D.1 CODEPARROT NEXT-TOKEN PREDICTION As mentioned, much of our setup is derived from https://huggingface.co/learn/ nlp-course/chapter7/6. Model The model is a 18-layer GPT-style auto-regressive decoder-only transformer. We use width d = 768, and H = 12 heads in multi-head attention. We remove dropout entirely as our focus is on training speed, and we are always in a single-epoch regime so regularisation hurts training speed. The MLP uses ReLU activation unless stated otherwise, and we use MLP hidden dimension 3072 = 4d. The only exception to this is in Fig. 6, where we reduce the MLP hidden dimension to 1536 = 2d to account for the increased memory requirements of larger depths. For any of our simplified model we initialise βFF = 0.1 in Eq. (1) to account for the lack of skip, apart from the 18-layer models in Fig. 6, where βFF = 0.2 due to the narrower width. We use RMSNorm (Zhang & Sennrich, 2019) where applicable with epsilon 1e − 8, and add a final normalisation after the decoder. Sinusoidal positional encodings are used and added at the embedding level. Training We use AdamW optimiser (Loshchilov & Hutter, 2017) with weight decay 0.1 which we tuned on a small grid, and found to work well for both baselines and our models. We do not apply weight decay to any scalar gain parameter. We clip gradients with with clipping parameter 1, and use epsilon of 1e − 8 and default betas of (0.9, 0.999) in AdamW. As discussed, we use a linear decay rate with 5% of all steps used for linear warmup. The optimal learning rate was tuned in all cases, and for our best (SAS and SAS-P) models, was found to be 1e−3, which exactly matched that of the default Pre-LN. This held true also when we scaled to 72 layers. V-SkipInit needed a lower learning rate for the depth scaling experiments (3e−4 and 1e−4 for depths 18 and 72 respectively). We use batch size of 128 with microbatches of size 32. Dataset The Codeparrot dataset is a large corpus of 20 million python files from GitHub. We take the dataset, pre-processing and tokeniser from https://huggingface.co/learn/ nlp-course/chapter7/6. We use sequence length T = 128 throughout, and our tokeniser has 50K vocabulary size. Our base experiments train for around 43K steps on batch size 128 and sequence length 128 which is around 700M tokens. In Fig. 8 we scale this to 2B tokens. Task The model is trained on next-token prediction using cross-entropy loss. D.2 BERT ENCODER-ONLY As discussed in Sec. 5, we inherit much of our hyperparameters from the Cramming setup of Geiping & Goldstein (2023), and also base our implementation from their excellent codebase.[9] We highlight important implementation details here. Model We use a 16-layer encoder only model, with width d = 768 and 12 heads. We use MLP width 3072 = 4d, but now we use GLU (Dauphin et al., 2017) with GeLU activation, which essentially halves the hidden dimension. We use LayerNorm (Ba et al., 2016) for normalisation where applicable with epsilon 1e − 12 as taken from Geiping & Goldstein (2023); we always use a final LN after all the layers. Again, we remove all dropout, and use a sequence length of 128. We found our simplified skipless models prefered smaller MLP block scales and initialise βFF = 0.05. Parameter Initialisation The initialisations are identical to those in Codeparrot, and are detailed above. Datasets Like Geiping & Goldstein (2023), we train on the Pile dataset (Gao et al., 2020), with a WordPiece tokeniser of vocabulary size 32768, and a sequence length of 128. Our fastest runs took around 600K steps with microbatch size 64 in 24 hours, which corresponds to around 5B tokens. Training We again trained with AdamW optimiser, with weight decay 0.1. AdamW had hyparameters [β1, β2] = [0.9, 0.98], and epsilon 1e − 12. We used a microbatch of 64 (to fit on a RTX-2080Ti), and scale the batch size to reach 8192 linearly after 60% of total training like in Geiping & Goldstein (2023). We use the same aggressive learning rate as Geiping & Goldstein (2023), which increase linearly to max value after 75% of all training steps, before linear decay, and tune the maximum learning rate to 3e − 3 for our SAS and SAS-P models. This was slightly too large for the SAS-P model without normalisation, so we reduce to 2e − 3. We inherit the clipping parameter of 0.5 from Geiping & Goldstein (2023). Fine-tuning We followed the same protocol as Geiping & Goldstein (2023), In particular, we finetune for 5 epochs with fixed hyperparameters across tasks. We found dropout to be important for good downstream performane (unlike during pre-training), and set dropout probability p = 0.1. We use batch size 32, with a maximum learning of 1.5e − 4. We keep other hyperparameters, e.g. the choice of cosine decay and AdamW epsilon 1e − 6, like from Geiping & Goldstein (2023). Task The model is trained on the masked language modelling task with masking probability 0.25, as in Geiping & Goldstein (2023). This paper is available on arxiv under CC 4.0 license. [9] https://github.com/JonasGeiping/cramming Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Authors: Authors: (1) Bobby He, Department of Computer Science, ETH Zurich (Correspondence to: bobby.he@inf.ethz.ch.); (2) Thomas Hofmann, Department of Computer Science, ETH Zurich. Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Preliminaries Preliminaries Simplifying Transformer Blocks Simplifying Transformer Blocks Further Experimental Analysis Further Experimental Analysis Discussion, Reproducibility Statement, Acknowledgements and References Discussion, Reproducibility Statement, Acknowledgements and References A Duality Between Downweighted Residual and Restricting Updates In Linear Layers A Duality Between Downweighted Residual and Restricting Updates In Linear Layers B Block Layouts B Block Layouts C Additional Experiments C Additional Experiments D Implementation Details D Implementation Details D IMPLEMENTATION DETAILS In this section we add remaining implementation details that were not discussed in the main paper. We break down our implementation details into two subsections, one for the next-token prediction task on CodeParrot and one for our Crammed BERT (Geiping & Goldstein, 2023) masked language modelling experiments pretrained on the Pile dataset (Gao et al., 2020) and fine-tuned to downstream GLUE benchmark (Wang et al., 2019). To avoid repetition, any details that are mentioned in one subsection but not the other are shared between both subsections. All runtime results on CodeParrot were run on a single A5000 GPU. D.1 CODEPARROT NEXT-TOKEN PREDICTION As mentioned, much of our setup is derived from https://huggingface.co/learn/ nlp-course/chapter7/6. Model The model is a 18-layer GPT-style auto-regressive decoder-only transformer. We use width d = 768, and H = 12 heads in multi-head attention. We remove dropout entirely as our focus is on training speed, and we are always in a single-epoch regime so regularisation hurts training speed. The MLP uses ReLU activation unless stated otherwise, and we use MLP hidden dimension 3072 = 4d. The only exception to this is in Fig. 6, where we reduce the MLP hidden dimension to 1536 = 2d to account for the increased memory requirements of larger depths. Model For any of our simplified model we initialise βFF = 0.1 in Eq. (1) to account for the lack of skip, apart from the 18-layer models in Fig. 6, where βFF = 0.2 due to the narrower width. We use RMSNorm (Zhang & Sennrich, 2019) where applicable with epsilon 1e − 8, and add a final normalisation after the decoder. Sinusoidal positional encodings are used and added at the embedding level. Training We use AdamW optimiser (Loshchilov & Hutter, 2017) with weight decay 0.1 which we tuned on a small grid, and found to work well for both baselines and our models. We do not apply weight decay to any scalar gain parameter. We clip gradients with with clipping parameter 1, and use epsilon of 1e − 8 and default betas of (0.9, 0.999) in AdamW. As discussed, we use a linear decay rate with 5% of all steps used for linear warmup. The optimal learning rate was tuned in all cases, and for our best (SAS and SAS-P) models, was found to be 1e−3, which exactly matched that of the default Pre-LN. This held true also when we scaled to 72 layers. V-SkipInit needed a lower learning rate for the depth scaling experiments (3e−4 and 1e−4 for depths 18 and 72 respectively). We use batch size of 128 with microbatches of size 32. Dataset The Codeparrot dataset is a large corpus of 20 million python files from GitHub. We take the dataset, pre-processing and tokeniser from https://huggingface.co/learn/ nlp-course/chapter7/6. We use sequence length T = 128 throughout, and our tokeniser has 50K vocabulary size. Our base experiments train for around 43K steps on batch size 128 and sequence length 128 which is around 700M tokens. In Fig. 8 we scale this to 2B tokens. Dataset Task The model is trained on next-token prediction using cross-entropy loss. Task D.2 BERT ENCODER-ONLY As discussed in Sec. 5, we inherit much of our hyperparameters from the Cramming setup of Geiping & Goldstein (2023), and also base our implementation from their excellent codebase.[9] We highlight important implementation details here. Model We use a 16-layer encoder only model, with width d = 768 and 12 heads. We use MLP width 3072 = 4d, but now we use GLU (Dauphin et al., 2017) with GeLU activation, which essentially halves the hidden dimension. We use LayerNorm (Ba et al., 2016) for normalisation where applicable with epsilon 1e − 12 as taken from Geiping & Goldstein (2023); we always use a final LN after all the layers. Again, we remove all dropout, and use a sequence length of 128. We found our simplified skipless models prefered smaller MLP block scales and initialise βFF = 0.05. Model Parameter Initialisation The initialisations are identical to those in Codeparrot, and are detailed above. Parameter Initialisation Datasets Like Geiping & Goldstein (2023), we train on the Pile dataset (Gao et al., 2020), with a WordPiece tokeniser of vocabulary size 32768, and a sequence length of 128. Our fastest runs took around 600K steps with microbatch size 64 in 24 hours, which corresponds to around 5B tokens. Datasets Training We again trained with AdamW optimiser, with weight decay 0.1. AdamW had hyparameters [β1, β2] = [0.9, 0.98], and epsilon 1e − 12. We used a microbatch of 64 (to fit on a RTX-2080Ti), and scale the batch size to reach 8192 linearly after 60% of total training like in Geiping & Goldstein (2023). We use the same aggressive learning rate as Geiping & Goldstein (2023), which increase linearly to max value after 75% of all training steps, before linear decay, and tune the maximum learning rate to 3e − 3 for our SAS and SAS-P models. This was slightly too large for the SAS-P model without normalisation, so we reduce to 2e − 3. We inherit the clipping parameter of 0.5 from Geiping & Goldstein (2023). Training Fine-tuning We followed the same protocol as Geiping & Goldstein (2023), In particular, we finetune for 5 epochs with fixed hyperparameters across tasks. We found dropout to be important for good downstream performane (unlike during pre-training), and set dropout probability p = 0.1. We use batch size 32, with a maximum learning of 1.5e − 4. We keep other hyperparameters, e.g. the choice of cosine decay and AdamW epsilon 1e − 6, like from Geiping & Goldstein (2023). Fine-tuning Task The model is trained on the masked language modelling task with masking probability 0.25, as in Geiping & Goldstein (2023). Task This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv [9] https://github.com/JonasGeiping/cramming

Stories, News & Mentions about GOLDCAST

Goldcast’s Stories on HackerNoon

Goldcast's News & Mentions