**Product Manager, Credit Card**\n\nWe’re looking for an entrepreneurial product manager to take a lead role in continuing to build the first health and wellness credit card

Limit

This paper is available on arxiv under CC0 1.0 DEED license. Authors: (1) Salim Chouaki, LIX, CNRS, Inria, Ecole Polytechnique, Institut Polytechnique de Paris; (2) Oana Goga, LIX, CNRS, Inria, Ecole Polytechnique, Institut Polytechnique de Paris; (3) Hamed Haddadi, Imperial College London, Brave Software; (4) Peter Snyder, Brave Software. Table of Links Abstract and Introduction Background Measurement Methodology Results Limitations Related Work Conclusion, Acknowledgments, and References Appendix 5 LIMITATIONS Our measurement methodology has some limitations. First, we only look for user identifiers transferred in query parameters and do not detect them when they are transferred in other methods. For instance, previous work [33, 39] found that trackers sometimes decorate their own URL in the document.referrer header with user identifiers and reads them on the destination page. Second, we run all our crawling iterations from the same IP address. Consequently, if some query parameters are IP address based, they will have the same value across all iterations, and thus we would not consider them as user identifiers. Finally, our results are subject to variation based on the ads we selected and the search queries we used. Different search queries could potentially trigger distinct ads and lead to diverse advertisers, potentially exhibiting different behaviors. Nonetheless, our primary objective is to demonstrate the potential for third-party tracking when interacting with ads on private search engines.

Understanding the Privacy Risks of Popular Search Engine Advertising Systems: Limitations

Authors: (1) Renato P. dos Santos, CIAGE – Centre for Generative Artificial Intelligence in Cognition and Education. Table of Links Abstract and Introduction Materials And Methods Results and Analyses Prompts and generated texts Conceptualizing chemical reactions Deepening on understanding of chemical reactions Question about combustion Question about a graph of gases turning into water over time Question about the difference between atoms, molecules, and moles Deepening on the concept of mole Question about changing of state Question about an animated representation of water molecules undergoing phase changes Question about plasma, a state of matter Question about chemical bondings Question about illustration of chemical bonds Question about the essence of the type of chemical bonding Further analysis Conclusions Limitations of the study and possible future studies Author Contributions, Conflicts of interest, Acknowledgements, and References Limitations of the study and possible future studies Despite its inherent limitations, including its single case design and the potential for bias, the exploratory depth of the study uncovered hidden potential within these systems, even amid serious concerns about generalizability. Future research could include: • Refining the crafting of prompts. • Exploring new features of these and other GenAIbots being introduced with increasing frequency. • Establishing standardised benchmarks to evaluate and compare chatbots and AI systems' performance, accuracy, and reliability. • Conducting long-term studies to observe the evolution of chatbots' capabilities and their impact on user interactions over time. • Conducting research with real students in classroom settings and beyond to assess these AI systems' practical educational applications and challenges. • Investigating chatbots' learning and adaptation capabilities to individual user needs and preferences over time. • Researching the integration of multimodal inputs (e.g., text, voice, image) to enhance chatbot capabilities and user interaction experiences. When implementing GenAIbots in Chemistry learning, it is crucial to evaluate benefits and drawbacks judiciously, ensuring accurate information delivery and considering the implications of reduced human interaction. These concerns may be alleviated by integrating GenAIbots with other educational tools or activities promoting collaborative dialogue among learners. Author Contributions The author confirms sole responsibility for the following: study conception and design, data collection, analysis and interpretation of results, and manuscript preparation. Conflicts of interest There are no conflicts to declare. Acknowledgements The authors warmly acknowledge Melanie Swan for her invaluable suggestion, which led to the transition from using the term 'objects-to-think-with' to 'agents-to-thinkwith'. References Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionising education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152 Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4337484 Bardin, L. (1977). L’analyse de contenu. PUF - Presses Universitaires de France. Bitzenbauer, P. (2023). ChatGPT in physics education: A pilot study on easy-to-implement activities. Contemporary Educational Technology, 15(3), ep430. https://doi.org/10.30935/cedtech/13176 Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, 2005.14165. http://arxiv.org/abs/2005.14165 Castro Nascimento, C. M., & Pimentel, A. S. (2023). Do Large Language Models Understand Chemistry? A Conversation with ChatGPT. Journal of Chemical Information and Modeling, 63(6), 1649–1655. https://doi.org/10.1021/acs.jcim.3c00285 David, E. (2023, September 21). Microsoft to add DALL-E 3 to Bing Chat. The Verge; Vox Media. https://www.theverge.com/2023/9/21/23873690/microsoft-new-ai-features-bingsearch-shopping-dall-e-3 Dewi, C. A., Pahriah, P., & Purmadi, A. (2021). The Urgency of Digital Literacy for Generation Z Students in Chemistry Learning. International Journal of Emerging Technologies in Learning, 16(11), 88–103. https://doi.org/10.3991/ijet.v16i11.19871 Dunlop, L., Hodgson, A., & Stubbs, J. E. (2020). Building capabilities in chemistry education: happiness and discomfort through philosophical dialogue in chemistry. Chemistry Education Research and Practice, 21(1), 438–451. https://doi.org/10.1039/C9RP00141G Flavell, J. H. (1976). Metacognitive aspects of problem-solving. In L. B. Resnick (Ed.), The Nature of Intelligence (pp. 231–236). Lawrence Erlbaum. Floridi, L., & Chiriatti, M. (2020). GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines, 30(4), 681–694. https://doi.org/10.1007/s11023-020-09548-1 Franciscu, S. (2023). ChatGPT: A Natural Language Generation Model for Chatbots. https://doi.org/10.13140/RG.2.2.24777.83044 Gregorcic, B., & Pendrill, A.-M. (2023). ChatGPT and the frustrated Socrates. Physics Education, 58(3), 035021. https://doi.org/10.1088/1361-6552/acc299 Latour, B. (1991). Nous n’avons jamais été modernes: Essai d’anthropologie symétrique. La Découverte. Leon, A. J., & Vidhani, D. (2023). ChatGPT Needs a Chemistry Tutor, Too. Journal of Chemical Education. https://doi.org/10.1021/acs.jchemed.3c00288 Liu, F., Budiu, R., Zhang, A., & Cionca, E. (2023, October 1). ChatGPT, Bard, or Bing Chat? Differences Among 3 Generative-AI Bots. https://www.nngroup.com/articles/ai-botcomparison/ Metz, C., & Hsu, T. (2023, September 20). ChatGPT Can Now Generate Images, Too. New York Times - Technology. https://www.nytimes.com/2023/09/20/technology/chatgpt-dalle3-images-openai.html Mishra, A., Soni, U., Arunkumar, A., Huang, J., Kwon, B. C., & Bryan, C. (2023). PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models. ArXiv, 2304.01964. https://doi.org/10.48550/arXiv.2304.01964 Mollick, E. R. (2023, April 26). A guide to prompting AI (for what it is worth): A little bit of magic,but mostly just practice. One Useful Thing Blog. https://www.oneusefulthing.org/p/a-guidetoprompting-ai-for-what Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033 OpenAI. (2023). GPT-4 Technical Report. OpenAI. https://cdn.openai.com/papers/gpt-4.pdf Papert, S. A. (1980). Mindstorms - Children, Computers and Powerful Ideas. Basic Books. http://www.arvindguptatoys.com/arvindgupta/mindstorms.pdf Permatasari, M. B., Rahayu, S., & Dasna, W. (2022). Chemistry Learning Using Multiple Representations: A Systematic Literature Review. J.Sci.Learn.2022, 5(2), 334–341. https://doi.org/10.17509/jsl.v5i2.42656 Pimentel, A., Wagener, A., Silveira, E. F. da, Picciani, P., Salles, B., Follmer, C., & Jr., O. N. O. (2023). Challenging ChatGPT with Chemistry-Related Subjects. ChemRxiv. https://doi.org/10.26434/chemrxiv-2023-xl6w3 Schlosser, M. (2019). Agency. In The Stanford Encyclopedia of Philosophy. https://plato.stanford.edu/archives/win2019/entries/agency/ Swan, M. (2015, January 21). We Should Consider the Future World as one of Multi-Species Intelligence. Response to The Edge Question 2015: What Do You Think About Machines That Think?; Edge.org. http://edge.org/response-detail/26070 Taylor, C. A., Hogarth, H., Hacking, E. B., & Bastos, E. (2022). Posthuman Object Pedagogies: Thinking with Things to Think with Theory for Innovative Educational Research. Cultural and Pedagogical Inquiry, 14(1), 206–221. https://doi.org/10.18733/cpi29662 Timilsena, N. P., Maharjan, K. B., & Devkota, K. M. (2022). Teachers’ And Students’ Experiences In Chemistry Learning Difficulties. Journal of Positive School Psychology, 6(10), 2856–2867. https://www.journalppw.com/index.php/jpsp/article/view/13764 Tümay, H. (2016). Reconsidering learning difficulties and misconceptions in chemistry: emergence in chemistry and its implications for chemical education. Chemistry Education Research and Practice, 17(2), 229–245. https://doi.org/10.1039/C6RP00008H Turkle, S. (1984). The Second Self: Computers and the Human Spirit. Simon and Schuster. Yin, R. K. (2011). Applications of Case Study Research. SAGE Publications, Inc. This paper is under CC BY-SA 4.0 DEED license. available on arxiv

Overcoming Limitations in AI Chatbot Research for Future Educational Impact

ভবিষ্যতের শিক্ষাগত প্রভাবের জন্য এআই চ্যাটবট গবেষণায় সীমাবদ্ধতা অতিক্রম করা

Überwindung von Einschränkungen in der KI-Chatbot-Forschung für zukünftige Bildungseffekte

Superar las limitaciones en la investigación de chatbots de IA para lograr un impacto educativo futuro

Surmonter les limites de la recherche sur les chatbots IA pour un impact éducatif futur

भविष्य के शैक्षिक प्रभाव के लिए AI चैटबॉट अनुसंधान में सीमाओं पर काबू पाना

将来の教育への影響のために AI チャットボット研究の限界を克服する

미래 교육적 영향을 위한 AI 챗봇 연구의 한계 극복

Superando as limitações na pesquisa do AI Chatbot para um impacto educacional futuro

Преодоление ограничений в исследованиях чат-ботов с использованием искусственного интеллекта для будущего воздействия на образование

Gelecekteki Eğitim Etkisi için Yapay Zeka Chatbot Araştırmalarında Sınırlamaların Aşılması

Khắc phục những hạn chế trong nghiên cứu Chatbot AI để tạo ra tác động giáo dục trong tương lai

克服人工智能聊天机器人研究的局限性，为未来教育带来影响

This paper is under CC 4.0 license. available on arxiv Authors: (1) Jinrui Yang, School of Computing & Information Systems, The University of Melbourne (Email: jinruiy@student.unimelb.edu.au); (2) Timothy Baldwin, School of Computing & Information Systems, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence, UAE (Email: (tbaldwin,trevor.cohn)@unimelb.edu.au); (3) Trevor Cohn, School of Computing & Information Systems, The University of Melbourne. Table of Links Abstract and Intro Background and Related Work Multi-EuP Experiments and Findings Language Bias Discussion Conclusion, Limitations, Ethics Statement, Acknowledgements, References, and Appendix 6 Conclusion In this paper, we introduce Multi-EuP, a novel dataset for multilingual information retrieval across 24 languages, collected from European Parliament debates. The demographic information provided by the Multi-EuP dataset serves a dual purpose: not only does it contribute to multilingual retrieval tasks, but it also holds significant potential for advancing research in the realm of fairness and bias. This dataset can play a pivotal role in investigating issues of equitable representations and mitigation of biases within document ranking settings. Multi-EuP facilitates diverse information retrieval (IR) scenarios, encompassing one-vs-one, one-vs-many, and many-vs-many settings. We demonstrated the utility of Multi-EuP as a benchmark for evaluating both monolingual and multilingual IR. Our study reveals the presence of language bias in multilingual IR when employing BM25. We further validate the effectiveness of mitigating this bias through the strategic implementation of whitespace as a language tokenizer. We propose to conduct future work in three main areas. First, we intend to expand our investigation of language bias to encompass a broader range of ranking methods, including neural methods such as mDPR (Zhang et al., 2021), mColBERT (Lawrie et al., 2023) and PLAID-X(Santhanam et al., 2022). Second, we will expand the dataset by developing an automated API to retrieve data published by the European Parliament (EP), thereby ensuring realtime synchronization of our dataset. Lastly, our current experiments have explored language bias only, but we plan to further investigate gender bias, age bias, and nationality bias. Limitations The limitations of the Multi-EuP dataset are notable but navigable. Primarily, the temporal coverage of the dataset is confined to the past three years. This temporal constraint arises due to the fact that, preceding 2020, documents released by the EU were predominantly available in mono-lingual versions only. However, a potential remedy lies in the amalgamation of the Europarl (Koehn, 2005) collection, enabling a more comprehensive and holistic MultiEuP dataset. Furthermore, it is worth noting the domain skew of the dataset, in that Multi-EuP inevitably centers on political matters. While this presents challenges, particularly in terms of the intricate nuances of political language, it inherently serves as an excellent foundational stepping stone for delving into the intricacies of multilingual retrieval. We believe, however, that this dataset can serve as a launching pad for broader explorations encompassing crossdomain and open-domain transfer learning scenarios, thus contributing to the broader landscape of language understanding and retrieval. Ethics Statement The dataset contains publicly-available EP data that does not include personal or sensitive information, with the exception of information relating to public officeholders, e.g., the names of the active members of the European Parliament, European Council, or other official administration bodies. The collected data is licensed under the Creative Commons Attribution 4.0 International licence. [8] Acknowledgements This research was funded by Melbourne Research Scholarship and undertaken using the LIEF HPCGPGPU Facility hosted at the University of Melbourne. This facility was established with the assistance of LIEF Grant LE170100200. We would like to thank George Buchanan for providing valuable feedback. References Luiz Henrique Bonifacio, Israel Campiotti, Roberto de Alencar Lotufo, and Rodrigo Frassetto Nogueira. 2021. mMARCO: A multilingual version of MS MARCO passage ranking dataset. CoRR, abs/2108.13897. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. 8 https://eur-lex.europa.eu/cont Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for opendomain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. CoRR, abs/2004.12832. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand. Dawn Lawrie, Eugene Yang, Douglas W. Oard, and James Mayfield. 2023. Neural approaches to multilingual information retrieval. arXiv cs.IR 2209.01335. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, JhengHong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. https://github.com/ castorini/pyserini. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268. Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074–1084, Valencia, Spain. Association for Computational Linguistics. Razieh Rahimi, Azadeh Shakery, and Irwin King. 2015. Multilingual information retrieval in the language modeling framework. Information Retrieval Journal, 18:246–281. Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022. PLAID: An efficient engine for late interaction retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 1747–1756, New York, NY, USA. Association for Computing Machinery. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUSMT – building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation. Eva Vanmassenhove and Christian Hardmeier. 2018. Europarl datasets with demographic speaker information. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, page 391, Alicante, Spain. Denny Vrandeciˇ c and Markus Krötzsch. 2014. ´ Wikidata: A free collaborative knowledge base. Communications of the ACM, 57:78–85. Konrad Wojtasik, Vadim Shishkin, Kacper Wołowiec, Arkadiusz Janz, and Maciej Piasecki. 2023. BEIRPL: Zero shot information retrieval benchmark for the Polish language. arXiv cs.IR 2305.19840. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. arXiv cs.CL 2108.08787. A. Appendix [8] https://eur-lex.europa.eu/content/ legal-notice/legal-notice.html

Multi-EuP: Analysis of Bias in Information Retrieval - Conclusion, Limitations, and Ethics Statement

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Aleksandra Urman, She is a corresponding author from Department of Informatics, University of Zurich, Switzerland;
(2) Mykola Makhortykh, Institute of Communication and Media Studies, University of Bern, Switzerland;
(3) Roberto Ulloa, GESIS - Leibniz-Institut für Sozialwissenschaften, Germany;
(4) Juhi Kulshrestha, Department of Politics and Public Administration, University of Konstanz, Germany. Table of Links Abstract and Introduction
Web search results: biases, inaccuracies and their consequences
Conspiracy theories in the online environments
Research questions
Methodology
Results
Discussion
Limitations
Conclusion and Bibliography Limitations One limitation of the present study is that our selection of queries does not cover the whole range of existing conspiracy theories (Douglas et al., 2019) or all possible queries regarding a specific conspiracy theory, however, encompassing all theories or queries is arguably impossible, and our study should be treated as a first step towards more comprehensive analyses of conspiratorial information spread via SEs. Though the number of included theories is limited, we suggest that our approach is already more comprehensive than that of the majority of studies that focus on the spread of one specific theory online (with notable exceptions such as Mahl et al., 2021). As there is no established resource regarding the popularity of certain theories worldwide, our selection of queries was based on the internal discussions between authors, during which we tried to select the theories that are, first, in broad circulation and, second, that are “popular” worldwide, not within a single country. Another limitation of the present study is our focus on just three anglophone Western locations and on one language - English. This makes our analysis Western- and English-centric. We aim to address this in future work by conducting studies on the topic with broader linguistic and geographic focus. Finally, while we collected the data over two collection rounds to increase the robustness of the findings, these rounds were relatively close to each other. In the future, research infrastructure that enables more longitudinal data collection will be established to trace long-term changes (or their lack) in SEs’ outputs. This paper is available on arxiv under CC 4.0 license. Authors: (1) Aleksandra Urman, She is a corresponding author from Department of Informatics, University of Zurich, Switzerland; (2) Mykola Makhortykh, Institute of Communication and Media Studies, University of Bern, Switzerland; (3) Roberto Ulloa, GESIS - Leibniz-Institut für Sozialwissenschaften, Germany; (4) Juhi Kulshrestha, Department of Politics and Public Administration, University of Konstanz, Germany. This paper is available on arxiv under CC 4.0 license. Authors: (1) Aleksandra Urman, She is a corresponding author from Department of Informatics, University of Zurich, Switzerland; (2) Mykola Makhortykh, Institute of Communication and Media Studies, University of Bern, Switzerland; (3) Roberto Ulloa, GESIS - Leibniz-Institut für Sozialwissenschaften, Germany; (4) Juhi Kulshrestha, Department of Politics and Public Administration, University of Konstanz, Germany. Table of Links Abstract and Introduction Web search results: biases, inaccuracies and their consequences Conspiracy theories in the online environments Research questions Methodology Results Discussion Limitations Conclusion and Bibliography Abstract and Introduction Abstract and Introduction Web search results: biases, inaccuracies and their consequences Web search results: biases, inaccuracies and their consequences Conspiracy theories in the online environments Conspiracy theories in the online environments Research questions Research questions Methodology Methodology Results Results Discussion Discussion Limitations Limitations Conclusion and Bibliography Conclusion and Bibliography Limitations One limitation of the present study is that our selection of queries does not cover the whole range of existing conspiracy theories (Douglas et al., 2019) or all possible queries regarding a specific conspiracy theory, however, encompassing all theories or queries is arguably impossible, and our study should be treated as a first step towards more comprehensive analyses of conspiratorial information spread via SEs. Though the number of included theories is limited, we suggest that our approach is already more comprehensive than that of the majority of studies that focus on the spread of one specific theory online (with notable exceptions such as Mahl et al., 2021). As there is no established resource regarding the popularity of certain theories worldwide, our selection of queries was based on the internal discussions between authors, during which we tried to select the theories that are, first, in broad circulation and, second, that are “popular” worldwide, not within a single country. Another limitation of the present study is our focus on just three anglophone Western locations and on one language - English. This makes our analysis Western- and English-centric. We aim to address this in future work by conducting studies on the topic with broader linguistic and geographic focus. Finally, while we collected the data over two collection rounds to increase the robustness of the findings, these rounds were relatively close to each other. In the future, research infrastructure that enables more longitudinal data collection will be established to trace long-term changes (or their lack) in SEs’ outputs.

A Comparative Algorithm Audit of Conspiracies on the Net: Limitations

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk;
(2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk;
(3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction
Related work
Background
Methods
Experiments
"Conclusion, Limitations, and References"
Derivations
Algorithms
Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777. This paper is available on arxiv under CC 4.0 license. Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Sam Bowyer, Equal contribution, Department of Mathematics and sam.bowyer@bristol.ac.uk; (2)  Thomas Heap, Equal contribution, Department of Computer Science University of Bristol and thomas.heap@bristol.ac.uk; (3) Laurence Aitchison, Department of Computer Science University of Bristol and laurence.aitchison@bristol.ac.uk. Table of Links Abstract & Introduction Related work Background Methods Experiments "Conclusion, Limitations, and References" Derivations Algorithms Experimental Datasets And Model Abstract & Introduction Abstract & Introduction Related work Related work Background Background Methods Methods Experiments Experiments "Conclusion, Limitations, and References" "Conclusion, Limitations, and References" Derivations Derivations Algorithms Algorithms Experimental Datasets And Model Experimental Datasets And Model Conclusion and Limitations We gave a new and far simpler method for computing posterior moments, marginals and samples in massively parallel importance sampling based on differentiating a slightly modified marginal likelihood estimator. The method has limitations, in that while it is considerably more effective than e.g. VI, HMC and global importance sampling, it is more complex. Additionally, at least a naive implementation may be quite costly in terms of memory consumption, limiting how the number of importance samples we can draw for each variable. That said, it should be possible to eliminate almost all of this overhead by careful optimizations to avoid allocating large intermediate tensors, following the strategy in KeOps (Charlier et al. 2021). References Aitchison, L. 2019. Tensor Monte Carlo: particle methods for the GPU era. Advances in Neural Information Processing Systems, 32. Albarghouthi, A.; D’Antoni, L.; Drews, S.; and Nori, A. V. 2017. Fairsquare: probabilistic verification of program fairness. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–30. Andrieu, C.; Doucet, A.; and Holenstein, R. 2010. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(3): 269–342. Bidyuk, B.; and Dechter, R. 2007. Cutset sampling for Bayesian networks. Journal of Artificial Intelligence Research, 28: 1–48. Bornschein, J.; and Bengio, Y. 2014. Reweighted wakesleep. arXiv preprint arXiv:1406.2751. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Burda, Y.; Grosse, R.; and Salakhutdinov, R. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. Charlier, B.; Feydy, J.; Glaunes, J. A.; Collin, F.-D.; and Du- ` rif, G. 2021. Kernel Operations on the GPU, with Autodiff,  without Memory Overflows. Journal of Machine Learning Research. Chatterjee, S.; and Diaconis, P. 2018. The sample size required in importance sampling. The Annals of Applied Probability, 28(2): 1099–1135. Claret, G.; Rajamani, S. K.; Nori, A. V.; Gordon, A. D.; and Borgstrom, J. 2013. Bayesian inference using data flow anal- ¨ ysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, 92–102. Corenflos, A.; Chopin, N.; and Sarkk ¨ a, S. 2022. De- ¨ Sequentialized Monte Carlo: a parallel-in time particle smoother. arXiv preprint arXiv:2202.02264. Crucinio, F. R.; and Johansen, A. M. 2023. Properties of Marginal Sequential Monte Carlo Methods. ArXiv:2303.03498 [math, stat]. Dawid, A. P. 1992. Applications of a general propagation algorithm for probabilistic expert systems. Statistics and computing, 2(1): 25–36. Dechter, R.; Broka, F.; Kask, K.; and Ihler, A. 2018. Abstraction Sampling in Graphical Models. AAAI. Del Moral, P.; Doucet, A.; and Jasra, A. 2006. Sequential Monte Carlo Samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3): 411–436. DOE. 2023. Bus Breakdown and Delays. url https://data.cityofnewyork.us/Transportation/BusBreakdown-and-Delays/ez4e-fazm Accessed on: 05.05.2023. Doucet, A.; Johansen, A. M.; et al. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704): 3. Geffner, T.; and Domke, J. 2022. Variational Inference with Locally Enhanced Bounds for Hierarchical Models. arXiv preprint arXiv:2203.04432. Gehr, T.; Misailovic, S.; and Vechev, M. 2016. PSI: Exact symbolic inference for probabilistic programs. In International Conference on Computer Aided Verification, 62–83. Springer. Geldenhuys, J.; Dwyer, M. B.; and Visser, W. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, 166–176. Gogate, V.; and Dechter, R. 2012. Importance samplingbased estimation over and/or search spaces for graphical models. Artificial Intelligence, 184: 38–77. Goodman, N. D.; and Stuhlmuller, A. 2014. The design and implementation of probabilistic programming languages. Gordon, N. J.; Salmond, D. J.; and Smith, A. F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), volume 140, 107–113. IET. Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19. Heap, T.; and Laurence, A. 2023. Massively parallel reweighted wake-sleep. UAI. Hoffman, M. D.; Gelman, A.; et al. 2014. The No-UTurn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1): 1593–1623. Holtzen, S.; Van den Broeck, G.; and Millstein, T. 2020. Scaling exact inference for discrete probabilistic programs. Proceedings of the ACM on Programming Languages, 4(OOPSLA): 1–31. Jordan, M. I.; Ghahramani, Z.; Jaakkola, T. S.; and Saul, L. K. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2): 183–233. Kingma, D. P.; Welling, M.; et al. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4): 307–392. Kuntz, J.; Crucinio, F. R.; and Johansen, A. M. 2023. The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems. ArXiv:2110.15782 [math, stat]. Lai, J.; Domke, J.; and Sheldon, D. 2022. Variational Marginal Particle Filters. In International Conference on Artificial Intelligence and Statistics, 875–895. PMLR. Le, T. A.; Igl, M.; Rainforth, T.; Jin, T.; and Wood, F. 2017. Auto-encoding sequential monte carlo. arXiv preprint arXiv:1705.10306. Le, T. A.; Kosiorek, A. R.; Siddharth, N.; Teh, Y. W.; and Wood, F. 2020. Revisiting reweighted wake sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 1039–1049. PMLR. Lindsten, F.; Johansen, A. M.; Naesseth, C. A.; Kirkpatrick, B.; Schon, T. B.; Aston, J.; and Bouchard-Cotˆ e, A. 2017. ´ Divide-and-conquer with sequential Monte Carlo. Journal of Computational and Graphical Statistics, 26(2): 445–458. Maddison, C. J.; Lawson, J.; Tucker, G.; Heess, N.; Norouzi, M.; Mnih, A.; Doucet, A.; and Teh, Y. 2017. Filtering variational objectives. Advances in Neural Information Processing Systems, 30. Naesseth, C.; Linderman, S.; Ranganath, R.; and Blei, D. 2018. Variational sequential monte carlo. In International conference on artificial intelligence and statistics, 968–977. PMLR. Narayanan, P.; Carette, J.; Romano, W.; Shan, C.-c.; and Zinkov, R. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming, 62–79. Springer Obermeyer, F.; Bingham, E.; Jankowiak, M.; Pradhan, N.; Chiu, J.; Rush, A.; and Goodman, N. 2019. Tensor variable elimination for plated factor graphs. In International Conference on Machine Learning, 4871–4880. PMLR. Pfeffer, A. 2005. The design and implementation of IBAL: A generalpurpose probabilistic programming language. In Harvard Univesity. Citeseer. Salvatier, J.; Wiecki, T. V.; and Fonnesbeck, C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2: e55. Publisher: PeerJ Inc. Sankaranarayanan, S.; Chakarov, A.; and Gulwani, S. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, 447–458. Wang, D.; Hoffmann, J.; and Reps, T. 2018. PMAF: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices, 53(4): 513–528. Weinberg, S. 1995. The quantum theory of fields, volume 2. Cambridge university press. Zavatone-Veth, J.; Canatar, A.; Ruben, B.; and Pehlevan, C. 2021. Asymptotics of representation learning in finite Bayesian neural networks. Advances in neural information processing systems, 34: 24765–24777.

Using Autodiff to Estimate Posterior Moments: Conclusion, Limitations, and References

Authors:
(1) Ankur Nath, Department of Computer Science and Engineering, Texas A&M University;
(2) Alan Kuhnle, Department of Computer Science and Engineering, Texas A&M University. Table of Links Abstract & Introduction Related work Evaluation for Max-Cut Evaluation for SAT Summary and Outlook, References Supplementary Materials 4 EVALUATION for SAT 4.1 Problem Formulation 4.2 Learned Heuristics In this subsection, we delve into GNNSAT (Yolcu and P´oczos, 2019) and why we think there is a need for reevaluating GNNSAT. GNNSAT In their influential paper, Yolcu and P´oczos (2019) utilized a GNN to train on factor graph representations of boolean formulas in conjunctive normal form. Their goal was to learn a variable selection heuristic for a stochastic local search algorithm, WalkSAT (Selman et al., 1993). The algorithm demonstrated promising performance compared to WalkSAT. However, it fell short in terms of generalization abilities when compared to the WalkSAT. A direct comparison with SOTA SAT heuristics was omitted. As benchmark instances are extremely large and run time is the metric of performance for comparing with SAT heuristics, the scalability issues of GNNs represent a bottleneck for conducting a fair comparison with SOTA SAT heuristics. It is important to note that there have been significant improvements (McAllester et al., 1997; Hoos et al., 2002; Li et al., 2007; Mazure et al., 1997) in SAT heuristics since the WalkSAT algorithm. However, many of these improvements do not currently represent SOTA SAT heuristics. While we acknowledge that many neural networkbased approaches may face challenges in competing with SOTA SAT solvers in terms of run time due to scalability issues with GNNs, we believe it is still valuable to compare them with algorithms that demonstrate moderate performance. For instance, Kurin et al. (2020) compared their work with MiniSAT(E´en and S¨orensson, 2003). Since run time is an issue for GNN, they compared their algorithm with respect to the number of steps (one step is equivalent to choosing a variable and setting it to a value). This comparison provides insights into performance gaps between approaches and showcases the potential of learned SAT solvers to significantly reduce the number of steps. Such reductions, when coupled with advancements in domain-specific processors, can lead to a decrease in runtime, making learned SAT solvers more practical in the future. Therefore, we focus on evaluating how much improvement GNNSAT provides in terms of steps to solve a problem rather than solely considering absolute runtime. Specifically, our goal is to evaluate whether the performance of the learned heuristic remains competitive in terms of steps, particularly for small instances, when compared to simple heuristics. Addressing this question holds significant implications for the research community in devising strategies to bridge the performance gap. To achieve this objective, we undertake a comparative analysis between SoftTabu and GNNSAT, as both algorithms involve the selection of a variable at each time step and setting its value. 4.3 Dataset We conduct experiments utilizing randomly generated formulas derived from a diverse family of distributions, encompassing random 3-SAT, clique detection, graph coloring, and dominating set. These specific problem distributions were utilized in the evaluation of GNNSAT. Random K-SAT problems, especially when critically constrained, are hard, particularly in proximity to the satisfactory/unsatisfactory phase boundary (Mitchell et al., 1992; Selman et al., 1996). These problems serve as standard benchmarks for assessing SAT, with the threshold for 3-SAT occurring when problems have approximately 4.26 times as many clauses as variables. The other three problems fall under the category of NP-complete graph problems. For each of these problems, we randomly sample an Erd˝os-R´enyi graph denoted as G(N, p), where N is the number of vertices and p is the probability of an edge between any two vertices independently from every other edge. We use CNFgen (Lauria et al., 2017) for instance generation and MiniSAT for filtering out unsatisfiable formulas. It is worth noting that we exclude vertex covering—also a benchmark distribution for GNNSAT—due to the unavailability of support for this family of problems in the CNFgen package. 4.4 Is Scalability the Only Limiting Factor? To evaluate the algorithms, we sample 100 satisfiable formulas from each of the four problem distributions discussed above to obtain evaluation sets. We then run 25 search trials (with each trial starting at a random initial assignment) for each problem. Since comparing SoftTabu with GNNSAT in runtime will be unfair for the reasons stated above, we follow the evaluation method proposed in KhudaBukhsh et al. (2016). There are three performance metrics for each problem distribution: the average number of steps, the median of the median number of steps (the inner median is over trials on each problem and the outer median is over the problems in the evaluation sets), and the percentage of instances considered solved (the median number of steps less than the allowed number of steps). The main goal of this evaluation is to see if, with the help of GNN, GNNSAT can find the solution in fewer steps than SoftTabu. As shown in Table 3, SoftTabu demonstrates superior performance compared to the learned heuristic, thus shedding light on the performance limitations of the learned local search heuristic. Although GNNSAT is a novel attempt, it seems to show only marginal improvement in performance. Therefore, it is possible that GNNSAT is constrained by both performance and scalability. This paper is available on arxiv under CC 4.0 DEED license. Authors: (1) Ankur Nath, Department of Computer Science and Engineering, Texas A&M University; (2) Alan Kuhnle, Department of Computer Science and Engineering, Texas A&M University. Authors: Authors: (1) Ankur Nath, Department of Computer Science and Engineering, Texas A&M University; (2) Alan Kuhnle, Department of Computer Science and Engineering, Texas A&M University. Table of Links Abstract & Introduction Abstract & Introduction Related work Related work Evaluation for Max-Cut Evaluation for Max-Cut Evaluation for SAT Evaluation for SAT Summary and Outlook, References Summary and Outlook, References Supplementary Materials Supplementary Materials 4 EVALUATION for SAT 4.1 Problem Formulation 4.2 Learned Heuristics In this subsection, we delve into GNNSAT (Yolcu and P´oczos, 2019) and why we think there is a need for reevaluating GNNSAT. GNNSAT In their influential paper, Yolcu and P´oczos (2019) utilized a GNN to train on factor graph representations of boolean formulas in conjunctive normal form. Their goal was to learn a variable selection heuristic for a stochastic local search algorithm, WalkSAT (Selman et al., 1993). The algorithm demonstrated promising performance compared to WalkSAT. However, it fell short in terms of generalization abilities when compared to the WalkSAT. A direct comparison with SOTA SAT heuristics was omitted. As benchmark instances are extremely large and run time is the metric of performance for comparing with SAT heuristics, the scalability issues of GNNs represent a bottleneck for conducting a fair comparison with SOTA SAT heuristics. GNNSAT It is important to note that there have been significant improvements (McAllester et al., 1997; Hoos et al., 2002; Li et al., 2007; Mazure et al., 1997) in SAT heuristics since the WalkSAT algorithm. However, many of these improvements do not currently represent SOTA SAT heuristics. While we acknowledge that many neural networkbased approaches may face challenges in competing with SOTA SAT solvers in terms of run time due to scalability issues with GNNs, we believe it is still valuable to compare them with algorithms that demonstrate moderate performance. For instance, Kurin et al. (2020) compared their work with MiniSAT(E´en and S¨orensson, 2003). Since run time is an issue for GNN, they compared their algorithm with respect to the number of steps (one step is equivalent to choosing a variable and setting it to a value). This comparison provides insights into performance gaps between approaches and showcases the potential of learned SAT solvers to significantly reduce the number of steps. Such reductions, when coupled with advancements in domain-specific processors, can lead to a decrease in runtime, making learned SAT solvers more practical in the future. Therefore, we focus on evaluating how much improvement GNNSAT provides in terms of steps to solve a problem rather than solely considering absolute runtime. Specifically, our goal is to evaluate whether the performance of the learned heuristic remains competitive in terms of steps, particularly for small instances, when compared to simple heuristics. Addressing this question holds significant implications for the research community in devising strategies to bridge the performance gap. To achieve this objective, we undertake a comparative analysis between SoftTabu and GNNSAT, as both algorithms involve the selection of a variable at each time step and setting its value. 4.3 Dataset We conduct experiments utilizing randomly generated formulas derived from a diverse family of distributions, encompassing random 3-SAT, clique detection, graph coloring, and dominating set. These specific problem distributions were utilized in the evaluation of GNNSAT. Random K-SAT problems, especially when critically constrained, are hard, particularly in proximity to the satisfactory/unsatisfactory phase boundary (Mitchell et al., 1992; Selman et al., 1996). These problems serve as standard benchmarks for assessing SAT, with the threshold for 3-SAT occurring when problems have approximately 4.26 times as many clauses as variables. The other three problems fall under the category of NP-complete graph problems. For each of these problems, we randomly sample an Erd˝os-R´enyi graph denoted as G(N, p), where N is the number of vertices and p is the probability of an edge between any two vertices independently from every other edge. We use CNFgen (Lauria et al., 2017) for instance generation and MiniSAT for filtering out unsatisfiable formulas. It is worth noting that we exclude vertex covering—also a benchmark distribution for GNNSAT—due to the unavailability of support for this family of problems in the CNFgen package. 4.4 Is Scalability the Only Limiting Factor? To evaluate the algorithms, we sample 100 satisfiable formulas from each of the four problem distributions discussed above to obtain evaluation sets. We then run 25 search trials (with each trial starting at a random initial assignment) for each problem. Since comparing SoftTabu with GNNSAT in runtime will be unfair for the reasons stated above, we follow the evaluation method proposed in KhudaBukhsh et al. (2016). There are three performance metrics for each problem distribution: the average number of steps, the median of the median number of steps (the inner median is over trials on each problem and the outer median is over the problems in the evaluation sets), and the percentage of instances considered solved (the median number of steps less than the allowed number of steps). The main goal of this evaluation is to see if, with the help of GNN, GNNSAT can find the solution in fewer steps than SoftTabu. As shown in Table 3, SoftTabu demonstrates superior performance compared to the learned heuristic, thus shedding light on the performance limitations of the learned local search heuristic. Although GNNSAT is a novel attempt, it seems to show only marginal improvement in performance. Therefore, it is possible that GNNSAT is constrained by both performance and scalability. This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license. available on arxiv

Understanding the Limitations of GNNSAT in SAT Heuristic Optimization

Authors:
(1) Ankur Nath, Department of Computer Science and Engineering, Texas A&M University;
(2) Alan Kuhnle, Department of Computer Science and Engineering, Texas A&M University. Table of Links Abstract & Introduction Related work Evaluation for Max-Cut Evaluation for SAT Summary and Outlook, References Supplementary Materials Abstract In recent years, combining neural networks with local search heuristics has become popular in the field of combinatorial optimization. Despite its considerable computational demands, this approach has exhibited promising outcomes with minimal manual engineering. However, we have identified three critical limitations in the empirical evaluation of these integration attempts. Firstly, instances with moderate complexity and weak baselines pose a challenge in accurately evaluating the effectiveness of learning-based approaches. Secondly, the absence of an ablation study makes it difficult to quantify and attribute improvements accurately to the deep learning architecture. Lastly, the generalization of learned heuristics across diverse distributions remains underexplored. In this study, we conduct a comprehensive investigation into these identified limitations. Surprisingly, we demonstrate that a simple learned heuristic based on Tabu Search surpasses state-of-the-art (SOTA) learned heuristics in terms of performance and generalizability. Our findings challenge prevailing assumptions and open up exciting avenues for future research and innovation in combinatorial optimization. 1 INTRODUCTION Designing effective heuristics or approximation algorithms for NP-hard combinatorial optimization (CO) problems is a challenging task, often requiring domain knowledge and extensive trial-and-error. Thus, the idea of automating this demanding and tedious design process through machine learning to learn algorithms that exploit the inherent structure of a problem has gained significant interest among researchers (Bello et al., 2016; Khalil et al., 2017; Bengio et al., 2021; Dong et al., 2021). Particularly, a considerable portion of these works (Khalil et al., 2017; Barrett et al., 2020; Yolcu and P´oczos, 2019) have concentrated on employing Graph Neural Networks (GNNs) for CO problems. Despite the computational demands, these GNN-based approaches have demonstrated competitive performance compared to SOTA heuristics tailored to specific problems. While these potential advancements offer great promise, some concerns remain. The superior performance of the learned heuristics can be attributed to the selection of specific instances and baselines. Specifically, if the baselines are weak, the learned heuristics can easily outperform them. Without hard instances and proper baseline selection, the learned heuristics can easily show comparable performance with the SOTA heuristics, and this can lead to an overestimation of the actual capabilities of the learned heuristics. Moreover, comparisons with SOTA heuristics are occasionally left out due to scalability challenges with deep learning architectures. A subset of learning-based approaches (Khalil et al., 2017; Yolcu and P´oczos, 2019; Barrett et al., 2020, 2022) incorporate the functionality or behavior of traditional heuristics, potentially offering improved or enhanced performance by integrating heuristics principles with machine learning components. If a comprehensive comparison with the integrated heuristics and an ablation study of the deep learning architecture are lacking, it becomes challenging to determine the specific contribution of the deep learning architecture. Consequently, if the integrated heuristics are robust, the learned heuristics can seamlessly outperform the baseline heuristics, while the deep learning architecture plays a little role. Another great achievement of learned heuristics (Khalil et al., 2017; Barrett et al., 2020; Toenshof et al., 2019), initially trained on smaller and specific instances from a particular distribution, exhibits impressive performance when tested on larger instances from different distributions. This achievement stands as a significant accomplishment, aligning with the core objective of employing learning-based approaches: to mitigate the necessity for extensive customization and domain-specific knowledge often required by heuristics. Although classical heuristics with hyperparameters may also encounter challenges if they are fine-tuned for a specific distribution, they may also generalize across different distributions. The primary inquiry revolves around whether the learned heuristics indeed demonstrate superior generalizability compared to classical heuristics. A thorough and insightful comparison of learned heuristics against classical heuristics provides valuable insights into the generalizability of learned heuristics. Learned heuristics often lack theoretical guarantees, making empirical evaluation the sole method to comprehend the strengths and limitations of proposed methodologies. We believe that several fundamental, yet pivotal questions remain unexplored in the empirical evaluation of these works. While these inquiries are pertinent to all types of learned heuristics, we ask and answer them by analyzing highly cited and recent peer-reviewed publications focused on learning local search heuristics. Our goal is not to provide exhaustive benchmarks for the CO problems discussed in our work but rather to assist future researchers in evaluating their research. Concretely, we ask and answer the following questions: 1. Can learned heuristics be over-parameterized? Absolutely. Replacing the GNN in ECO-DQN with linear regression and utilizing a subset of features from ECO-DQN (Barrett et al., 2020) that links to Tabu Search (Glover, 1990), we introduce a pruned version of ECO-DQN termed SoftTabu. Our study demonstrates that SoftTabu showcases superior performance and generalizability in comparison to ECO-DQN for the NPhard Maximum-Cut (Max-Cut) problem. 2. Can baseline bias be attributed to the superior performance of learned heuristics? Yes, we demonstrate that SoftTabu, a vanilla learned heuristic, can outperform S2V-DQN (Khalil et al., 2017) for the Max-Cut problem and GNNSAT (Yolcu and P´oczos, 2019) for Boolean Satisfiability (SAT). *3. Can learned heuristics demonstrate superior generalizability owing to instance selection bias?*Yes, we demonstrate that ECO-DQN shows poor generalization on harder instances and easily gets trapped in the search space. This paper is available on arxiv under CC 4.0 DEED license. Authors: (1) Ankur Nath, Department of Computer Science and Engineering, Texas A&M University; (2) Alan Kuhnle, Department of Computer Science and Engineering, Texas A&M University. Authors: Authors: (1) Ankur Nath, Department of Computer Science and Engineering, Texas A&M University; (2) Alan Kuhnle, Department of Computer Science and Engineering, Texas A&M University. Table of Links Abstract & Introduction Abstract & Introduction Related work Related work Evaluation for Max-Cut Evaluation for Max-Cut Evaluation for SAT Evaluation for SAT Summary and Outlook, References Summary and Outlook, References Supplementary Materials Supplementary Materials Abstract In recent years, combining neural networks with local search heuristics has become popular in the field of combinatorial optimization. Despite its considerable computational demands, this approach has exhibited promising outcomes with minimal manual engineering. However, we have identified three critical limitations in the empirical evaluation of these integration attempts. Firstly, instances with moderate complexity and weak baselines pose a challenge in accurately evaluating the effectiveness of learning-based approaches. Secondly, the absence of an ablation study makes it difficult to quantify and attribute improvements accurately to the deep learning architecture. Lastly, the generalization of learned heuristics across diverse distributions remains underexplored. In this study, we conduct a comprehensive investigation into these identified limitations. Surprisingly, we demonstrate that a simple learned heuristic based on Tabu Search surpasses state-of-the-art (SOTA) learned heuristics in terms of performance and generalizability. Our findings challenge prevailing assumptions and open up exciting avenues for future research and innovation in combinatorial optimization. 1 INTRODUCTION Designing effective heuristics or approximation algorithms for NP-hard combinatorial optimization (CO) problems is a challenging task, often requiring domain knowledge and extensive trial-and-error. Thus, the idea of automating this demanding and tedious design process through machine learning to learn algorithms that exploit the inherent structure of a problem has gained significant interest among researchers (Bello et al., 2016; Khalil et al., 2017; Bengio et al., 2021; Dong et al., 2021). Particularly, a considerable portion of these works (Khalil et al., 2017; Barrett et al., 2020; Yolcu and P´oczos, 2019) have concentrated on employing Graph Neural Networks (GNNs) for CO problems. Despite the computational demands, these GNN-based approaches have demonstrated competitive performance compared to SOTA heuristics tailored to specific problems. While these potential advancements offer great promise, some concerns remain. The superior performance of the learned heuristics can be attributed to the selection of specific instances and baselines. Specifically, if the baselines are weak, the learned heuristics can easily outperform them. Without hard instances and proper baseline selection, the learned heuristics can easily show comparable performance with the SOTA heuristics, and this can lead to an overestimation of the actual capabilities of the learned heuristics. Moreover, comparisons with SOTA heuristics are occasionally left out due to scalability challenges with deep learning architectures. A subset of learning-based approaches (Khalil et al., 2017; Yolcu and P´oczos, 2019; Barrett et al., 2020, 2022) incorporate the functionality or behavior of traditional heuristics, potentially offering improved or enhanced performance by integrating heuristics principles with machine learning components. If a comprehensive comparison with the integrated heuristics and an ablation study of the deep learning architecture are lacking, it becomes challenging to determine the specific contribution of the deep learning architecture. Consequently, if the integrated heuristics are robust, the learned heuristics can seamlessly outperform the baseline heuristics, while the deep learning architecture plays a little role. Another great achievement of learned heuristics (Khalil et al., 2017; Barrett et al., 2020; Toenshof et al., 2019), initially trained on smaller and specific instances from a particular distribution, exhibits impressive performance when tested on larger instances from different distributions. This achievement stands as a significant accomplishment, aligning with the core objective of employing learning-based approaches: to mitigate the necessity for extensive customization and domain-specific knowledge often required by heuristics. Although classical heuristics with hyperparameters may also encounter challenges if they are fine-tuned for a specific distribution, they may also generalize across different distributions. The primary inquiry revolves around whether the learned heuristics indeed demonstrate superior generalizability compared to classical heuristics. A thorough and insightful comparison of learned heuristics against classical heuristics provides valuable insights into the generalizability of learned heuristics. Learned heuristics often lack theoretical guarantees, making empirical evaluation the sole method to comprehend the strengths and limitations of proposed methodologies. We believe that several fundamental, yet pivotal questions remain unexplored in the empirical evaluation of these works. While these inquiries are pertinent to all types of learned heuristics, we ask and answer them by analyzing highly cited and recent peer-reviewed publications focused on learning local search heuristics. Our goal is not to provide exhaustive benchmarks for the CO problems discussed in our work but rather to assist future researchers in evaluating their research. Concretely, we ask and answer the following questions: 1. Can learned heuristics be over-parameterized? Absolutely. Replacing the GNN in ECO-DQN with linear regression and utilizing a subset of features from ECO-DQN (Barrett et al., 2020) that links to Tabu Search (Glover, 1990), we introduce a pruned version of ECO-DQN termed SoftTabu. Our study demonstrates that SoftTabu showcases superior performance and generalizability in comparison to ECO-DQN for the NPhard Maximum-Cut (Max-Cut) problem. 1. Can learned heuristics be over-parameterized? Absolutely. 2. Can baseline bias be attributed to the superior performance of learned heuristics? Yes, we demonstrate that SoftTabu, a vanilla learned heuristic, can outperform S2V-DQN (Khalil et al., 2017) for the Max-Cut problem and GNNSAT (Yolcu and P´oczos, 2019) for Boolean Satisfiability (SAT). 2. Can baseline bias be attributed to the superior performance of learned heuristics? *3. Can learned heuristics demonstrate superior generalizability owing to instance selection bias?*Yes, we demonstrate that ECO-DQN shows poor generalization on harder instances and easily gets trapped in the search space. This paper is available on arxiv under CC 4.0 DEED license. This paper is available on arxiv under CC 4.0 DEED license.

Unveiling the Limits of Learned Local Search Heuristics: Are You the Mightiest of the Meek?

শেখা স্থানীয় অনুসন্ধান হিউরিস্টিকসের সীমা উন্মোচন করা: আপনি কি নম্রদের মধ্যে সবচেয়ে শক্তিশালী?

Die Grenzen erlernter Heuristiken für die lokale Suche aufdecken: Sind Sie der Mächtigste unter den Sanftmütigen?

Revelando los límites de la heurística de búsqueda local aprendida: ¿eres el más poderoso entre los mansos?

Dévoiler les limites des heuristiques de recherche locale apprises : êtes-vous le plus puissant des doux ?

सीखी गई स्थानीय खोज हेयुरिस्टिक्स की सीमाओं का अनावरण: क्या आप नम्र लोगों में सबसे शक्तिशाली हैं?

学習したローカル検索ヒューリスティックスの限界を明らかにする: あなたは最も強い弱者ですか?

학습된 지역 검색 휴리스틱의 한계 공개: 당신은 가장 온유한 사람입니까?

Revelando os limites das heurísticas de pesquisa local aprendidas: você é o mais poderoso dos mansos?

Раскрытие границ изученной эвристики локального поиска: вы самый могущественный из кротких?

Öğrenilmiş Yerel Arama Buluşsal Yöntemlerinin Sınırlarını Ortaya Çıkarıyoruz: Uysalların En Güçlüsü Siz misiniz?

Tiết lộ các giới hạn của phương pháp phỏng đoán tìm kiếm địa phương đã học: Bạn có phải là người mạnh mẽ nhất trong số những người nhu mì không?

揭示学习本地搜索启发式方法的局限性：你是最强大的吗？

Authors: (1) Muzhaffar Hazman, University of Galway, Ireland; (2) Susan McKeever, Technological University Dublin, Ireland; (3) Josephine Griffith, University of Galway, Ireland. Table of Links Abstract and Introduction Related Works Methodology Results Limitations and Future Works Conclusion, Acknowledgments, and References A Hyperparameters and Settings B Metric: Weighted F1-Score C Architectural Details D Performance Benchmarking E Contingency Table: Baseline vs. Text-STILT 5 Limitations and Future Works To generate comparable results between Baseline and Text-STILT, we kept many hyperparameters constant. Additional work would be required to determine the maximum achievable performance of Text-STILT on the chosen task. Despite the efficacy of Text-STILT over ImageSTILT, these results do not suggest that only the text modality is significant in classifying multimodal memes. Previous works have performed modality ablation studies in this problem space (Bucur et al., 2022; Pramanick et al., 2021b; Keswani et al., 2020) with multimodal architectures remaining the apparent state of the practise. All models in this work are similarly multimodal. In the future, we plan to reformulate Image-STILT with respect to the approach and data used to isolate the cause of its non-performance on the downstream task. Furthermore, we did not test Text-STILT on classifiers that represent the image modality of a meme in textual forms, as others did (Singh et al., 2022; Pramanick et al., 2021b). Notwithstanding our results, Text-STILT may not benefit all multimodal meme classifiers. Phang et al. (2019) showed that STILT offers varying degrees of benefit depending on the encoders chosen. Future work is needed to verify if these observations hold across the wide range of pretrained encoders commonly used in meme classifiers. In particular, some modifications to unimodal STILTs are needed to be applied to single-stream multimodal encoders, as those used in other works. Furthermore, Pruksachatkun et al. (2020) showed that intermediate training benefits various text-only tasks differently. We have yet to identify other meme classification tasks that would benefit from unimodal STILTs. Thus, we plan to conduct more extensive experimentation to validate the effectiveness of Text-STILT on other meme classification tasks, e.g. pairing hate-speech detection in text (Toraman et al., 2022) as an intermediate task for hateful meme detection (Kiela et al., 2020). This paper is under CC 4.0 license. available on arxiv

LIMIT

limit.com

-- employees

Since n.d.

COMPANY RANKING

LIMIT

EVERGREEN INDEX #7250

Representing this company? Click here to claim and customize this page!

Limit WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

LIMIT

limit.com

-- employees

Since n.d.

COMPANY RANKING

LIMIT

EVERGREEN INDEX #7250

RELATED COMPANIESRANK

STORIES ABOUT LIMIT

Representing this company? Click here to claim and customize this page!

Limit WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!