Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Community Challenges Overview and 2.1 CCKS 2. Community Challenges Overview and 2.1 CCKS 2.2 CHIP and 2.3 CCIR, CSMI, CCL and DCIC 2.2 CHIP and 2.3 CCIR, CSMI, CCL and DCIC 3. Evaluation Tasks Overview and 3.1 Information Extraction 3. Evaluation Tasks Overview and 3.1 Information Extraction 3.2 Text Classification and Text Similarity 3.2 Text Classification and Text Similarity 3.3 Knowledge Graph and Question Answering 3.3 Knowledge Graph and Question Answering 3.4 Text Generation and Knowledge Reasoning and 3.5 Large Language Model Evaluation 3.4 Text Generation and Knowledge Reasoning and 3.5 Large Language Model Evaluation 4. Translational Informatics in Biomedical Text Mining 4. Translational Informatics in Biomedical Text Mining 5. Discussion and Perspective 5.1. Contributions of Community Challenges 5.1. Contributions of Community Challenges 5.2. Limitations of Current Community Challenges 5.2. Limitations of Current Community Challenges 5.3. Future Perspectives in the Era of Large Language Models, and References 5.3. Future Perspectives in the Era of Large Language Models, and References Figure Legends and Tables Figure Legends and Tables 5.1. Contributions of Community Challenges Community challenges has made significant contributions for biomedical text mining research in many aspects. We summarize five notable contributions ranging from research level to application level, including driving the development of data standards and evaluation benchmarks, promoting technological innovation and development, fostering interdisciplinary collaboration and communication, promoting education and training, facilitating translational informatics application (see Figure 4). Datasets and benchmarks Datasets and benchmarks The community challenges generally release datasets that are related to specific NLP tasks. These datasets are manually annotated by domain experts, and are sourced from a wide range of biomedical materials, such as published literatures, clinical records, and internet forums. These datasets serve as valuable resources for training and evaluating models, as well as benchmarks for comparing the performance of different approaches. For example, researchers have developed the benchmark CBLUE [64], which combines datasets from multiple shared tasks and aims to evaluate the model's ability in Chinese biomedical language understanding. Moreover, Recently, researchers have also transformed numerous tasks into prompt-based language generation tasks and construct an evaluation benchmark PromptCBLUE [59], which aims to evaluate the capabilities of large language models in healthcare domain. Technology innovation and development Technology innovation and development The community challenges encourage participants to propose state-of-the-art algorithms and methodologies to address problems in biomedical text mining. This competitive environment stimulates researchers' innovative thinking and technological advancements, driving the development and breakthroughs in the field. Taking the information extraction tasks as an example. Initially, early solutions relied on rule-based and dictionary-based methods. Then, deep neural network methods, particularly long short-term memory (LSTM) architecture, gradually gained traction and garnered significant attention. Subsequently, BERT renowned for their powerful text representation capabilities, were often used by participants and found applications in various biomedical fields [68]. In recent research, large language models such as ChatGPT have also been leveraged for purposes such as data augmentation or zero-shot information extraction [69], showcasing their potential for advancing the field even further. Interdisciplinary collaboration and communication Interdisciplinary collaboration and communication Numerous experts with different research background participate in these community challenges. These experts, hailing from both academia and industry, come together through academic conferences to showcase their algorithm architectures, performance outcomes, and explore potential collaboration opportunities. Additionally, the community challenges also provide benefits to participants through the publication of papers on specific task topics. This interdisciplinary collaboration and communication play a crucial role in promoting knowledge sharing and cooperation among different domains, thus driving the development of biomedical text mining. Education and training Education and training The evaluation tasks serve as valuable learning and practical resources for students and beginners in the BioNLP domain. They allow participants to focus on specific tasks, engage in competition, discuss with experts, and improve their skills by evaluating their methods and receiving constructive feedback. After the task is completed, the researchers can further utilize the dataset to evaluate the algorithms and publish articles. Moreover, these shared tasks also provide open access to relevant datasets, which are often used by universities for teaching courses and practical exercises to assess student learning performance. It is worth noting that some datasets explicitly state their restriction to be used only during community challenges and are not made available thereafter. Translational informatics application Translational informatics application Biomedical text mining has the potential to contribute significantly to the standardization of knowledge. By extracting valuable information from vast volumes of biomedical literature, it becomes possible to establish high-quality knowledge bases and ontologies. These resources, in turn, can offer timely evidence-based information for researchers, clinicians, and policy makers. For example, mining the gene-disease semantic associations from literatures can help researchers in discovering novel gene candidates related to diseases [70], while extracting PICOS information from the literature can contribute to the construction of clinical evidence knowledge bases [71]. Biomedical text mining plays a crucial role in clinical decision support. Through the extraction and normalization of clinical information, large quantities of unstructured clinical textual data can be converted into structured and computable formats. This transformation enables clinicians to access comprehensive and accurate patient information, facilitating clinical diagnosis and treatment recommendations. Moreover, biomedical text mining techniques can also be employed in epidemic trend prediction and risk assessment. By mining extensive clinical textual data, it becomes possible to identify key features and patterns associated with patients, which enabling the implementation of appropriate intervention measures to improve patient health outcomes. Biomedical text mining holds significant potential for commercial applications. In the community challenges, multiples evaluation tasks are organized by company. These tasks typically employ data from real-world healthcare sources and have clear clinical application scenarios, such as patient health consultation, and automated question answering. The successful development of solutions for these tasks can provide opportunities and competitive advantages for businesses. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv Authors: (1) Hui Zong, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China and the author contributed equally; (2) Rongrong Wu, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China and the author contributed equally; (3) Jiaxue Cha, Shanghai Key Laboratory of Signaling and Disease Research, Laboratory of Receptor-Based Bio-Medicine, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, 200092, China; (4) Erman Wu, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China; (5) Jiakun Li, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China and Department of Urology, West China Hospital, Sichuan University, Chengdu, 610041, China; (6) Liang Tao, Faculty of Business Information, Shanghai Business School, Shanghai, 201400, China; (7) Zuofeng Li, Takeda Co. Ltd., Shanghai, 200040, China; (8) Buzhou Tang, Department of Computer Science, Harbin Institute of Technology, Shenzhen, 518055, China; (9) Bairong Shen, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China and a Corresponding author. Authors: Authors: (1) Hui Zong, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China and the author contributed equally; (2) Rongrong Wu, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China and the author contributed equally; (3) Jiaxue Cha, Shanghai Key Laboratory of Signaling and Disease Research, Laboratory of Receptor-Based Bio-Medicine, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, 200092, China; (4) Erman Wu, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China; (5) Jiakun Li, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China and Department of Urology, West China Hospital, Sichuan University, Chengdu, 610041, China; (6) Liang Tao, Faculty of Business Information, Shanghai Business School, Shanghai, 201400, China; (7) Zuofeng Li, Takeda Co. Ltd., Shanghai, 200040, China; (8) Buzhou Tang, Department of Computer Science, Harbin Institute of Technology, Shenzhen, 518055, China; (9) Bairong Shen, Joint Laboratory of Artificial Intelligence for Critical Care Medicine, Department of Critical Care Medicine and Institutes for Systems Genetics, Frontiers Science Center for Disease related Molecular Network, West China Hospital, Sichuan University, Chengdu, 610041, China and a Corresponding author.
