Miko

miko

HackerNoon Reporter: Please tell us briefly about your  background. I am a Finnish American software engineer. What's your startup called? And in a sentence or two, what does it do? is a Google and Facebook authenticated bidding-fee-auction app for Amazon items. Exhibia.com What is the origin story? The original Yale University 1971 dollar auction game was digitized in the early 2000, however it was digitized incorrectly and commonsense auction initiation criteria was ignore. What do you love about your team, and why are you the ones to solve this problem? We launched one of the first penny auction sites in the USA and learned really early on the game what was fundamentally wrong and how to fix it. The “precommitment mechanism” in penny auction method is not formal and it doesn’t check that every live auction has the “minimum required number of bidders”. If you weren’t building your startup, what would you be doing? building software At the moment, how do you measure success? What are your core metrics? Signups What’s most exciting about your traction to date? Granted patents and we are generating users daily. 90% of software startups fail because no one wants to use them. What technologies are you currently most excited about, and most worried about? And why? Cryptocurrencies. The original inventor from Yale University Dr.Martin Shubik compares the dollar auction game to a coin-toss, or token-toss game. The auction method can be used with any crypto currency and we plan to launch our own Bidcoin. What drew you to get published on HackerNoon? What do you like most about our platform? Being a software-engineer myself, HackerNoon is an important source of information. What advice would you give to the 21-year-old version of yourself? Save money. What is something surprising you've learned this year that your contemporaries would benefit from knowing? That the auction operator needs to make sure before auction is started, that “minimum required number of bidders is present“. Exhibia was nominated as in Startups of the Year hosted by HackerNoon. one of the best startups in Miami

"90% of software startups fail because no one wants to use them" according to Exhibia CEO Miko Lasso

AI Bias Isn't Always Loud—Sometimes Its Laissez-Faire

Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Method and 3.1 Phase 1: Taxonomy Generation 3 Method and 3.1 Phase 1: Taxonomy Generation 3.2 Phase 2: LLM-Augmented Text Classification 3.2 Phase 2: LLM-Augmented Text Classification 4 Evaluation Suite and 4.1 Phase 1 Evaluation Strategies 4 Evaluation Suite and 4.1 Phase 1 Evaluation Strategies 4.2 Phase 2 Evaluation Strategies 4.2 Phase 2 Evaluation Strategies 5 Experiments and 5.1 Data 5 Experiments and 5.1 Data 5.2 Taxonomy Generation 5.2 Taxonomy Generation 5.3 LLM-Augmented Text Classification 5.3 LLM-Augmented Text Classification 5.4 Summary of Findings and Suggestions 5.4 Summary of Findings and Suggestions 6 Discussion and Future Work, and References 6 Discussion and Future Work, and References A. Taxonomies A. Taxonomies B. Additional Results B. Additional Results C. Implementation Details C. Implementation Details D. Prompt Templates D. Prompt Templates 6 DISCUSSION AND FUTURE WORK This work has the potential to create significant impact for research and application of AI technologies in text mining. Our framework has demonstrated the ability to use LLMs as taxonomy generators, as well as data labelers and evaluators. These automations could lead to significant efficiency gains and cost savings for a variety of domains and applications that rely on understanding, structuring and analyzing massive volumes of unstructured text. It could also broadly democratize the process of mining knowledge from text, empowering non-expert users and enterprises to interact with and interpret their data through natural language, thereby leading to better insights and data-driven decision making for a range of industries and sectors. Additionally, our framework and research findings relate to other work that leverages LLMs for taxonomy creation and text clustering, and has important empirical lessons for the efficient use of instruction-following models in these scenarios. Despite these initial successes, there are some important challenges and future directions that are worth exploring. As we have already noted, LLMs are expensive and slow. In future work, we hope to explore ways to improve the speed, efficiency and robustness of our framework, through hybrid approaches that further explore the combination of LLMs with embedding-based methods, or model distillation that fine-tunes a smaller model through instructions from a larger one. Evaluation continues to be a crucial and open challenge for future work, and we plan to explore ways of performing more robust LLM-aided evaluations in future work, for example by fine-tuning a model to expand its reasoning capabilities beyond pairwise judgement tasks. While this work has focused largely on text mining in the conversational domain, we also hope to explore the extensibility of our framework to other domains as well. Finally, many domains have ethical considerations from the perspective of privacy and security that must be taken into account when performing large-scale automated text mining, and we hope to engage with these challenges more deeply in future work. REFERENCES [1] Charu C Aggarwal and ChengXiang Zhai. 2012. A survey of text clustering algorithms. Mining text data (2012), 77–128. [2] David Arthur, Sergei Vassilvitskii, et al. 2007. k-means++: The advantages of careful seeding. In Soda, Vol. 7. 1027–1035. [3] Léon Bottou. 1998. Online algorithms and stochastic approximations. Online learning in neural networks (1998). [4] B Barla Cambazoglu, Leila Tavakoli, Falk Scholer, Mark Sanderson, and Bruce Croft. 2021. An intent taxonomy for questions asked in web search. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval. 85–94. [5] Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. 2009. Reading tea leaves: How humans interpret topic models. Advances in neural information processing systems 22 (2009). [6] Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46. [7] Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement 33, 3 (1973), 613–619. [8] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056 (2023). [9] Simon Haykin. 1998. Neural networks: a comprehensive foundation. Prentice Hall PTR. [10] Andreas Hotho, Andreas Nürnberger, and Gerhard Paaß. 2005. A brief survey of text mining. Journal for Language Technology and Computational Linguistics 20, 1 (2005), 19–62. [11] Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. FastText.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651 (2016). [12] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of Tricks for Efficient Text Classification. arXiv preprint arXiv:1607.01759 (2016). [13] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 30 (2017). [14] Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). https://api.semanticscholar.org/CorpusID:6628106 https://api.semanticscholar.org/CorpusID: [15] Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen White, and Sujay Jauhar. 2023. Making Large Language Models Better Data Creators. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 15349–15360. https://doi.org/10.18653/v1/2023.emnlp-main.948 https://doi.org/10.18653/v1/2023.emnlp-main.948 [16] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 (2023). [17] Geoffrey J McLachlan and Kaye E Basford. 1988. Mixture models: Inference and applications to clustering. Vol. 38. M. Dekker New York. [18] Chau Minh Pham, Alexander Hoyle, Simeng Sun, and Mohit Iyyer. 2023. TopicGPT: A Prompt-based Topic Modeling Framework. arXiv preprint arXiv:2311.01449 (2023). [19] Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic Prompt Optimization with “Gradient Descent” and Beam Search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 7957–7968. https://doi.org/10.18653/v1/2023.emnlp-main.494 https://doi.org/10.18653/v1/ [20] Daniel E Rose and Danny Levinson. 2004. Understanding user goals in web search. In Proceedings of the 13th international conference on World Wide Web. 13–19. [21] Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics 20 (1987), 53–65. [22] Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia 6, 12 (2008), e26752. [23] Chirag Shah, Ryen W White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Snigdha Sarathi Das, Ali Montazer, Sathish Manivannan, Jennifer Neville, Xiaochuan Ni, et al. 2023. Using large language models to generate, validate, and apply user intent taxonomies. arXiv preprint arXiv:2309.13063 (2023). [24] Jingbo Shang, Xinyang Zhang, Liyuan Liu, Sha Li, and Jiawei Han. 2020. Nettaxo: Automated topic taxonomy construction from text-rich network. In Proceedings of the Web Conference 2020. 1908–1919. [25] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing. 1631–1642. [26] Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. One Embedder, Any Task: Instruction-Finetuned Text Embeddings. https://arxiv.org/abs/2212.09741 https://arxiv.org/abs/2212.0 [27] Ah-Hwee Tan et al. 1999. Text mining: The state of the art and the challenges. In Proceedings of the PAKDD 1999 workshop on knowledge discovery from advanced databases, Vol. 8. 65–70. [28] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023. Large language models can accurately predict searcher preferences. arXiv preprint arXiv:2309.10621 (2023). [29] Ike Vayansky and Sathish AP Kumar. 2020. A review of topic modeling methods. Information Systems 94 (2020), 101582. [30] Zihan Wang, Jingbo Shang, and Ruiqi Zhong. 2023. Goal-Driven Explainable Clustering via Language Descriptions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 10626–10649. https://doi.org/10.18653/v1/2023.emnlp-main.657 https://doi.org/10.18653/v1/2023.emnlp-main.657 [31] Anuradha Welivita and Pearl Pu. 2020. A Taxonomy of Empathetic Response Intents in Human Social Conversations. In Proceedings of the 28th International Conference on Computational Linguistics. 4886–4899. [32] Qingkai Zeng, Jinfeng Lin, Wenhao Yu, Jane Cleland-Huang, and Meng Jiang. 2021. Enhancing taxonomy completion with concept generation via fusing relational representations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2104–2113. [33] Chao Zhang, Fangbo Tao, Xiusi Chen, Jiaming Shen, Meng Jiang, Brian Sadler, Michelle Vanni, and Jiawei Han. 2018. Taxogen: Unsupervised topic taxonomy construction by adaptive term embedding and clustering. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2701–2709. [34] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can Large Language Models Transform Computational Social Science? arXiv preprint arXiv:2305.03514 (2023). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv Authors:
(1) Mengting Wan, Microsoft Corporation and Microsoft Corporation;
(2) Tara Safavi (Corresponding authors), Microsoft Corporation;
(3) Sujay Kumar Jauhar, Microsoft Corporation;
(4) Yujin Kim, Microsoft Corporation;
(5) Scott Counts, Microsoft Corporation;
(6) Jennifer Neville, Microsoft Corporation;
(7) Siddharth Suri, Microsoft Corporation;
(8) Chirag Shah, University of Washington and Work done while working at Microsoft;
(9) Ryen W. White, Microsoft Corporation;
(10) Longqi Yang, Microsoft Corporation;
(11) Reid Andersen, Microsoft Corporation;
(12) Georg Buscher, Microsoft Corporation;
(13) Dhruv Joshi, Microsoft Corporation;
(14) Nagu Rangan, Microsoft Corporation. Authors: Authors: (1) Mengting Wan, Microsoft Corporation and Microsoft Corporation; (2) Tara Safavi (Corresponding authors), Microsoft Corporation; (3) Sujay Kumar Jauhar, Microsoft Corporation; (4) Yujin Kim, Microsoft Corporation; (5) Scott Counts, Microsoft Corporation; (6) Jennifer Neville, Microsoft Corporation; (7) Siddharth Suri, Microsoft Corporation; (8) Chirag Shah, University of Washington and Work done while working at Microsoft; (9) Ryen W. White, Microsoft Corporation; (10) Longqi Yang, Microsoft Corporation; (11) Reid Andersen, Microsoft Corporation; (12) Georg Buscher, Microsoft Corporation; (13) Dhruv Joshi, Microsoft Corporation; (14) Nagu Rangan, Microsoft Corporation.

TnT-LLM: Democratizing Text Mining with Automated Taxonomy and Scalable Classification

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related works 2 Related works 3 Methodology and 3.1 Causal language model as a classification model 3 Methodology and 3.1 Causal language model as a classification model 3.2 Functional token 3.2 Functional token 3.3 Dataset collection 3.3 Dataset collection 3.4 Model development and training 3.4 Model development and training 4 Experiments and 4.1 Android function calls 4 Experiments and 4.1 Android function calls 4.2 Extension to Vehicle, Yelp, and DoorDash function sets 4.2 Extension to Vehicle, Yelp, and DoorDash function sets 4.3 Full and partial training datasets and 4.4 Full training and LoRA training 4.3 Full and partial training datasets and 4.4 Full training and LoRA training 4.5 Parallel and nested function call and 4.6 Weighted loss function for special tokens 4.5 Parallel and nested function call and 4.6 Weighted loss function for special tokens 5 Discussion and future works and References 5 Discussion and future works and References Appendix Appendix A.1 Android function examples A.1 Android function examples A.2 Vehicle function examples A.2 Vehicle function examples 5 Discussion and future works Our current training initiative proves that any specific function can be encapsulated into a newly coined term, functional token, a novel token type seamlessly integrated into both the tokenizer and the model. This model, through a cost-effective training process amounting to merely two cents, facilitates the deployment of AI agents characterized by their remarkably low latency and high accuracy. The potential impacts of our research are extensive. For application developers, including those at DoorDash and Yelp, our model paves the way for training on application-specific scenarios. Developers can pinpoint the APIs most utilized by their audience, transform these into functional tokens for the Octopus model, and proceed with deployment. This strategy has the capacity to fully automate app workflows, emulating functionalities akin to Apple’s Siri, albeit with significantly enhanced response speeds and accuracy. Furthermore, the model’s application within operating systems of PCs, smartphones, and wearable technology presents another exciting avenue. Software developers could train minor LoRAs specific to the operating system. By accumulating multiple LoRAs, the model facilitates efficient function calling across diverse system components. For instance, incorporating this model into the Android ecosystem would enable Yelp and DoorDash developers to train distinct LoRAs, thus rendering the model operational on mobile platforms as well. Looking ahead, we aim to develop a model dedicated to on-device reasoning. Our ambitions are dual-pronged: firstly, to achieve notable speed enhancements for cloud deployments, vastly outpacing GPT-4 in speed metrics. Secondly, to support local deployment, offering a valuable solution for users mindful of privacy or operational costs. This dual deployment strategy not only extends the model’s utility across cloud and local environments, but also caters to user preferences for either speed and efficiency or privacy and cost savings. References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [3] Erik Brynjolfsson, Danielle Li, and Lindsey R Raymond. Generative ai at work. Technical report, National Bureau of Economic Research, 2023. [4] Rich Caruana. Multitask learning. Machine learning, 28:41–75, 1997. [5] Imran Chaudhri. Humane ai, 2024. URL https://humane.com/. Accessed on March 31, 2024. [6] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024. [7] Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. Towards next-generation intelligent assistants leveraging llm techniques. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5792–5793, 2023. [8] Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large-scale api calls. arXiv preprint arXiv:2402.04253, 2024. [9] Div Garg. Multion ai, 2024. URL https://www.multion.ai/. Accessed on March 31, 2024. [10] Gemma Team, Google DeepMind. Gemma: Open models based on gemini research and technology, 2023. URL https://goo.gle/GemmaReport. [11] Allyson I Hauptman, Beau G Schelble, Nathan J McNeese, and Kapil Chalil Madathil. Adapt and overcome: Perceptions of adaptive autonomous agents for human-ai teaming. Computers in Human Behavior, 138: 107451, 2023. [12] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. [13] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023. [14] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. [15] Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983, 2023. [16] Zhanming Jie, Trung Quoc Luong, Xinbo Zhang, Xiaoran Jin, and Hang Li. Design of chain-of-thought in math problem solving, 2023. [17] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledgeintensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020. [18] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. [19] Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. A survey on retrieval-augmented text generation. arXiv preprint arXiv:2202.01110, 2022. [20] Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303.16434, 2023. [21] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. [22] Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499, 2023. [23] Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402.14905, 2024. [24] llama.cpp team. llama-cpp. Software available at https://github.com/ggerganov/llama.cpp, 2023. Accessed on March 31, 2024. [25] David Luan. Adept ai, 2024. URL https://www.adept.ai/. Accessed on March 31, 2024. [26] Jesse Lyu. Rabbit r1, 2024. URL https://www.rabbit.tech/. Accessed on March 31, 2024. [27] Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. Generation-augmented retrieval for open-domain question answering. arXiv preprint arXiv:2009.08553, 2020. [28] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. [29] Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014, 2023. [30] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023. [31] Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, and Nathan Cooper. Stable code 3b, 2023. URL https://huggingface.co/stabilityai/stable-code-3b. [32] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023. [33] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI blog, 2018. [34] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [35] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331, 2023. [36] Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. Tptu: Task planning and tool usage of large language model-based ai agents. arXiv preprint arXiv:2308.03427, 2023. [37] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024. [38] Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. Small llms are weak tool learners: A multi-llm agent. arXiv preprint arXiv:2401.07324, 2024. [39] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024. [40] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023. [41] Benjamin Spector and Chris Re. Accelerating llm inference with staged speculative decoding. arXiv preprint arXiv:2308.04623, 2023. [42] Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Damon Mosk-Aoyama, Kurt Keutzer, Jiantao Jiao, and Jian Zhang. Nexusraven: a commercially-permissive language model for function calling. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. [43] Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314, 2023. [44] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023. [45] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [46] MLC team. Mlc-llm. Software available at https://github.com/mlc-ai/mlc-llm, 2023. Accessed on March 31, 2024. [47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [48] Trelis. Llama-2-7b-chat-hf-function-calling-v3: A model fine-tuned for function calling. Hugging Face Model Repository, 2023. URL https://huggingface.co/Trelis/ Llama-2-7b-chat-hf-function-calling-v3. Accessed on March 31, 2024. [49] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024. [50] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023. [51] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. [52] Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Empowering llm to use smartphone for intelligent task automation. arXiv preprint arXiv:2308.15272, 2023. [53] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023. [54] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023. [55] Daliang Xu, Wangsong Yin, Xin Jin, Ying Zhang, Shiyun Wei, Mengwei Xu, and Xuanzhe Liu. Llmcad: Fast and scalable on-device large language model inference. arXiv preprint arXiv:2309.04255, 2023. [56] Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023. [57] Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211, 2024. [58] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv Authors:
(1) Wei Chen, Stanford University, with equal contribution and a corresponding author {weichen6}@stanford.edu;
(2) Zhiyuan Li, Stanford University and a corresponding author {zhiyuan8}@stanford.edu. Authors: Authors: (1) Wei Chen, Stanford University, with equal contribution and a corresponding author {weichen6}@stanford.edu; (2) Zhiyuan Li, Stanford University and a corresponding author {zhiyuan8}@stanford.edu.

The Future of Octopus v2: What Does it Entail?

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Data 2.1 Data Sources 2.1 Data Sources 2.2 SS and SI Categories 2.2 SS and SI Categories 3 Methods 3.1 Lexicon Creation and Expansion 3.1 Lexicon Creation and Expansion 3.2 Annotations 3.2 Annotations 3.3 System Description 3.3 System Description 4 Results 4.1 Demographics and 4.2 System Performance 4.1 Demographics and 4.2 System Performance 5 Discussion 5 Discussion 5.1 Limitations 5.1 Limitations 6 Conclusion, Reproducibility, Funding, Acknowledgments, Author Contributions, and References 6 Conclusion, Reproducibility, Funding, Acknowledgments, Author Contributions, and References SUPPLEMENTARY SUPPLEMENTARY Guidelines for Annotating Social Support and Social Isolation in Clinical Notes Guidelines for Annotating Social Support and Social Isolation in Clinical Notes Other Supervised Models Other Supervised Models OTHER SUPERVISED MODELS We performed a sentence-level classification using SVM and BERT. The entity-level annotation were converted to sentence-level. SVM: Initially, we used two VSM methods namely term frequency-inverse document frequency (TF-IDF) and word2vec [1] to convert the clinical notes into vectors. However, the word2vecbased system performed better than the TF-IDF-based system and we reported the performance of word2vec-based system next. We used the same embeddings that we used to identify similar words. We used Eq 1 to convert the word vectors to a single vector. The embeddings settings are as follows: vector size: 300, minimum word count: 3, and window size: 10. We used the linear kernel in the SVM classifier. SVM References Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.


Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,


Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020.


Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Table S10: Erroneous examples from Rule-based NLP systems (RBS) and LLM-based NLP systems. Table S10: This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv Authors:
(1) Braja Gopal Patra, Weill Cornell Medicine, New York, NY, USA and co-first authors;
(2) Lauren A. Lepow, Icahn School of Medicine at Mount Sinai, New York, NY, USA and co-first authors;
(3) Praneet Kasi Reddy Jagadeesh Kumar. Weill Cornell Medicine, New York, NY, USA;
(4) Veer Vekaria, Weill Cornell Medicine, New York, NY, USA;
(5) Mohit Manoj Sharma, Weill Cornell Medicine, New York, NY, USA;
(6) Prakash Adekkanattu, Weill Cornell Medicine, New York, NY, USA;
(7) Brian Fennessy, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(8) Gavin Hynes, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(9) Isotta Landi, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(10) Jorge A. Sanchez-Ruiz, Mayo Clinic, Rochester, MN, USA;
(11) Euijung Ryu, Mayo Clinic, Rochester, MN, USA;
(12) Joanna M. Biernacka, Mayo Clinic, Rochester, MN, USA;
(13) Girish N. Nadkarni, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(14) Ardesheer Talati, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA;
(15) Myrna Weissman, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA;
(16) Mark Olfson, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA, New York State Psychiatric Institute, New York, NY, USA, and Columbia University Irving Medical Center, New York, NY, USA;
(17) J. John Mann, Columbia University Irving Medical Center, New York, NY, USA;
(18) Alexander W. Charney, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(19) Jyotishman Pathak, Weill Cornell Medicine, New York, NY, USA. Authors: Authors: (1) Braja Gopal Patra, Weill Cornell Medicine, New York, NY, USA and co-first authors; (2) Lauren A. Lepow, Icahn School of Medicine at Mount Sinai, New York, NY, USA and co-first authors; (3) Praneet Kasi Reddy Jagadeesh Kumar. Weill Cornell Medicine, New York, NY, USA; (4) Veer Vekaria, Weill Cornell Medicine, New York, NY, USA; (5) Mohit Manoj Sharma, Weill Cornell Medicine, New York, NY, USA; (6) Prakash Adekkanattu, Weill Cornell Medicine, New York, NY, USA; (7) Brian Fennessy, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (8) Gavin Hynes, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (9) Isotta Landi, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (10) Jorge A. Sanchez-Ruiz, Mayo Clinic, Rochester, MN, USA; (11) Euijung Ryu, Mayo Clinic, Rochester, MN, USA; (12) Joanna M. Biernacka, Mayo Clinic, Rochester, MN, USA; (13) Girish N. Nadkarni, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (14) Ardesheer Talati, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA; (15) Myrna Weissman, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA; (16) Mark Olfson, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA, New York State Psychiatric Institute, New York, NY, USA, and Columbia University Irving Medical Center, New York, NY, USA; (17) J. John Mann, Columbia University Irving Medical Center, New York, NY, USA; (18) Alexander W. Charney, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (19) Jyotishman Pathak, Weill Cornell Medicine, New York, NY, USA.

Supervised Models for Clinical Text: Evaluating SVM and BERT Performance

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Data 2.1 Data Sources 2.1 Data Sources 2.2 SS and SI Categories 2.2 SS and SI Categories 3 Methods 3.1 Lexicon Creation and Expansion 3.1 Lexicon Creation and Expansion 3.2 Annotations 3.2 Annotations 3.3 System Description 3.3 System Description 4 Results 4.1 Demographics and 4.2 System Performance 4.1 Demographics and 4.2 System Performance 5 Discussion 5 Discussion 5.1 Limitations 5.1 Limitations 6 Conclusion, Reproducibility, Funding, Acknowledgments, Author Contributions, and References 6 Conclusion, Reproducibility, Funding, Acknowledgments, Author Contributions, and References SUPPLEMENTARY SUPPLEMENTARY Guidelines for Annotating Social Support and Social Isolation in Clinical Notes Guidelines for Annotating Social Support and Social Isolation in Clinical Notes Other Supervised Models Other Supervised Models 6 CONCLUSION We offer two open-source NLP systems with different approaches, as well as a manual annotation guideline for identifying SS and SI. The rule-based approach and the LLM approach each have strengths and limitations in performing this challenging task of creating a portable system to identify fine and coarse-grained categories of SS and SI in psychiatric patients. REPRODUCIBILITY All code for rule- and LLM-based systems are available on GitHub. Link: https://github.com/CornellMHILab/Social Support Social Isolation Extraction. Link The annotation guidelines and lexicons are available in the supplementary files. FUNDING This study was funded in part by grants from the National Institutes of Health (R01 MH119177, R01 MH121907, and R01 MH121922). ACKNOWLEDGEMENTS This work was supported in part through the computational and data resources and staff expertise provided by Scientific Computing and Data at the Icahn School of Medicine at Mount Sinai Health System and supported by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences. Research reported in this publication was also supported by the Office of Research Infrastructure of the National Institutes of Health under award number S10OD026880 and S10OD030463. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. AUTHOR CONTRIBUTIONS BGP and LAL worked on planning, data collection, data annotation, data extraction, software development, writing, and editing. PR worked on LLM implementation and editing. VV helped in data annotation, writing, and editing. MMS, BF, and GH helped in data annotation. JP and PA helped in planning, advising, and editing. MW, JJM, and MO helped in initial category identification, lexicon development, advising, and editing. JASR, ER, JMB, IL, GNN, AT, and AWC helped in advising and editing. JP and AWC provided resources and funding. REFERENCES [1] World Health Organization. Social determinants of health. https://www.who.int/ health-topics/social-determinants-of-health#tab=tab_1. Online; accessed 29 October 2023. [2] Braja G Patra, Mohit M Sharma, Veer Vekaria, Prakash Adekkanattu, Olga V Patterson, Benjamin Glicksberg, Lauren A Lepow, Euijung Ryu, Joanna M Biernacka, Al’ona Furmanchuk, et al. Extracting social determinants of health from electronic health records using natural language processing: a systematic review. Journal of the American Medical Informatics Association, 28(12):2716–2727, 2021. [3] Maria Polikandrioti. Perceived social isolation in heart failure. The Journal of Innovations in Cardiac Rhythm Management, 13(6):5041, 2022. [4] Wengen Zhu, Yuzhong Wu, Yuanyuan Zhou, Weihao Liang, Ruicong Xue, Zexuan Wu, Dexi Wu, Jiangui He, Yugang Dong, and Chen Liu. Living alone and clinical outcomes in patients with heart failure with preserved ejection fraction. Psychosomatic Medicine, 83(5):470–476, 2021. [5] Tin-Kwang Lin, Bo-Cheng Hsu, Yi-Da Li, Chi-Hsien Chen, Jiunn-Wen Lin, Chen-Yu Chien, and Chia-Ying Weng. The impact of sources of perceived social support on readmissions in patients with heart failure. Journal of Psychosomatic Research, 154:110723, 2022. [6] Maddi Olano-Lizarraga, Sara Wallstr¨om, Jes´us Mart´ın-Mart´ın, and Axel Wolf. Causes, experiences and consequences of the impact of chronic heart failure on the person´ s social dimension: a scoping review. Health & Social Care in the Community, 30(4):e842–e858, 2022. [7] Crystal W Cen´e, Theresa M Beckie, Mario Sims, Shakira F Suglia, Brooke Aggarwal, Nathalie Moise, Monik C Jim´enez, Bamba Gaye, and Louise D McCullough. Effects of objective and perceived social isolation on cardiovascular and brain health: A scientific statement from the american heart association. Journal of the American Heart Association, 11(16):e026493, 2022. [8] Ryo Naito, Martin McKee, Darryl Leong, Shrikant Bangdiwala, Sumathy Rangarajan, Shofiqul Islam, and Salim Yusuf. Social isolation as a risk factor for all-cause mortality: Systematic review and meta-analysis of cohort studies. PloS one, 18(1):e0280308, 2023. [9] US Surgeon General. Our epidemic of loneliness and isolation. The US Surgeon General’s Advisory on the Healing Effects of Social Connection and Community 2023, 2023. [10] John T Cacioppo and Stephanie Cacioppo. Social relationships and health: The toxic effects of perceived social isolation. Social and personality psychology compass, 8(2):58–72, 2014. [11] Myung Hyun Kim, Ji Hyun An, Hye Rin Lee, Seo Hyun Jeong, Soo Jin Hwang, and Jin Pyo Hong. Social isolation, loneliness and their relationships with mental health status in south korea. Psychiatry Investigation, 18(7):652, 2021. [12] Priya J Wickramaratne, Tenzin Yangchen, Lauren Lepow, Braja G Patra, Benjamin S Glicksberg, Ardesheer Talati, Prakash Adekkanattu, Euijung Ryu, Joanna M Biernacka, Alexander Charney, et al. Social connectedness as a determinant of mental health: A scoping review. PLOS ONE, 17:1–30, 2022. 2 [13] Sara J Czaja, Jerad H Moxley, and Wendy A Rogers. Social support, isolation, loneliness, and health among older adults in the prism randomized controlled trial. Frontiers in Psychology, 12:728658, 2021. [14] M Lamblin, C Murawski, S Whittle, and A Fornito. Social connectedness, mental health and the adolescent brain. Neuroscience & Biobehavioral Reviews, 80:57–68, 2017. [15] Carrie Purcell, Susie Smillie, Michele Hilton Boon, Sharon A Simpson, and Rod S Taylor. Effectiveness of social network interventions to support cardiac rehabilitation and secondary prevention in the management of people with heart disease. Cochrane Database of Systematic Reviews, 2021(1), 2021. [16] Rosanne Freak-Poli, Joanne Ryan, Johannes T Neumann, Andrew Tonkin, Christopher M Reid, Robyn L Woods, Mark Nelson, Nigel Stocks, Michael Berk, John J McNeil, et al. Social isolation, social support and loneliness as predictors of cardiovascular disease incidence and mortality. BMC geriatrics, 21(1):1–14, 2021. [17] M Robin DiMatteo. Social support and patient adherence to medical treatment: a metaanalysis. Health psychology, 23(2):207, 2004. [18] M Robin DiMatteo. Variations in patients’ adherence to medical recommendations: a quantitative review of 50 years of research. Medical care, pages 200–209, 2004. [19] Theodore F Robles and Janice K Kiecolt-Glaser. The physiology of marriage: Pathways to health. Physiology & behavior, 79(3):409–416, 2003. [20] Bert N Uchino. Social support and health: a review of physiological processes potentially underlying links to disease outcomes. Journal of behavioral medicine, 29:377–387, 2006. [21] Camelia E Hostinar, Regina M Sullivan, and Megan R Gunnar. Psychobiological mechanisms underlying the social buffering of the hypothalamic–pituitary–adrenocortical axis: A review of animal models and human studies across development. Psychological bulletin, 140(1):256, 2014 [22] Julianne Holt-Lunstad. The potential public health relevance of social isolation and loneliness: Prevalence, epidemiology, and risk factors. Public Policy & Aging Report, 27(4):127–130, 2017. [23] Elisa M Trucco, Nicole M Fava, Michelle G Villar, Mari Kumar, and Matthew T Sutherland. Social isolation during the covid-19 pandemic impacts the link between child abuse and adolescent internalizing problems. Journal of youth and adolescence, pages 1–12, 2023. [24] Hena Thakur, Morgan Stutts, Jae Wan Choi, Jeff R Temple, and Joseph R Cohen. Adolescent loneliness during the covid-19 pandemic: the role of pre-pandemic risk factors. Child Indicators Research, 16(2):617–639, 2023. [25] Hannah P Truong, Alina A Luke, Gmerice Hammond, Rishi K Wadhera, Mat Reidhead, and Karen E Joynt Maddox. Utilization of social determinants of health icd-10 z-codes among hospitalized patients in the united states, 2016-2017. Medical care, 58(12):1037, 2020. [26] Vivienne J Zhu, Leslie A Lenert, Brian E Bunnell, Jihad S Obeid, Melanie Jefferson, and Chanita Hughes Halbert. Automatically identifying social isolation from clinical narratives for patients with prostate cancer. BMC medical informatics and decision making, 19(1):1–9, 2019. [27] Hadi Kharrazi, Laura J Anzaldi, Leilani Hernandez, Ashwini Davison, Cynthia M Boyd, Bruce Leff, Joe Kimura, and Jonathan P Weiner. The value of unstructured electronic health record data in geriatric syndrome case identification. Journal of the American Geriatrics Society, 66(8):1499–1507, 2018. [28] Jeffrey L Greenwald, Patrick R Cronin, Victoria Carballo, Goodarz Danaei, and Garry Choy. A novel model for predicting rehospitalization risk incorporating physical function, cognitive status, and psychosocial support using natural language processing. Medical Care, 55(3):261– 266, 2017. [29] Sunil Bhatt, P Connor Johnson, Netana H Markovitz, Tamryn Gray, Ryan D Nipp, Nneka Ufere, Julia Rice, Matthew J Reynolds, Mitchell W Lavoie, Madison A Clay, et al. The use of natural language processing to assess social support in patients with advanced cancer. The Oncologist, 28(2):165–171, 2023. [30] Amol S Navathe, Feiran Zhong, Victor J Lei, Frank Y Chang, Margarita Sordo, Maxim Topaz, Shamkant B Navathe, Roberto A Rocha, and Li Zhou. Hospital readmission and social risk factors identified from physician notes. Health services research, 53(2):1110–1136, 2018. [31] Liqin Wang, Joshua Lakin, Clay Riley, Zfania Korach, Laura N Frain, and Li Zhou. Disease trajectories and end-of-life care for dementias: latent topic modeling and trend analysis using clinical notes. In AMIA Annual Symposium Proceedings, volume 2018, page 1056. American Medical Informatics Association, 2018. [32] Marco Guevara, Shan Chen, Spencer Thomas, Tafadzwa L Chaunzwa, Idalid Franco, Benjamin H Kann, Shalini Moningi, Jack M Qian, Madeleine Goldstein, Susan Harper, et al. Large language models to identify social determinants of health in electronic health records. NPJ digital medicine, 7(1):6, 2024. [33] Kevin Lybarger, Oliver J Bear Don’t Walk IV, Meliha Yetisgen, and Ozlem Uzuner. Advance- ¨ ments in extracting social determinants of health information from narrative text, 2023. [34] Manuel Barrera Jr. Distinctions between social support concepts, measures, and models. American journal of community psychology, 14(4):413–445, 1986. [35] Lixia Ge, Chun Wei Yap, Reuben Ong, and Bee Hoon Heng. Social isolation, loneliness and their relationships with depressive symptoms: a population-based study. PloS one, 12(8):e0182145, 2017. [36] Christopher M Masi, Hsi-Yuan Chen, Louise C Hawkley, and John T Cacioppo. A metaanalysis of interventions to reduce loneliness. Personality and social psychology review, 15(3):219–266, 2011. [37] Catherine A Heaney and Barbara A Israel. Social networks and social support. Health behavior and health education: Theory, research, and practice, 4:189–210, 2008. [38] James S House. Social support and social structure. Sociological forum, 2(1):135–146, 1987. [39] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013. [40] NLPLAB. Brat rapid annotation tool. https://brat.nlplab.org. [41] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrialstrength natural language processing in python. Zenodo, 2020. [42] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. [43] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [44] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. [45] Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. arXiv preprint arXiv:2303.11315, 2023. [46] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than incontext learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv Authors:
(1) Braja Gopal Patra, Weill Cornell Medicine, New York, NY, USA and co-first authors;
(2) Lauren A. Lepow, Icahn School of Medicine at Mount Sinai, New York, NY, USA and co-first authors;
(3) Praneet Kasi Reddy Jagadeesh Kumar. Weill Cornell Medicine, New York, NY, USA;
(4) Veer Vekaria, Weill Cornell Medicine, New York, NY, USA;
(5) Mohit Manoj Sharma, Weill Cornell Medicine, New York, NY, USA;
(6) Prakash Adekkanattu, Weill Cornell Medicine, New York, NY, USA;
(7) Brian Fennessy, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(8) Gavin Hynes, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(9) Isotta Landi, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(10) Jorge A. Sanchez-Ruiz, Mayo Clinic, Rochester, MN, USA;
(11) Euijung Ryu, Mayo Clinic, Rochester, MN, USA;
(12) Joanna M. Biernacka, Mayo Clinic, Rochester, MN, USA;
(13) Girish N. Nadkarni, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(14) Ardesheer Talati, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA;
(15) Myrna Weissman, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA;
(16) Mark Olfson, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA, New York State Psychiatric Institute, New York, NY, USA, and Columbia University Irving Medical Center, New York, NY, USA;
(17) J. John Mann, Columbia University Irving Medical Center, New York, NY, USA;
(18) Alexander W. Charney, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(19) Jyotishman Pathak, Weill Cornell Medicine, New York, NY, USA. Authors: Authors: (1) Braja Gopal Patra, Weill Cornell Medicine, New York, NY, USA and co-first authors; (2) Lauren A. Lepow, Icahn School of Medicine at Mount Sinai, New York, NY, USA and co-first authors; (3) Praneet Kasi Reddy Jagadeesh Kumar. Weill Cornell Medicine, New York, NY, USA; (4) Veer Vekaria, Weill Cornell Medicine, New York, NY, USA; (5) Mohit Manoj Sharma, Weill Cornell Medicine, New York, NY, USA; (6) Prakash Adekkanattu, Weill Cornell Medicine, New York, NY, USA; (7) Brian Fennessy, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (8) Gavin Hynes, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (9) Isotta Landi, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (10) Jorge A. Sanchez-Ruiz, Mayo Clinic, Rochester, MN, USA; (11) Euijung Ryu, Mayo Clinic, Rochester, MN, USA; (12) Joanna M. Biernacka, Mayo Clinic, Rochester, MN, USA; (13) Girish N. Nadkarni, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (14) Ardesheer Talati, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA; (15) Myrna Weissman, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA; (16) Mark Olfson, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA, New York State Psychiatric Institute, New York, NY, USA, and Columbia University Irving Medical Center, New York, NY, USA; (17) J. John Mann, Columbia University Irving Medical Center, New York, NY, USA; (18) Alexander W. Charney, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (19) Jyotishman Pathak, Weill Cornell Medicine, New York, NY, USA.

Open-Source NLP Systems for Identifying Social Support and Isolation in Psychiatric Notes

Turning AI Into Better Thinkers With Pointer-Based Memory

The Rise of Tankies: A Data-Driven Look at Far-Left Extremism

Language Models and the Illusion of Understanding

Authors:
(1) Raphaël Millière, Department of Philosophy, Macquarie University (raphael.milliere@mq.edu.eu);
(2) Cameron Buckner, Department of Philosophy, University of Houston (cjbuckner@uh.edu). Table of Links Abstract and 1 Introduction 2. A primer on LLMs 2.1. Historical foundations 2.2. Transformer-based LLMs 3. Interface with classic philosophical issues 3.1. Compositionality 3.2. Nativism and language acquisition 3.3. Language understanding and grounding 3.4. World models 3.5. Transmission of cultural knowledge and linguistic scaffolding 4. Conclusion, Glossary, and References 2.1. Historical foundations The origins of large language models can be traced back to the inception of AI research. The early history of natural language processing (NLP) was marked by a schism between two competing paradigms: the symbolic and the stochastic approaches. A major influence on the symbolic paradigm in NLP was Noam Chomsky’s transformational-generative grammar (Chomsky 1957), which posited that the syntax of natural languages could be captured by a set of formal rules that generated well-formed sentences. Chomsky’s work laid the foundation for the development of rule-based syntactic parsers, which leverage linguistic theory to decompose sentences into their constituent parts. Early conversational NLP systems, such as Winograd’s SHRDLU (Winograd 1971), required syntactic parsers with a complex set of ad hoc rules to process user input. In parallel, the stochastic paradigm was pioneered by researchers such as mathematician Warren Weaver, who was influenced by Claude Shannon’s information theory. In a memorandum written in 1949, Weaver proposed the use of computers for machine translation employing statistical techniques (Weaver 1955). This work paved the way for the development of statistical language models, such as n-gram models, which estimate the likelihood of word sequences based on observed frequencies of word combinations in a corpus (Jelinek 1998). Initially, however, the stochastic paradigm was lagging behind symbolic approaches to NLP, showing only modest success in toy models with limited applications. Another important theoretical stepping stone on the road to modern language models is the so-called distributional hypothesis, first proposed by the linguist Zellig Harris in the 1950s (Harris 1954). This idea was grounded in the structuralist view of language, which posits that linguistic units acquire meaning through their patterns of co-occurrence with other units in the system. Harris specifically suggested that the meaning of a word could be inferred by examining its distributional properties, or the contexts in which it occurs. Firth (1957) aptly summarized this hypothesis with the slogan “You shall know a word by the company it keeps,” acknowledging the influence of Wittgenstein (1953)’s conception of meaning-as-use to highlight the importance of context in understanding linguistic meaning. As research on the distributional hypothesis progressed, scholars began exploring the possibility of representing word meanings as vectors in a multidimensional space 1. Early empirical work in this area stemmed from psychology and examined the meaning of words along various dimensions, such as valence and potency (Osgood 1952). While this work introduced the idea of representing meaning in a multidimensional vector space, it relied on explicit participant ratings about word connotations along different scales (e.g., good–bad), rather than analyzing the distributional properties of a linguistic corpus. Subsequent research in information retrieval combined vector-based representations with a data-driven approach, developing automated techniques for representing documents and words as vectors in high-dimensional vector spaces (Salton et al. 1975). After decades of experimental research, these ideas eventually reached maturity with the development of word embedding models using artificial neural networks (Bengio et al. 2000). These models are based on the insight that the distributional properties of words can be learned by training a neural network to predict a word’s context given the word itself, or vice versa. Unlike previous statistical methods such as n-gram models, word embedding models encode words into dense, low-dimensional vector representations (Fig. 1). The resulting vector space drastically reduces the dimensionality of linguistic data while preserving information about meaningful linguistic relationships beyond simple co-occurrence statistics. Notably, many semantic and syntactic relationships between words are reflected in linear substructures within the vector space of word embedding models. For example, Word2Vec (Mikolov et al. 2013) demonstrated that word embeddings can capture both semantic and syntactic regularities, as evidenced by the ability to solve word analogy tasks through simple vector arithmetic that reveal the latent linguistic structure encoded in the vector space (e.g., 𝑘𝑖𝑛𝑔 + 𝑤𝑜𝑚𝑎𝑛 − 𝑚𝑎𝑛 ≈ 𝑞𝑢𝑒𝑒𝑛, or 𝑤𝑎𝑙𝑘𝑖𝑛𝑔 + 𝑠𝑤𝑎𝑚 − 𝑤𝑎𝑙𝑘𝑒𝑑 ≈ 𝑠𝑤𝑖𝑚𝑚𝑖𝑛𝑔). The development of word embedding models marked a turning point in the history of NLP, providing a powerful and efficient means of representing linguistic units in a continuous vector space based on their statistical distribution in a large corpus. However, these models have several significant limitations. First, they are not capable of capturing polysemy and homonymy, because they assign a single or “static” embedding to each word type, which cannot account for changes in meaning based on context; for example, “bank” is assigned a unique embedding regardless of whether it refers to the side of a river or the financial institution. Second, they rely on “shallow” artificial neural network architectures with a single hidden layer, which limits their ability to model complex relationships between words. Finally, being designed to represent language at the level of individual words, they are not well-suited to model complex linguistic expression, such as phrases, sentences, and paragraphs. While it is possible to represent a sentence as a vector by averaging the embeddings of every word in the sentence, this is a very poor way of representing sentence-level meaning, as it loses information about compositional structure reflected in word order. In other words, word embedding models merely treat language as a “bag of words”; for example, “a law book” and “a book law” are treated identically as the unordered set {'a','book','law'}. The shortcomings of shallow word embedding models were addressed with the introduction of “deep” language models, going back to recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997) and the gated recurrent unit (GRU) (Cho et al. 2014). These deep neural network architectures incorporate a memory-like mechanism, allowing them to remember and process sequences of inputs over time, rather than individual, isolated words. Despite this advantage over word embedding models, they suffer from their own limitations: they are slow to train and struggle with long sequences of text. These issues were addressed with the introduction of the Transformer architecture by Vaswani et al. (2017), which laid the groundwork for modern LLMs. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Raphaël Millière, Department of Philosophy, Macquarie University (raphael.milliere@mq.edu.eu); (2) Cameron Buckner, Department of Philosophy, University of Houston (cjbuckner@uh.edu). Authors: Authors: (1) Raphaël Millière, Department of Philosophy, Macquarie University (raphael.milliere@mq.edu.eu); (2) Cameron Buckner, Department of Philosophy, University of Houston (cjbuckner@uh.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. A primer on LLMs 2.1. Historical foundations 2.1. Historical foundations 2.2. Transformer-based LLMs 2.2. Transformer-based LLMs 3. Interface with classic philosophical issues 3. Interface with classic philosophical issues 3.1. Compositionality 3.1. Compositionality 3.2. Nativism and language acquisition 3.2. Nativism and language acquisition 3.3. Language understanding and grounding 3.3. Language understanding and grounding 3.4. World models 3.4. World models 3.5. Transmission of cultural knowledge and linguistic scaffolding 3.5. Transmission of cultural knowledge and linguistic scaffolding 4. Conclusion, Glossary, and References 4. Conclusion, Glossary, and References 2.1. Historical foundations The origins of large language models can be traced back to the inception of AI research. The early history of natural language processing (NLP) was marked by a schism between two competing paradigms: the symbolic and the stochastic approaches. A major influence on the symbolic paradigm in NLP was Noam Chomsky’s transformational-generative grammar (Chomsky 1957), which posited that the syntax of natural languages could be captured by a set of formal rules that generated well-formed sentences. Chomsky’s work laid the foundation for the development of rule-based syntactic parsers, which leverage linguistic theory to decompose sentences into their constituent parts. Early conversational NLP systems, such as Winograd’s SHRDLU (Winograd 1971), required syntactic parsers with a complex set of ad hoc rules to process user input. In parallel, the stochastic paradigm was pioneered by researchers such as mathematician Warren Weaver, who was influenced by Claude Shannon’s information theory. In a memorandum written in 1949, Weaver proposed the use of computers for machine translation employing statistical techniques (Weaver 1955). This work paved the way for the development of statistical language models, such as n-gram models, which estimate the likelihood of word sequences based on observed frequencies of word combinations in a corpus (Jelinek 1998). Initially, however, the stochastic paradigm was lagging behind symbolic approaches to NLP, showing only modest success in toy models with limited applications. Another important theoretical stepping stone on the road to modern language models is the so-called distributional hypothesis, first proposed by the linguist Zellig Harris in the 1950s (Harris 1954). This idea was grounded in the structuralist view of language, which posits that linguistic units acquire meaning through their patterns of co-occurrence with other units in the system. Harris specifically suggested that the meaning of a word could be inferred by examining its distributional properties, or the contexts in which it occurs. Firth (1957) aptly summarized this hypothesis with the slogan “You shall know a word by the company it keeps,” acknowledging the influence of Wittgenstein (1953)’s conception of meaning-as-use to highlight the importance of context in understanding linguistic meaning. As research on the distributional hypothesis progressed, scholars began exploring the possibility of representing word meanings as vectors in a multidimensional space 1. Early empirical work in this area stemmed from psychology and examined the meaning of words along various dimensions, such as valence and potency (Osgood 1952). While this work introduced the idea of representing meaning in a multidimensional vector space, it relied on explicit participant ratings about word connotations along different scales (e.g., good–bad), rather than analyzing the distributional properties of a linguistic corpus. Subsequent research in information retrieval combined vector-based representations with a data-driven approach, developing automated techniques for representing documents and words as vectors in high-dimensional vector spaces (Salton et al. 1975). After decades of experimental research, these ideas eventually reached maturity with the development of word embedding models using artificial neural networks (Bengio et al. 2000). These models are based on the insight that the distributional properties of words can be learned by training a neural network to predict a word’s context given the word itself, or vice versa. Unlike previous statistical methods such as n-gram models, word embedding models encode words into dense, low-dimensional vector representations (Fig. 1). The resulting vector space drastically reduces the dimensionality of linguistic data while preserving information about meaningful linguistic relationships beyond simple co-occurrence statistics. Notably, many semantic and syntactic relationships between words are reflected in linear substructures within the vector space of word embedding models. For example, Word2Vec (Mikolov et al. 2013) demonstrated that word embeddings can capture both semantic and syntactic regularities, as evidenced by the ability to solve word analogy tasks through simple vector arithmetic that reveal the latent linguistic structure encoded in the vector space (e.g., 𝑘𝑖𝑛𝑔 + 𝑤𝑜𝑚𝑎𝑛 − 𝑚𝑎𝑛 ≈ 𝑞𝑢𝑒𝑒𝑛, or 𝑤𝑎𝑙𝑘𝑖𝑛𝑔 + 𝑠𝑤𝑎𝑚 − 𝑤𝑎𝑙𝑘𝑒𝑑 ≈ 𝑠𝑤𝑖𝑚𝑚𝑖𝑛𝑔). The development of word embedding models marked a turning point in the history of NLP, providing a powerful and efficient means of representing linguistic units in a continuous vector space based on their statistical distribution in a large corpus. However, these models have several significant limitations. First, they are not capable of capturing polysemy and homonymy, because they assign a single or “static” embedding to each word type, which cannot account for changes in meaning based on context; for example, “bank” is assigned a unique embedding regardless of whether it refers to the side of a river or the financial institution. Second, they rely on “shallow” artificial neural network architectures with a single hidden layer, which limits their ability to model complex relationships between words. Finally, being designed to represent language at the level of individual words, they are not well-suited to model complex linguistic expression, such as phrases, sentences, and paragraphs. While it is possible to represent a sentence as a vector by averaging the embeddings of every word in the sentence, this is a very poor way of representing sentence-level meaning, as it loses information about compositional structure reflected in word order. In other words, word embedding models merely treat language as a “bag of words”; for example, “a law book” and “a book law” are treated identically as the unordered set {'a','book','law'}. The shortcomings of shallow word embedding models were addressed with the introduction of “deep” language models, going back to recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997) and the gated recurrent unit (GRU) (Cho et al. 2014). These deep neural network architectures incorporate a memory-like mechanism, allowing them to remember and process sequences of inputs over time, rather than individual, isolated words. Despite this advantage over word embedding models, they suffer from their own limitations: they are slow to train and struggle with long sequences of text. These issues were addressed with the introduction of the Transformer architecture by Vaswani et al. (2017), which laid the groundwork for modern LLMs. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

The Shift from Symbolic AI to Deep Learning in Natural Language Processing 

AI Simbólico ukhat Manqha Yatiqañ ukar mayjt’ayawix Procesamiento de Lenguas Naturales ukan 

El paso de la IA simbólica al aprendizaje profundo en el procesamiento del lenguaje natural

تغییر از هوش مصنوعی نمادین به یادگیری عمیق در پردازش زبان طبیعی 

Prijelaz sa simboličke umjetne inteligencije na duboko učenje u obradi prirodnog jezika 

Eltérés a szimbolikus mesterséges intelligencia felől a mélytanulás felé a természetes nyelvi feldolgozásban 

自然言語処理におけるシンボリック AI からディープラーニングへの移行

자연어 처리에서 심볼릭 AI에서 딥러닝으로의 전환

Премин од симболична вештачка интелигенција кон длабоко учење во обработка на природни јазици 

Peralihan daripada AI Simbolik kepada Pembelajaran Mendalam dalam Pemprosesan Bahasa Semulajadi 

Гузариш аз AI рамзӣ ба омӯзиши амиқ дар коркарди забони табиӣ 

Simwoliki AI-den tebigy dilleri gaýtadan işlemekde çuňňur öwrenişe geçiş 

قدرتی زبان کی پروسیسنگ میں علامتی AI سے گہری سیکھنے کی طرف شفٹ 

Stories, News & Mentions about MIKO

Categories

Trending Topics

Stories, News & Mentions about MIKO

Miko’s Stories on HackerNoon

Categories

Trending Topics