People running organisations are struggling with disconnected teams and silos that are leading to missed deadlines, lost opportunities and performance challenges. Where to start and what to do quickly that has lasting benefit? Imagine you could run a scan of your business (like an MRI) to see how the system was working and where to focus. Well, there is one… Cophi is a revolutionary AI-solution that helps you understand and improve collaboration across your business. Engagement surveys show how your organisation feels, Cophi shows how it works. Through a quick team-to-team feedback survey, Cophi scans your organisation’s workflows and builds a surprisingly simple map showing exactly where collaboration is strong and where the weakest links are. It gives you clear, practical insights into your day-to-day operational, strategic and social networks. The invisible becomes visible, and with that people can immediately take action to remove bottlenecks and improve productivity. Fast, simple and accessible for everyone. Scientifically proven. Easy to repeat and track progress. And for consultants a rapid diagnostic tool for organisational transformations. Last year we worked with customers including Meta, Charlotte Tilbury, Mindful Chef, AAK, Dirkzwager and Karo Pharma to help their teams work better together.

Cophi

BAHIA BLANCA

This paper is under CC 4.0 license. available on arxiv Authors: (1) Ryen W. White, Microsoft Research, Redmond, WA, USA. Table of Links Abstract and Taking Search to task AI Copilots Challenges Opportunities The Undiscovered Country and References 5 THE UNDISCOVERED COUNTRY AI copilots will transform how we search. Tasks are central to people’s lives and more support is needed for complex tasks in search settings. Some limited support for these tasks already exists in search engines, but copilots will expand the task frontier to make more tasks actionable and address the “last mile” in search interaction: task completion [58]. Moving forward, search providers should invest in “better together” experiences that utilize copilots plus traditional search, make these joint experiences more seamless for searchers, and add more support for their use in practice, e.g., help people to quickly understand copilot capabilities and potential and/or recommend the best modality for the current task or task stage. This includes experiences where both modalities are offered separately and can be selected by searchers and those where there is unification and the selection happens automatically based on the query and the conversation context. The foundation models that power copilots have other search-related applications, e.g., for generating and applying intent taxonomies [43] or for evaluation [19]. We must retain a continued focus on human-AI cooperation, where searchers stay in control while the degree of system support increases as needed [44], and on AI safety. Searchers need to be able to trust copilots in general but also be able to verify their answers with minimal effort. Overall, the future is bright for IR, and AI research in general, with the advent of generative AI and the copilots that build upon it. Copilots will help augment and empower searchers in their information seeking journeys. Computer science researchers and practitioners should embrace this new era of assistive agents and engage across the full spectrum of exciting practical and scientific opportunities, both within information seeking as we focused on here, and onwards into other important domains such as personal productivity [5] and scientific discovery [22]. REFERENCES [1] Eugene Agichtein, Ryen W White, Susan T Dumais, and Paul N Bennet. 2012. Search, interrupted: understanding and predicting search task continuation. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. 315-324. [2] Marcia J Bates. 1990. Where should the person stop and the information search interface start? Information Processing & Management 26, 5 (1990), 575–591. [3] Nicholas J Belkin. 1980. Anomalous states of knowledge as a basis for information retrieval. Canadian journal of information science 5, 1 (1980), 133–143. [4] Paul N Bennett, Ryen W White, Wei Chu, Susan T Dumais, Peter Bailey, Fedor Borisyuk, and Xiaoyuan Cui. 2012. Modeling the impact of short-and long-term behavior on search personalization. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. 185–194. [5] Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2022. Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools. Queue 20, 6 (2022), 35–57. [6] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). [7] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 141–159. [8] Andrei Broder. 2002. A taxonomy of web search. In ACM Sigir forum, Vol. 36. ACM New York, NY, USA, 3–10. [9] Andrei Z Broder and Preston McAfee. 2023. Delphic Costs and Benefits in Web Search: A utilitarian and historical analysis. arXiv preprint arXiv:2308.07525 (2023). [10] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). [11] Katriina Byström and Kalervo Järvelin. 1995. Task complexity affects information seeking and use. Information processing & management 31, 2 (1995), 191–213. [12] Robert Capra and Jaime Arguello. 2023. How does AI chat change search behaviors? arXiv preprint arXiv:2307.03826 (2023). [13] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) (2023). [14] Antonia Creswell and Murray Shanahan. 2022. Faithful reasoning using large language models. arXiv preprint arXiv:2208.14271 (2022). [15] Brenda Dervin. 1998. Sense-making theory and practice: An overview of user interests in knowledge seeking and use. Journal of knowledge management 2, 2 (1998), 36–46. [16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). [17] Karl Duncker and Lynne S Lees. 1945. On problem-solving. Psychological monographs 58, 5 (1945), i. [18] Brad Everman, Trevor Villwock, Dayuan Chen, Noe Soto, Oliver Zhang, and Ziliang Zong. 2023. Evaluating the Carbon Impact of Large Language Models at the Inference Stage. In 2023 IEEE International Performance, Computing, and Communications Conference (IPCCC). IEEE, 150–157. [19] Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023. Perspectives on Large Language Models for Relevance Judgment. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval. 39–50. [20] Jianfeng Gao, Chenyan Xiong, Paul Bennett, and Nick Craswell. 2023. Neural Approaches to Conversational Information Retrieval. Vol. 44. Springer Nature. [21] Ahmed Hassan Awadallah, Ryen W White, Patrick Pantel, Susan T Dumais, and Yi-Min Wang. 2014. Supporting complex search tasks. In Proceedings of the 23rd ACM international conference on conference on information and knowledge management. 829–838. [22] Tom Hope, Doug Downey, Daniel S Weld, Oren Etzioni, and Eric Horvitz. 2023. A computational inflection for scientific discovery. Commun. ACM 66, 8 (2023), 62–73. [23] Peter Ingwersen and Kalervo Järvelin. 2005. The turn: Integration of information seeking and retrieval in context. Vol. 18. Springer Science & Business Media. [24] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. Comput. Surveys 55, 12 (2023), 1–38. [25] Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. 133–142. [26] Jeonghyun Kim. 2006. Task difficulty as a predictor and indicator of web searching interaction. In CHI’06 extended abstracts on human factors in computing systems. 959–964. [27] David R Krathwohl. 2002. A revision of Bloom’s taxonomy: An overview. Theory into practice 41, 4 (2002), 212–218. [28] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–9474. [29] Yuelin Li and Nicholas J Belkin. 2008. A faceted approach to conceptualizing tasks in information seeking. Information processing & management 44, 6 (2008), 1822–1837. [30] Yuanchun Li and Oriana Riva. 2021. Glider: A reinforcement learning approach to extract UI scripts from websites. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1420–1430. [31] Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning. PMLR, 6565–6576. [32] Gary Marchionini. 2006. Exploratory search: from finding to understanding. Commun. ACM 49, 4 (2006), 41–46. [33] James Mayfield, Eugene Yang, Dawn Lawrie, Samuel Barham, Orion Weller, Marc Mason, Suraj Nair, and Scott Miller. 2023. Synthetic Cross-language Information Retrieval Training Data. arXiv preprint arXiv:2305.00331 (2023). [34] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707 (2023). [35] Marc Najork. 2023. Generative Information Retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1–1. [36] Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D Ekstrand, Adam Roegiest, Aldo Lipani, Alex Beutel, Alexandra Olteanu, Ana Lucic, AnaAndreea Stoica, et al. 2021. FACTS-IR: fairness, accountability, confidentiality, transparency, and safety in information retrieval. In ACM SIGIR Forum, Vol. 53. ACM New York, NY, USA, 20–43. [37] Soo Young Rieh, Kevyn Collins-Thompson, Preben Hansen, and Hye-Jung Lee. 2016. Towards searching as a learning process: A review of current perspectives and future directions. Journal of Information Science 42, 1 (2016), 19–34. [38] Shawon Sarkar and Chirag Shah. 2021. An integrated model of task, information needs, sources and uncertainty to design task-aware search systems. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. 83–92. [39] Reijo Savolainen. 2012. Expectancy-value beliefs and information needs as motivators for task-based information seeking. Journal of Documentation 68, 4 (2012), 492–511. [40] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 (2023). [41] Chirag Shah. 2023. Generative AI and the Future of Information Access. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (Birmingham, United Kingdom) (CIKM ’23). Association for Computing Machinery, New York, NY, USA, 3. https://doi.org/10.1145/3583780.3615317 [42] Chirag Shah, Ryen White, Paul Thomas, Bhaskar Mitra, Shawon Sarkar, and Nicholas Belkin. 2023. Taking search to task. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval. 1–13. [43] Chirag Shah, Ryen W White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Snigdha Sarathi Das, Ali Montazer, Sathish Manivannan, Jennifer Neville, Xiaochuan Ni, et al. 2023. Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies. arXiv preprint arXiv:2309.13063 (2023). [44] Ben Shneiderman. 2022. Human-centered AI. Oxford University Press. [45] Adish Singla, Ryen White, and Jeff Huang. 2010. Studying trailfinding algorithms for enhanced web search. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. 443–450. [46] Jaime Teevan, Kevyn Collins-Thompson, Ryen W White, and Susan Dumais. 2014. Slow search. Commun. ACM 57, 8 (2014), 36–38. [47] Jaime Teevan, Susan T Dumais, and Eric Horvitz. 2005. Personalizing search via automated analysis of interests and activities. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. 449–456. [48] Jaime Teevan, Meredith Ringel Morris, and Steve Bush. 2009. Discovering and using groups to improve personalized search. In Proceedings of the second acm international conference on web search and data mining. 15–24. [49] Maartje ter Hoeve, Robert Sim, Elnaz Nouri, Adam Fourney, Maarten de Rijke, and Ryen W White. 2020. Conversations with documents: An exploration of document-centered assistance. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. 43–52. [50] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023. Large language models can accurately predict searcher preferences. arXiv preprint arXiv:2309.10621 (2023). [51] Randall H Trigg. 1988. Guided tours and tabletops: Tools for communicating in a hypertext environment. ACM Transactions on Information Systems (TOIS) 6, 4 (1988), 398–414. [52] Sarah K Tyler and Jaime Teevan. 2010. Large scale query log analysis of re-finding. In Proceedings of the third ACM international conference on Web search and data mining. 191–200. [53] Pertti Vakkari. 2001. A theory of the task-based information retrieval process: A summary and generalisation of a longitudinal study. Journal of documentation 57, 1 (2001), 44–60. [54] Pertti Vakkari. 2016. Searching as learning: A systematization based on literature. Journal of Information Science 42, 1 (2016), 7–18. [55] Nicholas Vincent. 2022. The Paradox of Reuse, Language Models Edition.https://nmvg.mataroa.blog/blog/the-paradox-of-reuse-language-modelsedition/. Accessed: 2023-09-12. [56] Yu Wang, Xiao Huang, and Ryen W White. 2013. Characterizing and supporting cross-device search tasks. In Proceedings of the sixth ACM international conference on Web search and data mining. 707–716. [57] Ryen W White. 2016. Interactions with search systems. Cambridge University Press. [58] Ryen W White. 2018. Opportunities and challenges in search interaction. Commun. ACM 61,12 (2018), 36–38. [59] Ryen W White. 2018. Skill discovery in virtual assistants. Commun. ACM 61, 11 (2018), 106–113. [60] Ryen W White. 2022. Intelligent futures in task assistance. Commun. ACM 65, 11 (2022), 35–39. [61] Ryen W. White. 2023. Tasks, Copilots, and the Future of Search. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (Taipei, Taiwan) (SIGIR ’23). Association for Computing Machinery, New York, NY, USA, 5–6. https://doi.org/10.1145/3539618.3593069 [62] Ryen W White, Mikhail Bilenko, and Silviu Cucerzan. 2007. Studying the use of popular destinations to enhance web search interaction. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 159–166. [63] Ryen W White, Wei Chu, Ahmed Hassan, Xiaodong He, Yang Song, and Hongning Wang. 2013. Enhancing personalized search by mining and modeling task behavior. In Proceedings of the 22nd international conference on World Wide Web. 1411–1420. [64] Ryen W White, Adam Fourney, Allen Herring, Paul N Bennett, Nirupama Chandrasekaran, Robert Sim, Elnaz Nouri, and Mark J Encarnación. 2019. Multi-device digital assistance. Commun. ACM 62, 10 (2019), 28–31. [65] Ryen W White, Ian Ruthven, and Joemon M Jose. 2005. A study of factors affecting the utility of implicit relevance feedback. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. 35–42. [66] Ryen W White, Ian Ruthven, Joemon M Jose, and CJ Van Rijsbergen. 2005. Evaluating implicit feedback models using searcher simulations. ACM Transactions on Information Systems (TOIS) 23, 3 (2005), 325–361. [67] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. AutoGen: Enabling nextgen LLM applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155 (2023). [68] Iris Xie. 2008. Interactive information retrieval in digital environments. IGI global. [69] Jinyun Yan, Wei Chu, and Ryen W White. 2014. Cohort modeling for enhanced personalized search. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval. 505–514. [70] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. 2021. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500 (2021). [71] Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck. 2020. Generating clarifying questions for information retrieval. In Proceedings of the web conference 2020. 418–428. [72] Jieyu Zhang, Ranjay Krishna, Ahmed H Awadallah, and Chi Wang. 2023. EcoAssistant: Using LLM Assistant More Affordably and Accurately. arXiv preprint arXiv:2310.03046 (2023). [73] Yi Zhang, Sujay Kumar Jauhar, Julia Kiseleva, Ryen White, and Dan Roth. 2021. Learning to decompose and organize complex tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2726–2735. [74] Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. 2023. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint arXiv:2304.04675 (2023). [75] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019).

Navigating Complex Search Tasks with AI Copilots: The Undiscovered Country and References

Navigating Complex Search Tasks with AI Copilots: Opportunities

This paper is under CC 4.0 license. available on arxiv : Authors (1) Ryen W. White, Microsoft Research, Redmond, WA, USA. Table of Links Abstract and Taking Search to task AI Copilots Challenges Opportunities The Undiscovered Country and References 3 CHALLENGES Despite the promise of copilots, there are significant challenges that should be acknowledged and we must find ways to overcome. Those include issues with the copilot output shown in response to searcher requests, the impacts that the copilots can have on searchers, and shifts in the degree of agency that humans have in the search process that result from the introduction of copilots. • : Searchers rely a lot on the answers from copilots,but those answers can be erroneous or non-sensical. So-called“hallucination” is a well-studied problem in foundation models [24]. Copilots can hallucinate for many reasons. One of the main reasons being gaps in the training data. RAG, discussed earlier, is a way to help address this by ensuring that the copilot has access to up-to-date, relevant information at inference time to help ground its responses. Injection of knowledge from other external sources, such as knowledge graphs and Wikipedia, can also help improve the accuracy of copilot responses. An issue related to copilots surfacing misinformation is toxicity (i.e., offensive or harmful content), which can also be present in the copilot output, and must be mitigated before answers are shown to searchers. Hallucinations • : Biases in the training data, e.g., social biases and stereotypes [31], affect the output of foundation models and hence the answers provided by copilots. Synthesis of content from different sources can amplify biases in this data. As with hallucinations, this is a well-studied problem [6]. Copilots are also subject to biases from learning from their own or other AI generated content (via feedback loops); biased historical sequences lead to biased downstream models. Copilots may also amplify existing cognitive biases, such as confirmation bias, by favoring responses that are aligned with searchers’ existing beliefs and values, and by providing responses that keep searchers engaged with the copilot, regardless of the ramifications for the searcher. Biases Learning may be affected/interrupted by the use of AI copilots since they remove the need for searchers to engage as fully with the search system and the information retrieved. Learning is already a core part of the search process [32, 37, 54]. Both exploratory search and search as learning involve considerable time and effort in finding and examining relevant content. While this could be viewed as a cost, this deep exposure to content also helps people learn. As mentioned earlier, copilot users can ask richer questions (allowing them to specify their tasks and goals more fully) but they then receive synthesized answers generated by the copilot, creating fewer, new, or simply different learning opportunities for humans that must be understood. • Human learning: Supporting search requires considering the degree of searcher involvement in the search process, which varies depending on the search task [2]. Copilots enable more strategic, • Human control: higher-order actions (higher up the “task tree” from Figure 1 than typical interactions with search systems). It is clear that searchers want control over the search process. They want to know what information is/not being included and why. This helps them understand and trust system output. As things stand, copilot users delegate full control of answer generation to the AI, but the rest is mixed, i.e., less control of search mechanics (queries, etc.) but more control of task specifications (via natural language and dialog). There is more than just a basic tension between automation and control. In reality, it is not a zero sum game. Designers of copilots need to ensure human control while increasing automation [44]. New frameworks for task completion are moving in this direction. For example, AutoGen [67], uses multiple specialized assistive AI copilots that engage with humans and with each other directly to help complete complex tasks, with humans staying informed and in control throughout. Overall, these are just a few of the challenges that affect the viability of copilots. There are other challenges, such as deeply ingrained search habits that may be a barrier to the adoption of new search functionality, despite the clear benefits to searchers.

Navigating Complex Search Tasks with AI Copilots: Challenges

This paper is under CC 4.0 license. available on arxiv Authors: (1) Ryen W. White, Microsoft Research, Redmond, WA, USA. Table of Links Abstract and Taking Search to task AI Copilots Challenges Opportunities The Undiscovered Country and References 2 AI COPILOTS Copilots[6] are applications of modern AI (foundation models, etc.) to help people with complex cognitive tasks. Copilots have conversational user interfaces and their users engage with them via natural language, they are powered by foundation models such as GPT-4, they are extensible with skills/tools/plugins, and they are scoped to specialized domains or applications (including search). Copilots are designed to to keep humans at the center of the task completion process and augment human capabilities to help them complete a broader range of tasks in less time. The Microsoft copilot stack (Figure 2) contains four main layers: (1) The copilot frontend, covering the user experience and extensibility with plugins, enabling developers to provide additional visible tools to the copilot; (2) The orchestration layer that handles the internal information flows, prompting, grounding, executing the plugins and processing their responses, among other things; (3) Copilots leverage the power of large foundation models that can be provided to the developer as is, specialized to specific tasks, domains, or applications, or developers can bring their own models to use to power copilot functionality, and; (4) This all runs on top of massive scale AI infrastructure hosted in the cloud on Azure. Underpinning all of this is Microsoft’s strong commitment to responsible AI, which ensures that copilots are safe, secure, and transparent, via an iterative, layered approach with mitigations spanning the model, prompts, grounding, and the user experience. Examples of copilots include GitHub Copilot,[7] an AI pair programmer that has been shown to reduce developer effort, enable more task success, and significantly expedite task completion [5]. Copilots are also emerging in search. Popular Web search engines such as Bing and Google are adding copilot functionalities to their search engines in the form of conversational agents such as Bing Chat and Google BARD. In search, copilots can help searchers to tackle a broader range of tasks than information finding and go deeper than surface (SERP-level) interactions with content by synthesizing answers on the searcher’s behalf. They also enable searchers to communicate their intents and goals more directly. Returning to the task tree (Figure 1), the focus on engaging copilots via natural language interactions allows both searchers and systems to consider higher-level task representations (macrotasks, subtasks) in addition to the more granular actions (queries, result clicks, pagination, and so on) that searchers already perform when engaging with traditional search engines. 2.1 Copilots in Search Copilots and chat experiences are a complement not a replacement to traditional search engines. Search engines have been around for decades and serve a valuable purpose: providing near instantaneous access to answers and resources for a broad range of search requests. These existing and emerging modalities can and should work well together to help searchers tackle a wider range of tasks. The capabilities of copilots to better understand intentions and provide assistance beyond fact finding and basic learning/investigation will expand the task frontier, broadening the range of complex search tasks that can be completed, e.g., direct support for tasks requiring creative inspiration (Figure 3). This all moves us a step closer toward intelligent search systems that can help with all-task completion, covering the full universe of tasks for which people might need search support. One way to define the range of tasks that copilots can support is though Bloom’s taxonomy of learning objectives [27]. Creation is at the pinnacle of that taxonomy and we have only scratched the surface in creativity support with next-word prediction through transformer models [16]. We are already seeing expansions into modalities beyond text (images, video, audio, and so on) and could consider support for other creative tasks including planning, analysis, and invention. There are also many other layers in Bloom’s taxonomy (e.g., evaluation - help searchers make judgments and decisions, application - help searchers complete new tasks, understanding - explain ideas and concepts to accelerate learning) that could form future task frontiers. Beyond greater capabilities, the introduction of copilots into search will also change how people will engage with search systems. In copilots, the mode of interaction is primarily natural language input, with some recent expansion toward multimodal inputs and outputs via the introduction of diffusion models such as DALL·E. Copilots can generate/synthesize direct answers, with source attribution for provenance, to build trust with users, and to drive traffic back to content creators, which is important to incentivize further content creation that fuels future foundation models. The overall search interaction flow is also different between search engines and copilots. When using copilots, searchers do not need to decompose their goal into sub-goals or sub-queries, examine SERPs and landing pages, and aggregate/synthesize relevant knowledge from retrieved information. Figure 4 has a comparison of the information seeking processes in the two modalities. In copilots, the responsibility for generating answers is delegated by the searcher to the system, which is not without its challenges in terms of human agency and human learning, as we discuss later in this article. 2.2 Adding Copilots to Search Engines It is not practical nor necessary to deploy copilots for all search tasks. Foundation model inference is expensive at massive scale and search engine algorithms have been honed over decades to provide relevant results for a broad range of tasks (e.g., navigation, factfinding). Conversational interfaces are less familiar for searchers and it will take time for searchers to adapt to this way of searching. Traditional search engines are sufficient when searchers know exactly what they want. Copilots are helpful for more complex search tasks or in situations where searchers may be struggling to find relevant information. Task complexity can be estimated using aggregate metrics such as the amount of engagement with the search engine (e.g., number of query reformulations) for similar tasks historically. As generative AI appears in more applications and searchers better understand search copilot capabilities, the tasks that searchers bring to search copilots will likely evolve. We will also see a growth in search experiences that unify traditional search and copilots. In a step towards this, search engines such as Bing and Google are already integrating dynamic answers from foundation models into their SERPs for some informational queries. Also, in the Google copilot (BARD), search results are displayed immediately below the copilot responses, allowing searchers to easily engage with them as desired. In the Bing copilot (Bing Chat), the search and copilot experiences are more separated and searchers must select a specific modality based on their task and personal preferences. Bing also provides searchers with control over other aspects, such as conversation style and tone, although it is not clear that searchers are sufficiently familiar with AI copilots at this time to use these nuanced controls effectively. Search copilots such as Bing Chat use retrieval augmented generation (RAG) [28] to ground copilot responses via timely and relevant results. This has many advantages, including: (1) There is no need to retrain the massive foundation model over time; (2) Search results provide relevant and fresh information to foundation models, and; (3) It provides a provenance signal linking generated content with online sources. In response to a searcher prompt, the foundation model generates internal queries iteratively that are used to retrieve the results that form context for the copilot answers created using generative AI. Displaying these queries to searchers inline in the dialog (as in Bing Chat) creates greater transparency and helps build trust with searchers that the system is understanding their tasks and goals. The Bing orchestrator can also pull in relevant instant answers from the search engine such as weather, stock, sports, and so on, and display those in copilot responses instead of or in addition to the answers generated by the foundation model. Figure 5 shows the high-level search process from query (+ conversation context) to the answer, and the role of various key system components. Copilots also enable search engines to support more complex search tasks. Using search alone would require more searcher effort to examine search results and manually generate answers or insights (see recent work on the Delphic costs and benefits of search [9]). Of course, there are different perspectives on task complexity, e.g., the copilot perspective (denoting the amount of computation, requests, etc. required for the system to complete the task) and the searcher perspective (denoting the amount of manual effort required for the human searcher to generate an answer and complete a task). Table 1 considers the task complexity from these two different perspectives and (again drawing from Bloom’s taxonomy) provides some current, anecdotal examples of the types of tasks that both searcher and systems may find to be more or less complex. Assuming that foundational model costs will drop and sophistication will increase, we focus here on the task complexity for searchers. [6] Copilots is the Microsoft terminology for an AI-powered digital assistant designed to  work alongside humans to augment their cognitive capabilities. [7] https://github.com/features/copilot

Navigating Complex Search Tasks with AI Copilots: AI Copilots

This paper is under CC 4.0 license. available on arxiv Authors: (1) Ryen W. White, Microsoft Research, Redmond, WA, USA. Table of Links Abstract and Taking Search to task AI Copilots Challenges Opportunities The Undiscovered Country and References ABSTRACT As many of us in the information retrieval (IR) research community know and appreciate, search is far from being a solved problem. Millions of people struggle with tasks on search engines every day. Often, their struggles relate to the intrinsic complexity of their task and the failure of search systems to fully understand the task and serve relevant results [58]. The task motivates the search, creating the gap/problematic situation that searchers attempt to bridge/resolve and drives search behavior as they work through different task facets. Complex search tasks require more than support for rudimentary fact finding or re-finding. Research on methods to support complex tasks includes work on generating query and website suggestions [21, 62], personalizing and contextualizing search [4], and developing new search experiences, including those that span time and space [1, 64]. The recent emergence of generative artificial intelligence (AI) and the arrival of assistive agents, or copilots, based on this technology, has the potential to offer further assistance to searchers, especially those engaged in complex tasks [41, 61]. There are profound implications from these advances for the design of intelligent systems and for the future of search itself. This article, based on a keynote by the author at the 2023 ACM SIGIR Conference, explores these issues and charts a course toward new horizons in information access guided by AI copilots. ACM Reference Format: Ryen W. White. 2023. Navigating Complex Search Tasks with AI Copilots. Under review at REDACTED. October, 2023 1 TAKING SEARCH TO TASK Tasks are a critical part of people’s daily lives. The market for dedicated task applications that help people with their “to do” tasks is likely to grow significantly (effectively triple in size) over the next few years.[1] There are many examples of such applications that can help both individuals (e.g., Microsoft To Do, Google Tasks, Todoist) and teams (e.g., Asana, Trello, Monday.com) tackle their tasks more effectively. Over time, these systems will increasingly integrate AI to better help their users capture, manage, and complete their tasks [60]. In information access scenarios such as search, tasks play an important role in motivating searching via gaps in knowledge and problematic situations [3, 15]. AI can be central in these search scenarios, too, especially in assisting with complex search tasks. 1.1 Tasks in Search Tasks drive the search process. The IR and information science communities have long studied tasks in search [42] and many information seeking models consider the role of task directly [3, 15]. Prior research has explored the different stages of task execution (e.g., pre-focus, focus formation, post-focus) [53], task levels [39], task facets [29], tasks defined on intents (e.g., informational, transactional, and navigational [8]; well-defined or ill-defined [23]; lookup, learn, or investigate [32]), the hierarchical structure of tasks [68], the characteristics of tasks, and the attributes of task searcher interaction, e.g., task difficulty and, of course, a focus in this article, task complexity [11, 26]. As a useful framing device to help conceptualize tasks and develop system support for them, tasks can be represented as trees comprising macrotasks (high level goals), subtasks (specific components of those goals), and actions (specific steps taken by searchers toward the completion of those components) [42]. Figure 1 presents an example of a “task tree” for a task involving an upcoming vacation to Paris, France. Examples of macrotasks, subtasks, and actions are included. Moves around this tree correspond to different task applications such as task recognition (up), task decomposition (down), and task prediction (across). Only actions (e.g., queries, clicks, and so on) are directly observable to traditional search engines. However, with recent advances in search copilots (more fully supporting natural language interactions via language understanding and language generation), more aspects of macrotasks and subtasks are becoming visible to search systems and more fully understood by those systems. Challenges in working with tasks include how to represent them within search systems, how to observe more task-relevant activity and content to develop richer task models, and how to develop task-oriented interfaces that place tasks and their completion at the forefront of user engagement. Task complexity deserves a special focus in this article given the challenges that searchers can still face with complex tasks and the significant potential of AI to help searchers tackle complex tasks. 1.2 Complex Search Tasks Recent estimates suggest that half of all Web searches are not answered.[2] Many of those searches are connected to complex search tasks. These tasks are ill-defined and/or multi-step, span multiple queries, sessions, and/or devices, and require deep engagement with search engines (many queries, backtracking, branching, etc.) to complete them [21]. Complex tasks also often have many facets and cognitive dimensions, and are closely connected to searcher characteristics such as domain expertise and task familiarity [38, 58]. To date, there have been significant attempts to support complex search tasks via humans (e.g., librarians, subject matter experts) and search systems (both general Web search engines and those tailored to specific industry verticals or domains). The main technological progress so far has been in areas such as query suggestion and contextual search, with new experiences also being developed that utilize multiple devices, provide cross-session support, and enable conversational search. We are now also seeing emerging search-related technologies in the area of generative AI [35]. Before proceeding, let us dive into these different types of existing and emerging search support for complex tasks in more detail. Researchers and practitioners have long developed and deployed support such as query suggestion and trail suggestion, e.g., [21, 45], including providing guided tours [51] and suggesting popular trail destinations [62] as ways to find relevant resources. This coincides with work on contextual search and personalized search, e.g., [4, 47, 63], where search systems can use data from the current searcher such as session activity, location, reading level, and so on, and the searcher’s long-term activity history, to provide more relevant results. Search engines may also use cohort activities to help with cold-start problems for new users and augment personal profiles for more established searchers [48, 69]. • Suggestions, personalization, and contextualization: Devices have different capabilities and can be used in different settings. Multidevice experiences, e.g., [64], utilizing multiple devices simultaneously to better support complex tasks such as recipe preparation, auto repair, and home improvement that have been decomposed into steps manually or automatically [73]. Cross-device and cross-session support [1, 56] can help with ongoing/background searches for complex tasks that persist over space and time. For example, being able to predict task continuation can help with “slow search” applications that focus more on result quality than the near instantaneous retrieval of search results [46]. • Multi-device, cross-device, and cross-session: Natural language is an expressive and powerful means of communicating intentions and preferences with search systems. The introduction of clarification questions on search engine result pages (SERPs) [71], progress on conversational search [20], and even “conversations” with documents (where searchers can inquire about document content via natural language dialog) [49], enable these systems to engage more fully with searchers to better understand their tasks and goals. There are now many emerging opportunities to better understand and support more tasks via large-scale foundation models such as GPT-4[3] and DALL·E 3,[4] including offering conversational task assistance via chatbots such as ChatGPT.5 • Conversational experiences and generative AI: All of these advances, and others, have paved the way for the emergence of AI copilots, assistive agents that can help people tackle complex search tasks. [1] https://www.verifiedmarketresearch.com/product/task-management-softwaremarket/ [2] https://blogs.microsoft.com/blog/2023/02/07/reinventing-search-with-a-new-aipowered-microsoft-bing-and-edge-your-copilot-for-the-web/ [3] https://openai.com/gpt-4 [4] https://openai.com/dall-e-3 [5] https://openai.com/chatgpt

Navigating Complex Search Tasks with AI Copilots: Abstract and Taking Search to task

The New York Times Company v. OpenAI Update Court Filing, retrieved on February 26, 2024 is part of . You can jump to any part in this filing . This part is 13 of 15. HackerNoon’s Legal PDF Series here C. The DMCA Claim Fails for Multiple Independent Reasons Count V is a claim for violation of Section 1202(b) of the Copyright Act, which prohibits the “[r]emoval or [a]lteration” of copyright management information or “CMI.” 17 U.S.C. § 1202(b). Congress passed that provision in the early days of the internet in recognition of the ease with which unauthorized copies of images and other works might proliferate in cyberspace. The provision encourages rightsholders to affix CMI to their works (and prohibits its removal) so that, if their works do proliferate on the internet, the public will be able to trace those works back to their owner. S. Rep. No. 105-190, at 16–17 (1998) (CMI intended to “track[] and monitor[]”). But Congress limited the statute with a “double-scienter requirement” that prevents its application when the CMI removal occurs as an unintended result of an “automatic [] process.” Zuma Press, Inc. v. Getty Images (US), Inc., 845 F. App’x 54, 57–58 (2d Cir. 2021). A typical CMI case might involve the surreptitious removal of a photograph’s “gutter credit” to conceal a failure to seek a license from the rightsholder. Mango v. BuzzFeed, Inc., 970 F.3d 167, 169–70, 173 (2d Cir. 2020). 1. The Times Did Not Specify the CMI at Issue Count V should be dismissed at the outset for failure to specify the CMI at issue. The Complaint’s relevant paragraph fails to state what CMI is included in what work, and simply repeats the statutory text. Compl. ¶ 182 (alleging “one or more forms of [CMI]” and parroting language of Section 1202(c)).[36] The only firm allegation states that the Times placed “copyright notices” and “terms of service” links on “every page of its websites.” Compl. ¶ 125. But, at least for some articles, it did not.[37] And when it did, the information was not “conveyed in connection with” the works, 17 U.S.C. § 1202(c) (defining CMI), but hidden in small text at the bottom of the page.[38] Judge Orrick of the Northern District of California rejected similar allegations as deficient in another recent AI case. Andersen v. Stability AI Ltd., No. 23-cv-00201, 2023 WL 7132064, at *11 (N.D. Cal. Oct. 30, 2023) (must plead “exact type of CMI included in [each] work”).[39] 2. The Training-Based Section 1202 Claim Fails The first Section 1202 violation alleged in the Complaint asserts that OpenAI “removed” CMI “in building the training datasets” in violation of Section 1202(b)(1) of the DMCA. Compl. ¶ 184. As a preliminary matter, to the extent this claim is based on the “building [of] training datasets” that occurred more than three years ago, it is time-barred. 17 U.S.C. § 507(b). The Complaint also fails to plausibly allege that any CMI was removed. The Times advances three theories of removal: (1) removal of CMI when OpenAI allegedly “scraped” articles from the Times’s website; (2) removal of CMI “from third-party datasets,” i.e., Common Crawl; and (3) removal of CMI during “the training process,” which the Times alleges “does not preserve any [CMI]” “by design.” Compl. ¶¶ 184, 187. As a preliminary matter, each theory fails for failure to allege what CMI was removed or “not preserve[d],” which is particularly damning as the Times concedes that some CMI was preserved.[40] Judge Martínez-Olguín of the Northern District of California recently held an identical set of allegations to be insufficient. See Tremblay v. OpenAI, Inc., No. 3:23-cv-03416, 2024 WL 557720, at *4 (N.D. Cal. Feb. 12, 2024) (allegation that “training process does not preserve any CMI” “by design” was “conclusory”). Each theory fails separately as well. The first theory fails because, while the Times states that OpenAI “scraped” its articles “directly from [its] websites,” none of the specific allegations actually suggest OpenAI designed its alleged “scrap[ing]” process to omit CMI. Compl. ¶ 184.[41] And the only allegations about OpenAI “scraping” articles from Times websites relate to the creation of WebText, which occurred over three years before this lawsuit. Supra Section IV.A.[42] The second theory fails because the Complaint lacks allegations about the inclusion (or exclusion) of the Times’s CMI in any “third-party datasets” like Common Crawl,[43] much less about OpenAI scrubbing any CMI from those datasets. First Nationwide Bank v. Gelt Funding Corp., 27 F.3d 763, 771 (2d Cir. 1994) (courts ignore “unwarranted deductions of fact”). And the third theory fails because there is no allegation in the Complaint supporting the conclusion that the “training process” excludes CMI “[b]y design.” Compl. ¶ 187; see also Tremblay, 2024 WL 557720, at *4. Moreover, the Times fails to allege facts that could show how the alleged CMI removal could “induce, enable, facilitate, or conceal an infringement” of copyright—much less how OpenAI could have “reasonable grounds to know” it would. 17 U.S.C. § 1202(b). The “point of CMI” is to provide information to “the public,” not to govern purely internal databases. Roberts v. BroadwayHD LLC, 518 F. Supp. 3d 719, 737 (S.D.N.Y. 2021); Compl. ¶ 59. As Judge Martínez-Olguín explained, it is far from obvious how “the alleged removal of CMI in an internal database [could] enable infringement.” Tremblay, 2024 WL 557720, at *4 (dismissing claim).[44] 3. The Output-Based Section 1202 Claim Fails The second category of Section 1202 violation in the Complaint alleges that (1) OpenAI violated Section 1202(b)(1)’s removal prohibition by failing to include the Times’s CMI in model outputs, Compl. ¶¶ 185–86; and (2) by displaying those outputs via ChatGPT or its API, OpenAI violated Section 1202(b)(3)’s prohibition on “distribut[ing]” works “knowing that [CMI] has been removed,” Compl. ¶ 189. Neither theory states a claim for relief. As a preliminary matter, the Times’s Section 1202(b)(3) claim fails because the Complaint does not allege that OpenAI “distribute[d]” any outputs. In this context, “distribution” requires a “sale or transfer of ownership extending beyond that of a mere public display.” Wright v. Miah, No. 22-cv-4132, 2023 WL 6219435, at *7 (E.D.N.Y. Sept. 7, 2023) (emphasis added).[45] But “mere public display” of outputs is all the Complaint alleges. See, e.g., Compl. ¶ 102. Regardless, this “output” theory fails because the outputs alleged in the Complaint are not wholesale copies of entire Times articles. They are, at best, reproductions of excerpts of those articles, some of which are little more than collections of scattered sentences. Supra 12. If the absence of CMI from such excerpts constituted a “removal” of that CMI, then DMCA liability would attach to any journalist who used a block quote in a book review without also including extensive information about the book’s publisher, terms and conditions, and original copyright notice. See supra note 22 (example of the Times including 200-word block quote in book review). To avoid such anomalous results, courts have cabined applications of Section 1202(b)(1) and (3) to circumstances in which the works in question were “substantially or entirely reproduced.” Fischer v. Forrest, 286 F. Supp. 3d 590, 609 (S.D.N.Y. 2018). As such, failure to include original CMI in anything less than an identical reproduction of all (or almost all) of the work does not qualify as CMI removal. Tremblay, 2024 WL 557720, at *5 (dismissing claim because “Plaintiffs have not alleged that [OpenAI] distributed their books or copies of [them]”).[46] As the Times has not alleged that OpenAI reproduced entire articles, the output-based claim fails. Even setting that aside, the Times’s output-based CMI claim fails for the independent reason that there was no CMI to remove from the relevant text. The Exhibit J outputs, for example, feature text from the middle of articles. Ex. J. at 2–126. As shown in the exhibit, the “Actual text from NYTimes” contains no information that could qualify as CMI. See, e.g., id. at 3; 17 U.S.C. § 1202(c) (defining CMI). So too for the ChatGPT outputs featured in the Complaint, which request the “first [and subsequent] paragraph[s]” from Times articles. See, e.g., Compl. ¶¶ 104, 106, 118, 121. None of those “paragraphs” contains any CMI that OpenAI could have “removed.” 4. The Times Fails to Allege a CMI-Based Injury Count V separately fails for lack of standing. “[T]o have standing” to sue for a DMCA violation, the Times “must show that [it] was injured by that violation.” Steele v. Bongiovi, 784 F. Supp. 2d 94, 97–98 (D. Mass. 2011); see also 17 U.S.C. § 1203(a). Here, the Complaint’s “Harm to the Times” section relates entirely to its inability to receive speculative licensing revenue, see Compl. ¶¶ 155–56, and the possibility that ChatGPT will “divert readers,” see id. ¶ 157. Neither injury has any nexus to CMI. Nor is there any imaginable harm here: because all of the Complaint’s outputs were either generated using the original Times article itself, see Ex. J, or referenced the Times by name, see, e.g., Compl. ¶ 104, any user who encountered those outputs would have no doubt as to the provenance of the text and could easily find it on the Times’s website (as ChatGPT often invites them to do, see id. ¶¶ 106, 134). Cf. Kelly, 77 F. Supp. 2d at 1122 (DMCA claim failed because users who encounter images are “given the name of the Web site from which Defendant obtained the image, where any associated [CMI] would be available”). Continue Reading . Here [36] Design Pics Inc. v. PBH Network, Inc., No. 20-cv-1096, 2020 WL 8413512, at *4 (E.D.N.Y. Oct. 27, 2020) (dismissing CMI claim based on “conclusory, boilerplate parroting of the statutory text”). [37] See, e.g., John Branch, Snow Fall: The Avalanche at Tunnel Creek, N.Y. Times, https://www.nytimes.com/projects/2012/snow-fall/index.html#/?part=tunnel-creek (last accessed Feb. 11, 2024); see also Compl. ¶¶ 104–05 & n.28 (citing this article). [38] See, e.g., Pete Wells, As Not Seen on TV, N.Y. Times (Nov. 13, 2012), https://www.nytimes.com/2012/11/14/dining/reviews/restaurant-review-guys-american-kitchen-bar-in-timessquare.html; see also Compl. ¶¶ 106–07 & n.29 (citing and quoting this article). [39] See also Wood v. Observer Holdings, LLC, No. 20-cv-07878, 2021 WL 2874100, at *8 (S.D.N.Y., July 8, 2021) (terms on “separate website” not CMI); GC2 Incorporated v. Int’l Game Tech. PLC, 255 F. Supp. 3d 812, 821–22 (N.D. Ill. 2017) (“terms of use notice near a copyrighted work” not “conveyed in connection with” work). [40] See, e.g., Compl. ¶ 106 (ChatGPT responding to query naming “Pete Wells” with a completion correctly identifying the publication date of Wells’ article); see also id. ¶ 104 (same, for title). [41] Kelly v. Arriba Soft Corp., 77 F. Supp. 2d 1116, 1122 (C.D. Cal. 1999), rev'd on other grounds by 336 F.3d 811 (9th Cir. 2003) (no Section 1202 liability vs. search engine “crawler [that] did not include [CMI]”). [42] The Times suggests that OpenAI’s “Browse with Bing” feature “scrap[es] Times Works from The Times’s websites,” Compl. ¶ 185, but the Complaint does not include a single allegation supporting that conclusion, see supra 13 & n.32 (noting that Browse with Bing fetched content from third-party sites, not the Times’s website). [43] OpenAI cannot have removed CMI from datasets that “contained no such [CMI]” in the first place. McGucken v. Shutterstock, Inc., No. 22-cv-00905, 2023 WL 6390530, at 11 (S.D.N.Y. Oct. 2, 2023) (rejecting DMCA claim). [44] See Victor Elias Photography, LLC v. Ice Portal, Inc., 43 F.4th 1313, 1325 (11th Cir. 2022) (requiring “some identifiable connection between the defendant’s actions and the infringement or the likelihood of infringement.”). [45] Id. at *10 (endorsing Section 1202(b)(3) claim where defendant distributed artwork on Etsy); MyPlayCity, Inc. v. Conduit Ltd., No. 10-cv-1615, 2012 WL 1107648, at *12 (S.D.N.Y. Mar. 30, 2012) (“distribution” means “actual dissemination of copies”); FurnitureDealer.Net, Inc v. Amazon.com, Inc, No. 18-cv-232, 2022 WL 891473, at *23 (D. Minn. Mar. 25, 2022) (“[P]ublic display does not constitute distribution, and thus is not a [DMCA] violation.”). [46] See also Doe 1 v. GitHub, Inc., No. 22-cv-06823, 2024 WL 235217, *9 (N.D. Cal. Jan. 22, 2024) (dismissing Section 1202(b) claim against OpenAI because outputs were “not identical” to originals); A’Lor Int’l, Ltd. v. Tapper Fine Jewelry, Inc., No. 12-cv-02215, 2012 WL 12921035, at *10 (C.D. Cal. Aug. 8, 2012) (“the plain language of the statute encompasses only removal and alteration;” does not “include [mere] omissions”); Faulkner Press, L.L.C. v. Class Notes, L.L.C., 756 F. Supp. 2d 1352, 1358–59 (N.D. Fla. 2010) (rejecting claim where “word for word” text was “copied into a different form and [] incorporated into” commercial materials); Kelly, 77 F. Supp. 2d at 1121–22 (“displaying thumbnails of Plaintiffs’ images without [] the corresponding [CMI]” was not CMI “removal”). About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case retrieved on February 26, 2024, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. fingfx.thomsonreuters.com

ChatGPT Never Produced "Wholesale Copies" of NYT's Articles

Authors: (1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China; (2) Peng Liang, School of Computer Science, Wuhan University, Wuhan, China; (3) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China; (4) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany; (4) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia; (4) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland. VII. CONCLUSIONS In this study, we focused on the issues users may encounter when using GitHub Copilot, as well as their underlying causes and potential solutions. Following identifying the RQs, we collected data from GitHub Issues, GitHub Discussions and SO posts. After manual screening, we obtained 476 GitHub Issues, 706 GitHub Discussions, and 184 SO posts related to Copilot and got a total of 1399 issues, 337 causes, and 497 solutions based on our data extraction criteria. The results indicate that Usage Issue and Compatibility Issue are the most common problems faced by users, Copilot Internal Issue, Network Connection Issue, and Editor/IDE Compatibility Issue are identified as the most usual causes of issues, and Bug Fixed by Copilot, Modify Configuration/Setting and Use Suitable Version are the predominant solution. Our findings suggest that Copilot should enhance compatibility across various IDEs and editors, simplify the configuration, improve the quality of generated code, and address concerns related to intellectual property and copyright. Additionally, users require more customization options to tailor Copilot’s behavior and have more control over the content generated by Copilot. In light of the additional time required for code suggestion verification when utilizing Copilot, the integration of a code explanation feature becomes imperative to enhance its overall utility and effectiveness in practical development scenarios. In the next step, we plan to combine a survey and code testing experiments to evaluate the actual usage of Copilot by users, as well as its performance in terms of security, maintainability, and other aspects. REFERENCES [1] M. Robillard, R. Walker, and T. Zimmermann, “Recommendation systems for software engineering,” IEEE Software, vol. 27, no. 4, pp. 80– 86, 2010. [2] S. Luan, D. Yang, C. Barnaby, K. Sen, and S. Chandra, “Aroma: Code recommendation via structural code search,” Proceedings of the ACM on Programming Languages, vol. 3, pp. 1–28, 2019. [3] Gartner Identifies the Top 10 Strategic Technology Trends for 2024. https://tinyurl.com/2p879w7s. [4] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al., “Program synthesis with large language models,” arXiv preprint abs/2108.07732, 2021. [5] GitHub Copilot · Your AI Pair Programmer. https://github.com/features/ copilot. [6] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in Proceedings of the 43rd IEEE Symposium on Security and Privacy (S&P), pp. 754–768, IEEE, 2022. [7] B. Yetistiren, I. Ozsoy, and E. Tuzun, “Assessing the quality of github copilot’s code generation,” in Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE), pp. 62–71, ACM, 2022. [8] N. Nguyen and S. Nadi, “An empirical evaluation of github copilot’s code suggestions,” in Proceedings of the 19th IEEE/ACM International Conference on Mining Software Repositories (MSR), pp. 1–5, IEEE, 2022. [9] S. Imai, “Is github copilot a substitute for human pair-programming? an empirical study,” in Proceedings of the 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 319–321, IEEE, 2022. [10] S. Barke, M. B. James, and N. Polikarpova, “Grounded copilot: How programmers interact with code-generating models,” Proceedings of the ACM on Programming Languages, vol. 7, pp. 1–27, 2023. [11] S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The impact of ai on developer productivity: Evidence from github copilot,” arXiv preprint abs/2302.06590, 2023. [12] B. G. Glaser, “The constant comparative method of qualitative analysis,” Social Problems, vol. 12, no. 4, pp. 436–445, 1965. [13] X. Zhou, P. Liang, B. Zhang, Z. Li, A. Ahmad, M. Shahin, and M. Waseem, Dataset of the Paper “On the Concerns of Developers When Using GitHub Copilot”, 2023. [14] J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960. [15] GitHub Copilot Chat beta. https://github.blog/ 2023-09-20-github-copilot-chat-beta-now-available-for-all-individuals/. [16] Copilot.vim. https://github.com/github/copilot.vim. [17] Configuring network settings for GitHub Copilot. https://tinyurl.com/ 39vphp46. [18] GitHub Copilot X. https://github.com/features/preview/copilot-x. [19] B. Zhang, P. Liang, X. Zhou, A. Ahmad, and M. Waseem, “Demystifying practices, challenges and expected features of using github copilot,” International Journal of Software Engineering and Knowledge Engineering, 2023. [20] C. Bird, D. Ford, T. Zimmermann, N. Forsgren, E. Kalliamvakou, T. Lowdermilk, and I. Gazit, “Taking flight with copilot: Early insights and opportunities of ai-powered pair-programming tools,” ACM Queue, vol. 20, no. 6, pp. 35–57, 2023. [21] R. Wang, R. Cheng, D. Ford, and T. Zimmermann, “Investigating and designing for trust in ai-powered code generation tools,” arXiv preprint abs/2305.11248, 2023. [22] C. Wang, J. Hu, C. Gao, Y. Jin, T. Xie, H. Huang, Z. Lei, and Y. Deng, “Practitioners’ expectations on code completion,” arXiv preprint abs/2301.03846, 2023. [23] M. Jaworski and D. Piotrkowski, “Study of software developers’ experience using the github copilot tool in the software development process,” arXiv preprint abs/2301.04991, 2023. [24] J. T. Liang, C. Yang, and B. A. Myers, “A large-scale survey on the usability of ai programming assistants: Successes and challenges,” in Proceedings of the 45th International Conference on Software Engineering (ICSE), ACM, 2024. [25] G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg, “Security implications of large language model code assistants: A user study,” arXiv preprint abs/2208.09727, 2022. This paper is under CC 4.0 license. available on arxiv

COPHI

cophi.io

1-10 employees

Since 2021

COMPANY RANKING

COPHI

EVERGREEN INDEX #323

Representing this company? Click here to claim and customize this page!

Cophi WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

COPHI

cophi.io

1-10 employees

Since 2021

COMPANY RANKING

COPHI

EVERGREEN INDEX #323

RELATED COMPANIESRANK

STORIES ABOUT COPHI

Representing this company? Click here to claim and customize this page!

Cophi WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!