Overcoming Limitations in AI Chatbot Research for Future Educational Impact

Despite its inherent limitations, including its single case design and the potential for bias, the exploratory depth of the study uncovered hidden potential within these systems, even amid serious concerns about generalizability.

Future research could include:

• Refining the crafting of prompts.

• Exploring new features of these and other GenAIbots being introduced with increasing frequency.

• Establishing standardised benchmarks to evaluate and compare chatbots and AI systems' performance, accuracy, and reliability.

• Conducting long-term studies to observe the evolution of chatbots' capabilities and their impact on user interactions over time.

• Conducting research with real students in classroom settings and beyond to assess these AI systems' practical educational applications and challenges.

• Investigating chatbots' learning and adaptation capabilities to individual user needs and preferences over time.

• Researching the integration of multimodal inputs (e.g., text, voice, image) to enhance chatbot capabilities and user interaction experiences.

When implementing GenAIbots in Chemistry learning, it is crucial to evaluate benefits and drawbacks judiciously, ensuring accurate information delivery and considering the implications of reduced human interaction. These concerns may be alleviated by integrating GenAIbots with other educational tools or activities promoting collaborative dialogue among learners.

Author Contributions

The author confirms sole responsibility for the following: study conception and design, data collection, analysis and interpretation of results, and manuscript preparation.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

The authors warmly acknowledge Melanie Swan for her invaluable suggestion, which led to the transition from using the term 'objects-to-think-with' to 'agents-to-thinkwith'.

References

Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionising education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152

Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4337484

Bardin, L. (1977). L’analyse de contenu. PUF - Presses Universitaires de France.

Bitzenbauer, P. (2023). ChatGPT in physics education: A pilot study on easy-to-implement activities. Contemporary Educational Technology, 15(3), ep430. https://doi.org/10.30935/cedtech/13176

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, 2005.14165. http://arxiv.org/abs/2005.14165

Castro Nascimento, C. M., & Pimentel, A. S. (2023). Do Large Language Models Understand Chemistry? A Conversation with ChatGPT. Journal of Chemical Information and Modeling, 63(6), 1649–1655. https://doi.org/10.1021/acs.jcim.3c00285

David, E. (2023, September 21). Microsoft to add DALL-E 3 to Bing Chat. The Verge; Vox Media. https://www.theverge.com/2023/9/21/23873690/microsoft-new-ai-features-bingsearch-shopping-dall-e-3

Dewi, C. A., Pahriah, P., & Purmadi, A. (2021). The Urgency of Digital Literacy for Generation Z Students in Chemistry Learning. International Journal of Emerging Technologies in Learning, 16(11), 88–103. https://doi.org/10.3991/ijet.v16i11.19871

Dunlop, L., Hodgson, A., & Stubbs, J. E. (2020). Building capabilities in chemistry education: happiness and discomfort through philosophical dialogue in chemistry. Chemistry Education Research and Practice, 21(1), 438–451. https://doi.org/10.1039/C9RP00141G

Flavell, J. H. (1976). Metacognitive aspects of problem-solving. In L. B. Resnick (Ed.), The Nature of Intelligence (pp. 231–236). Lawrence Erlbaum.

Floridi, L., & Chiriatti, M. (2020). GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines, 30(4), 681–694. https://doi.org/10.1007/s11023-020-09548-1

Franciscu, S. (2023). ChatGPT: A Natural Language Generation Model for Chatbots. https://doi.org/10.13140/RG.2.2.24777.83044

Gregorcic, B., & Pendrill, A.-M. (2023). ChatGPT and the frustrated Socrates. Physics Education, 58(3), 035021. https://doi.org/10.1088/1361-6552/acc299

Latour, B. (1991). Nous n’avons jamais été modernes: Essai d’anthropologie symétrique. La Découverte.

Leon, A. J., & Vidhani, D. (2023). ChatGPT Needs a Chemistry Tutor, Too. Journal of Chemical Education. https://doi.org/10.1021/acs.jchemed.3c00288

Liu, F., Budiu, R., Zhang, A., & Cionca, E. (2023, October 1). ChatGPT, Bard, or Bing Chat? Differences Among 3 Generative-AI Bots. https://www.nngroup.com/articles/ai-botcomparison/

Metz, C., & Hsu, T. (2023, September 20). ChatGPT Can Now Generate Images, Too. New York Times - Technology. https://www.nytimes.com/2023/09/20/technology/chatgpt-dalle3-images-openai.html

Mishra, A., Soni, U., Arunkumar, A., Huang, J., Kwon, B. C., & Bryan, C. (2023). PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models. ArXiv, 2304.01964. https://doi.org/10.48550/arXiv.2304.01964

Mollick, E. R. (2023, April 26). A guide to prompting AI (for what it is worth): A little bit of magic,but mostly just practice. One Useful Thing Blog. https://www.oneusefulthing.org/p/a-guidetoprompting-ai-for-what

Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033

OpenAI. (2023). GPT-4 Technical Report. OpenAI. https://cdn.openai.com/papers/gpt-4.pdf

Papert, S. A. (1980). Mindstorms - Children, Computers and Powerful Ideas. Basic Books. http://www.arvindguptatoys.com/arvindgupta/mindstorms.pdf

Permatasari, M. B., Rahayu, S., & Dasna, W. (2022). Chemistry Learning Using Multiple Representations: A Systematic Literature Review. J.Sci.Learn.2022, 5(2), 334–341. https://doi.org/10.17509/jsl.v5i2.42656

Pimentel, A., Wagener, A., Silveira, E. F. da, Picciani, P., Salles, B., Follmer, C., & Jr., O. N. O. (2023). Challenging ChatGPT with Chemistry-Related Subjects. ChemRxiv. https://doi.org/10.26434/chemrxiv-2023-xl6w3

Schlosser, M. (2019). Agency. In The Stanford Encyclopedia of Philosophy. https://plato.stanford.edu/archives/win2019/entries/agency/

Swan, M. (2015, January 21). We Should Consider the Future World as one of Multi-Species Intelligence. Response to The Edge Question 2015: What Do You Think About Machines That Think?; Edge.org. http://edge.org/response-detail/26070

Taylor, C. A., Hogarth, H., Hacking, E. B., & Bastos, E. (2022). Posthuman Object Pedagogies: Thinking with Things to Think with Theory for Innovative Educational Research. Cultural and Pedagogical Inquiry, 14(1), 206–221. https://doi.org/10.18733/cpi29662

Timilsena, N. P., Maharjan, K. B., & Devkota, K. M. (2022). Teachers’ And Students’ Experiences In Chemistry Learning Difficulties. Journal of Positive School Psychology, 6(10), 2856–2867. https://www.journalppw.com/index.php/jpsp/article/view/13764

Tümay, H. (2016). Reconsidering learning difficulties and misconceptions in chemistry: emergence in chemistry and its implications for chemical education. Chemistry Education Research and Practice, 17(2), 229–245. https://doi.org/10.1039/C6RP00008H

Turkle, S. (1984). The Second Self: Computers and the Human Spirit. Simon and Schuster.

Yin, R. K. (2011). Applications of Case Study Research. SAGE Publications, Inc.

This paper is available on arxiv under CC BY-SA 4.0 DEED license.