Overcoming Limitations in AI Chatbot Research for Future Educational Impactby@textmodels
256 reads

Overcoming Limitations in AI Chatbot Research for Future Educational Impact

Too Long; Didn't Read

Despite limitations like single-case design and potential bias, future AI chatbot studies can refine prompts, establish benchmarks, and explore multimodal inputs for enhanced educational impact. Strategies include long-term studies, real classroom research, and integrating GenAIbots with collaborative activities to address concerns about reduced human interaction.
featured image - Overcoming Limitations in AI Chatbot Research for Future Educational Impact
Writings, Papers and Blogs on Text Models HackerNoon profile picture


(1) Renato P. dos Santos, CIAGE – Centre for Generative Artificial Intelligence in Cognition and Education.

Abstract and Introduction

Materials And Methods

Results and Analyses

Prompts and generated texts

Conceptualizing chemical reactions

Deepening on understanding of chemical reactions

Question about combustion

Question about a graph of gases turning into water over time

Question about the difference between atoms, molecules, and moles

Deepening on the concept of mole

Question about changing of state

Question about an animated representation of water molecules undergoing phase changes

Question about plasma, a state of matter

Question about chemical bondings

Question about illustration of chemical bonds

Question about the essence of the type of chemical bonding

Further analysis


Limitations of the study and possible future studies

Author Contributions, Conflicts of interest, Acknowledgements, and References

Limitations of the study and possible future studies

Despite its inherent limitations, including its single case design and the potential for bias, the exploratory depth of the study uncovered hidden potential within these systems, even amid serious concerns about generalizability.

Future research could include:

• Refining the crafting of prompts.

• Exploring new features of these and other GenAIbots being introduced with increasing frequency.

• Establishing standardised benchmarks to evaluate and compare chatbots and AI systems' performance, accuracy, and reliability.

• Conducting long-term studies to observe the evolution of chatbots' capabilities and their impact on user interactions over time.

• Conducting research with real students in classroom settings and beyond to assess these AI systems' practical educational applications and challenges.

• Investigating chatbots' learning and adaptation capabilities to individual user needs and preferences over time.

• Researching the integration of multimodal inputs (e.g., text, voice, image) to enhance chatbot capabilities and user interaction experiences.

When implementing GenAIbots in Chemistry learning, it is crucial to evaluate benefits and drawbacks judiciously, ensuring accurate information delivery and considering the implications of reduced human interaction. These concerns may be alleviated by integrating GenAIbots with other educational tools or activities promoting collaborative dialogue among learners.

Author Contributions

The author confirms sole responsibility for the following: study conception and design, data collection, analysis and interpretation of results, and manuscript preparation.

Conflicts of interest

There are no conflicts to declare.


The authors warmly acknowledge Melanie Swan for her invaluable suggestion, which led to the transition from using the term 'objects-to-think-with' to 'agents-to-thinkwith'.


Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionising education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429.

Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. SSRN Electronic Journal.

Bardin, L. (1977). L’analyse de contenu. PUF - Presses Universitaires de France.

Bitzenbauer, P. (2023). ChatGPT in physics education: A pilot study on easy-to-implement activities. Contemporary Educational Technology, 15(3), ep430.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, 2005.14165.

Castro Nascimento, C. M., & Pimentel, A. S. (2023). Do Large Language Models Understand Chemistry? A Conversation with ChatGPT. Journal of Chemical Information and Modeling, 63(6), 1649–1655.

David, E. (2023, September 21). Microsoft to add DALL-E 3 to Bing Chat. The Verge; Vox Media.

Dewi, C. A., Pahriah, P., & Purmadi, A. (2021). The Urgency of Digital Literacy for Generation Z Students in Chemistry Learning. International Journal of Emerging Technologies in Learning, 16(11), 88–103.

Dunlop, L., Hodgson, A., & Stubbs, J. E. (2020). Building capabilities in chemistry education: happiness and discomfort through philosophical dialogue in chemistry. Chemistry Education Research and Practice, 21(1), 438–451.

Flavell, J. H. (1976). Metacognitive aspects of problem-solving. In L. B. Resnick (Ed.), The Nature of Intelligence (pp. 231–236). Lawrence Erlbaum.

Floridi, L., & Chiriatti, M. (2020). GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines, 30(4), 681–694.

Franciscu, S. (2023). ChatGPT: A Natural Language Generation Model for Chatbots.

Gregorcic, B., & Pendrill, A.-M. (2023). ChatGPT and the frustrated Socrates. Physics Education, 58(3), 035021.

Latour, B. (1991). Nous n’avons jamais été modernes: Essai d’anthropologie symétrique. La Découverte.

Leon, A. J., & Vidhani, D. (2023). ChatGPT Needs a Chemistry Tutor, Too. Journal of Chemical Education.

Liu, F., Budiu, R., Zhang, A., & Cionca, E. (2023, October 1). ChatGPT, Bard, or Bing Chat? Differences Among 3 Generative-AI Bots.

Metz, C., & Hsu, T. (2023, September 20). ChatGPT Can Now Generate Images, Too. New York Times - Technology.

Mishra, A., Soni, U., Arunkumar, A., Huang, J., Kwon, B. C., & Bryan, C. (2023). PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models. ArXiv, 2304.01964.

Mollick, E. R. (2023, April 26). A guide to prompting AI (for what it is worth): A little bit of magic,but mostly just practice. One Useful Thing Blog.

Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033.

OpenAI. (2023). GPT-4 Technical Report. OpenAI.

Papert, S. A. (1980). Mindstorms - Children, Computers and Powerful Ideas. Basic Books.

Permatasari, M. B., Rahayu, S., & Dasna, W. (2022). Chemistry Learning Using Multiple Representations: A Systematic Literature Review. J.Sci.Learn.2022, 5(2), 334–341.

Pimentel, A., Wagener, A., Silveira, E. F. da, Picciani, P., Salles, B., Follmer, C., & Jr., O. N. O. (2023). Challenging ChatGPT with Chemistry-Related Subjects. ChemRxiv.

Schlosser, M. (2019). Agency. In The Stanford Encyclopedia of Philosophy.

Swan, M. (2015, January 21). We Should Consider the Future World as one of Multi-Species Intelligence. Response to The Edge Question 2015: What Do You Think About Machines That Think?;

Taylor, C. A., Hogarth, H., Hacking, E. B., & Bastos, E. (2022). Posthuman Object Pedagogies: Thinking with Things to Think with Theory for Innovative Educational Research. Cultural and Pedagogical Inquiry, 14(1), 206–221.

Timilsena, N. P., Maharjan, K. B., & Devkota, K. M. (2022). Teachers’ And Students’ Experiences In Chemistry Learning Difficulties. Journal of Positive School Psychology, 6(10), 2856–2867.

Tümay, H. (2016). Reconsidering learning difficulties and misconceptions in chemistry: emergence in chemistry and its implications for chemical education. Chemistry Education Research and Practice, 17(2), 229–245.

Turkle, S. (1984). The Second Self: Computers and the Human Spirit. Simon and Schuster.

Yin, R. K. (2011). Applications of Case Study Research. SAGE Publications, Inc.

This paper is available on arxiv under CC BY-SA 4.0 DEED license.