paint-brush
Overcoming Limitations in AI Chatbot Research for Future Educational Impactby@textmodels
322 reads
322 reads

Overcoming Limitations in AI Chatbot Research for Future Educational Impact

Too Long; Didn't Read

Despite limitations like single-case design and potential bias, future AI chatbot studies can refine prompts, establish benchmarks, and explore multimodal inputs for enhanced educational impact. Strategies include long-term studies, real classroom research, and integrating GenAIbots with collaborative activities to address concerns about reduced human interaction.
featured image - Overcoming Limitations in AI Chatbot Research for Future Educational Impact
Writings, Papers and Blogs on Text Models HackerNoon profile picture

Authors:

(1) Renato P. dos Santos, CIAGE – Centre for Generative Artificial Intelligence in Cognition and Education.

Abstract and Introduction

Materials And Methods

Results and Analyses

Prompts and generated texts

Conceptualizing chemical reactions

Deepening on understanding of chemical reactions

Question about combustion

Question about a graph of gases turning into water over time

Question about the difference between atoms, molecules, and moles

Deepening on the concept of mole

Question about changing of state

Question about an animated representation of water molecules undergoing phase changes

Question about plasma, a state of matter

Question about chemical bondings

Question about illustration of chemical bonds

Question about the essence of the type of chemical bonding

Further analysis

Conclusions

Limitations of the study and possible future studies

Author Contributions, Conflicts of interest, Acknowledgements, and References

Limitations of the study and possible future studies

Despite its inherent limitations, including its single case design and the potential for bias, the exploratory depth of the study uncovered hidden potential within these systems, even amid serious concerns about generalizability.


Future research could include:


• Refining the crafting of prompts.


• Exploring new features of these and other GenAIbots being introduced with increasing frequency.


• Establishing standardised benchmarks to evaluate and compare chatbots and AI systems' performance, accuracy, and reliability.


• Conducting long-term studies to observe the evolution of chatbots' capabilities and their impact on user interactions over time.


• Conducting research with real students in classroom settings and beyond to assess these AI systems' practical educational applications and challenges.


• Investigating chatbots' learning and adaptation capabilities to individual user needs and preferences over time.


• Researching the integration of multimodal inputs (e.g., text, voice, image) to enhance chatbot capabilities and user interaction experiences.


When implementing GenAIbots in Chemistry learning, it is crucial to evaluate benefits and drawbacks judiciously, ensuring accurate information delivery and considering the implications of reduced human interaction. These concerns may be alleviated by integrating GenAIbots with other educational tools or activities promoting collaborative dialogue among learners.


Author Contributions

The author confirms sole responsibility for the following: study conception and design, data collection, analysis and interpretation of results, and manuscript preparation.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

The authors warmly acknowledge Melanie Swan for her invaluable suggestion, which led to the transition from using the term 'objects-to-think-with' to 'agents-to-thinkwith'.

References

Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionising education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152


Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4337484


Bardin, L. (1977). L’analyse de contenu. PUF - Presses Universitaires de France.


Bitzenbauer, P. (2023). ChatGPT in physics education: A pilot study on easy-to-implement activities. Contemporary Educational Technology, 15(3), ep430. https://doi.org/10.30935/cedtech/13176


Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, 2005.14165. http://arxiv.org/abs/2005.14165


Castro Nascimento, C. M., & Pimentel, A. S. (2023). Do Large Language Models Understand Chemistry? A Conversation with ChatGPT. Journal of Chemical Information and Modeling, 63(6), 1649–1655. https://doi.org/10.1021/acs.jcim.3c00285


David, E. (2023, September 21). Microsoft to add DALL-E 3 to Bing Chat. The Verge; Vox Media. https://www.theverge.com/2023/9/21/23873690/microsoft-new-ai-features-bingsearch-shopping-dall-e-3


Dewi, C. A., Pahriah, P., & Purmadi, A. (2021). The Urgency of Digital Literacy for Generation Z Students in Chemistry Learning. International Journal of Emerging Technologies in Learning, 16(11), 88–103. https://doi.org/10.3991/ijet.v16i11.19871


Dunlop, L., Hodgson, A., & Stubbs, J. E. (2020). Building capabilities in chemistry education: happiness and discomfort through philosophical dialogue in chemistry. Chemistry Education Research and Practice, 21(1), 438–451. https://doi.org/10.1039/C9RP00141G


Flavell, J. H. (1976). Metacognitive aspects of problem-solving. In L. B. Resnick (Ed.), The Nature of Intelligence (pp. 231–236). Lawrence Erlbaum.


Floridi, L., & Chiriatti, M. (2020). GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines, 30(4), 681–694. https://doi.org/10.1007/s11023-020-09548-1


Franciscu, S. (2023). ChatGPT: A Natural Language Generation Model for Chatbots. https://doi.org/10.13140/RG.2.2.24777.83044


Gregorcic, B., & Pendrill, A.-M. (2023). ChatGPT and the frustrated Socrates. Physics Education, 58(3), 035021. https://doi.org/10.1088/1361-6552/acc299


Latour, B. (1991). Nous n’avons jamais été modernes: Essai d’anthropologie symétrique. La Découverte.


Leon, A. J., & Vidhani, D. (2023). ChatGPT Needs a Chemistry Tutor, Too. Journal of Chemical Education. https://doi.org/10.1021/acs.jchemed.3c00288


Liu, F., Budiu, R., Zhang, A., & Cionca, E. (2023, October 1). ChatGPT, Bard, or Bing Chat? Differences Among 3 Generative-AI Bots. https://www.nngroup.com/articles/ai-botcomparison/


Metz, C., & Hsu, T. (2023, September 20). ChatGPT Can Now Generate Images, Too. New York Times - Technology. https://www.nytimes.com/2023/09/20/technology/chatgpt-dalle3-images-openai.html


Mishra, A., Soni, U., Arunkumar, A., Huang, J., Kwon, B. C., & Bryan, C. (2023). PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models. ArXiv, 2304.01964. https://doi.org/10.48550/arXiv.2304.01964


Mollick, E. R. (2023, April 26). A guide to prompting AI (for what it is worth): A little bit of magic,but mostly just practice. One Useful Thing Blog. https://www.oneusefulthing.org/p/a-guidetoprompting-ai-for-what


Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033


OpenAI. (2023). GPT-4 Technical Report. OpenAI. https://cdn.openai.com/papers/gpt-4.pdf


Papert, S. A. (1980). Mindstorms - Children, Computers and Powerful Ideas. Basic Books. http://www.arvindguptatoys.com/arvindgupta/mindstorms.pdf


Permatasari, M. B., Rahayu, S., & Dasna, W. (2022). Chemistry Learning Using Multiple Representations: A Systematic Literature Review. J.Sci.Learn.2022, 5(2), 334–341. https://doi.org/10.17509/jsl.v5i2.42656


Pimentel, A., Wagener, A., Silveira, E. F. da, Picciani, P., Salles, B., Follmer, C., & Jr., O. N. O. (2023). Challenging ChatGPT with Chemistry-Related Subjects. ChemRxiv. https://doi.org/10.26434/chemrxiv-2023-xl6w3


Schlosser, M. (2019). Agency. In The Stanford Encyclopedia of Philosophy. https://plato.stanford.edu/archives/win2019/entries/agency/


Swan, M. (2015, January 21). We Should Consider the Future World as one of Multi-Species Intelligence. Response to The Edge Question 2015: What Do You Think About Machines That Think?; Edge.org. http://edge.org/response-detail/26070


Taylor, C. A., Hogarth, H., Hacking, E. B., & Bastos, E. (2022). Posthuman Object Pedagogies: Thinking with Things to Think with Theory for Innovative Educational Research. Cultural and Pedagogical Inquiry, 14(1), 206–221. https://doi.org/10.18733/cpi29662


Timilsena, N. P., Maharjan, K. B., & Devkota, K. M. (2022). Teachers’ And Students’ Experiences In Chemistry Learning Difficulties. Journal of Positive School Psychology, 6(10), 2856–2867. https://www.journalppw.com/index.php/jpsp/article/view/13764


Tümay, H. (2016). Reconsidering learning difficulties and misconceptions in chemistry: emergence in chemistry and its implications for chemical education. Chemistry Education Research and Practice, 17(2), 229–245. https://doi.org/10.1039/C6RP00008H


Turkle, S. (1984). The Second Self: Computers and the Human Spirit. Simon and Schuster.


Yin, R. K. (2011). Applications of Case Study Research. SAGE Publications, Inc.


This paper is available on arxiv under CC BY-SA 4.0 DEED license.