Analyzing GenAIbots' Efficacy in Chemistry Education

Table of Links

Abstract and Introduction

Materials And Methods

Results and Analyses

Prompts and generated texts

Conceptualizing chemical reactions

Deepening on understanding of chemical reactions

Question about combustion

Question about a graph of gases turning into water over time

Question about the difference between atoms, molecules, and moles

Deepening on the concept of mole

Question about changing of state

Question about an animated representation of water molecules undergoing phase changes

Question about plasma, a state of matter

Question about chemical bondings

Question about illustration of chemical bonds

Question about the essence of the type of chemical bonding

Further analysis

Conclusions

Limitations of the study and possible future studies

Author Contributions, Conflicts of interest, Acknowledgements, and References

Materials And Methods

This study aimed to assess the efficacy of GenAIbots, including ChatGPT, Bing Chat, Bard, and Claude, as tools to augment Chemistry learning when characterised as agents-to-think-with.

Chemistry students can harness the capabilities of GenAIbots in diverse ways to elevate their academic journey. They can pose complex questions to obtain thorough explanations on difficult subjects, leverage the bot as a supplemental aid for challenging areas, or seek visual aids—including images, graphs, simulations, or even interactive games—to enhance their understanding of chemical concepts. Additionally, they can seek clarity on laboratory protocols or engage in interactive discussions to solidify their understanding. In this article, we specifically hone in on GenAIbots' utility as agents-to-think-with, virtual intelligent tutors addressing students' doubts, rectifying misconceptions, and providing in-depth insights into complex concepts.

Our focus also centres on the primary challenges encountered by Chemistry students, as highlighted by Timilsena et al. (2022), encompassing chemical reactions, chemical equilibrium, phase changes, gases, stoichiometry, atoms and molecules, acids and bases, and covalent bonding.

We undertook a comprehensive single case study, aligning with Yin's (2011) guidelines, to evaluate and contrast the performance of ChatGPT, Bing Chat, Bard, and Claude. In analysing the responses of these AI systems, we adopted the Content Analysis methodology according to Bardin (1977) to delve into participants' experiences and viewpoints. This method offers insightful observations on student interactions with these GenAIbots.

Prompting, the process of giving instructions, is crucial in interacting with language models. The process of meticulously precise, well-crafted sentences that elicit meaningful and valuable responses, or prompt-crafting, as Mishra et al. (2023) call it, is crucial for ensuring deep iterative reflections and engagements with GenAIbots as agents-to-think-with. Mollick (2023) offers a comprehensive guide to prompting, complete with detailed annotations and expanded examples. Two illustrative examples of prompt, crafted by ChatGPT itself, is presented below. The second one adds the intention of fostering profound iterative reflections and engagement in chemistry learning:

Please provide a clear and comprehensible explanation of the various types of chemical bonding (ionic, covalent, and hydrogen), covering the theoretical concepts behind them and offering real-world examples suitable for an undergraduate student’s understanding.

In the realm of chemistry, the concept of chemical bonding can be intricate, blending both theoretical principles and real-world applications. Walk me through the nuances of covalent, ionic, and hydrogen bonds. Additionally, challenge me with thought-provoking questions and analogies that can help solidify my understanding and encourage further exploration.

The initial prompt in the upcoming dialogue, refined through a series of AI experiments, serves as another example of engaging in a meaningful conversation with the GenAIbot rather than asking isolated questions that might merely yield standard textbook responses. Nevertheless, while this initial prompt serves to 'set the stage' for the dialogue, the subsequent interactions with the simulated student are more straightforward and devoid of any elaborate prompting. This design reflects a typical student's approach: they may lack expertise in crafting detailed prompts and simply seek answers to their chemistry doubts. The dialogue intends to capture this organic flow of inquiry rather than a meticulously structured question sequence.

GenAIbots can maintain context within a particular session, drawing upon and building on topics or responses discussed earlier. Even if a user returns to a previously accessed session that has not been deleted, the chatbot can still recall and refer to past interactions from that session. This continuity lets users pick up right where they left off, facilitating a more seamless and cohesive conversation. However, it's crucial to understand that this retention is exclusive to the individual session and doesn't transfer to separate chat sessions, ensuring user privacy and data security.

Dunlop et al. (2020) highlight the absence of philosophical dialogue in Chemistry Education and propose its integration into Higher Education. GenAIbots like ChatGPT, Bing Chat, Bard, and Claude can facilitate Socratic-like, philosophical dialogues, encouraging user reflection, critical thinking and a deeper understanding of Chemistry's principles, assumptions, and ethical implications. GenAIbots can bridge the gap between theory and practice by promoting meaningful discussions and providing a holistic educational experience.

Using GenAIbots as agents-to-think-with is new and innovative. Consequently, finding teachers willing to incorporate it into their classrooms for experimental purposes has been challenging. Due to this, the investigation did not involve real students. Instead, the researcher, acting as a STEM teacher at a Brazilian university, assumed the role of a student (identified as P1) in the interaction sessions. Empathy was incorporated into the research methodology to enable the researchers to act like students and understand their perspectives.

Four sessions were held, each emulating the learning experiences of Chemistry students using ChatGPT, Bing Chat, Bard, or Claude and focusing on their main difficulties, as discussed above. After the initial prompt to engage in a meaningful conversation with the GenAIbot and characterise the agents intervening in the dialogue, a simulated student begins in the dialogues with the widely recognised difficulty of balancing chemical reactions. The discussion then evolves in a somewhat unstructured order, where the response to the previous one inspires each new question.

Data collection involved GenAIbot interaction logs, which recorded the participant's prompts and responses, and reflective journals maintained by the participant. These materials were later analysed to identify recurring themes and patterns, shedding light on the perceived impact of GenAIbots as an agent-to-think-with in Chemistry learning.

From the initial readings of the participant's prompts and responses and reflective journals, the following seven categories of analysis emerged:

1. Instructional Strategies:

o Methods used by AI to impart knowledge.

2. Engagement & Interactivity:

o Prompting critical thinking and further exploration.

o Asking questions, inviting responses, and encouraging deeper dives.

3. Use of Analogy & Comparative Illustrations:

o Using parallels, metaphors, and real-world examples to explain concepts.

4. Reinforcement & Feedback:

o Recognizing and affirming user input.

o Correcting or building upon user understanding.

o Reiteration of essential concepts.

5. Detail Depth & Content Recommendation:

o The extent of detail provided, whether it's concise or expansive.

o Suggesting additional learning materials and providing relevant content.

6. Personalization & Personable Touch:

o Tailoring responses to individual needs.

o Using emoticons, casual tones, or other elements for a more personalised feel.

7. Meta-discussion & Perspective Framing:

o Addressing the broader relevance of concepts outside strict contexts.

o Contextualizing abstract concepts using familiar frameworks.

These categories will be used to comparatively analyse the answers of each GenAIbot to the individual participant’s questions below.