Authors:
(1) Renato P. dos Santos, CIAGE – Centre for Generative Artificial Intelligence in Cognition and Education. Table of Links Abstract and Introduction Materials And Methods Results and Analyses Prompts and generated texts Conceptualizing chemical reactions Deepening on understanding of chemical reactions Question about combustion Question about a graph of gases turning into water over time Question about the difference between atoms, molecules, and moles Deepening on the concept of mole Question about changing of state Question about an animated representation of water molecules undergoing phase changes Question about plasma, a state of matter Question about chemical bondings Question about illustration of chemical bonds Question about the essence of the type of chemical bonding Further analysis Conclusions Limitations of the study and possible future studies Author Contributions, Conflicts of interest, Acknowledgements, and References Materials And Methods This study aimed to assess the efficacy of GenAIbots, including ChatGPT, Bing Chat, Bard, and Claude, as tools to augment Chemistry learning when characterised as agents-to-think-with. Chemistry students can harness the capabilities of GenAIbots in diverse ways to elevate their academic journey. They can pose complex questions to obtain thorough explanations on difficult subjects, leverage the bot as a supplemental aid for challenging areas, or seek visual aids—including images, graphs, simulations, or even interactive games—to enhance their understanding of chemical concepts. Additionally, they can seek clarity on laboratory protocols or engage in interactive discussions to solidify their understanding. In this article, we specifically hone in on GenAIbots' utility as agents-to-think-with, virtual intelligent tutors addressing students' doubts, rectifying misconceptions, and providing in-depth insights into complex concepts. Our focus also centres on the primary challenges encountered by Chemistry students, as highlighted by Timilsena et al. (2022), encompassing chemical reactions, chemical equilibrium, phase changes, gases, stoichiometry, atoms and molecules, acids and bases, and covalent bonding. We undertook a comprehensive single case study, aligning with Yin's (2011) guidelines, to evaluate and contrast the performance of ChatGPT, Bing Chat, Bard, and Claude. In analysing the responses of these AI systems, we adopted the Content Analysis methodology according to Bardin (1977) to delve into participants' experiences and viewpoints. This method offers insightful observations on student interactions with these GenAIbots. Prompting, the process of giving instructions, is crucial in interacting with language models. The process of meticulously precise, well-crafted sentences that elicit meaningful and valuable responses, or prompt-crafting, as Mishra et al. (2023) call it, is crucial for ensuring deep iterative reflections and engagements with GenAIbots as agents-to-think-with. Mollick (2023) offers a comprehensive guide to prompting, complete with detailed annotations and expanded examples. Two illustrative examples of prompt, crafted by ChatGPT itself, is presented below. The second one adds the intention of fostering profound iterative reflections and engagement in chemistry learning: Please provide a clear and comprehensible explanation of the various types of chemical bonding (ionic, covalent, and hydrogen), covering the theoretical concepts behind them and offering real-world examples suitable for an undergraduate student’s understanding. In the realm of chemistry, the concept of chemical bonding can be intricate, blending both theoretical principles and real-world applications. Walk me through the nuances of covalent, ionic, and hydrogen bonds. Additionally, challenge me with thought-provoking questions and analogies that can help solidify my understanding and encourage further exploration. The initial prompt in the upcoming dialogue, refined through a series of AI experiments, serves as another example of engaging in a meaningful conversation with the GenAIbot rather than asking isolated questions that might merely yield standard textbook responses. Nevertheless, while this initial prompt serves to 'set the stage' for the dialogue, the subsequent interactions with the simulated student are more straightforward and devoid of any elaborate prompting. This design reflects a typical student's approach: they may lack expertise in crafting detailed prompts and simply seek answers to their chemistry doubts. The dialogue intends to capture this organic flow of inquiry rather than a meticulously structured question sequence. GenAIbots can maintain context within a particular session, drawing upon and building on topics or responses discussed earlier. Even if a user returns to a previously accessed session that has not been deleted, the chatbot can still recall and refer to past interactions from that session. This continuity lets users pick up right where they left off, facilitating a more seamless and cohesive conversation. However, it's crucial to understand that this retention is exclusive to the individual session and doesn't transfer to separate chat sessions, ensuring user privacy and data security. Dunlop et al. (2020) highlight the absence of philosophical dialogue in Chemistry Education and propose its integration into Higher Education. GenAIbots like ChatGPT, Bing Chat, Bard, and Claude can facilitate Socratic-like, philosophical dialogues, encouraging user reflection, critical thinking and a deeper understanding of Chemistry's principles, assumptions, and ethical implications. GenAIbots can bridge the gap between theory and practice by promoting meaningful discussions and providing a holistic educational experience. Using GenAIbots as agents-to-think-with is new and innovative. Consequently, finding teachers willing to incorporate it into their classrooms for experimental purposes has been challenging. Due to this, the investigation did not involve real students. Instead, the researcher, acting as a STEM teacher at a Brazilian university, assumed the role of a student (identified as P1) in the interaction sessions. Empathy was incorporated into the research methodology to enable the researchers to act like students and understand their perspectives. Four sessions were held, each emulating the learning experiences of Chemistry students using ChatGPT, Bing Chat, Bard, or Claude and focusing on their main difficulties, as discussed above. After the initial prompt to engage in a meaningful conversation with the GenAIbot and characterise the agents intervening in the dialogue, a simulated student begins in the dialogues with the widely recognised difficulty of balancing chemical reactions. The discussion then evolves in a somewhat unstructured order, where the response to the previous one inspires each new question. Data collection involved GenAIbot interaction logs, which recorded the participant's prompts and responses, and reflective journals maintained by the participant. These materials were later analysed to identify recurring themes and patterns, shedding light on the perceived impact of GenAIbots as an agent-to-think-with in Chemistry learning. From the initial readings of the participant's prompts and responses and reflective journals, the following seven categories of analysis emerged: 1. Instructional Strategies: o Methods used by AI to impart knowledge. 2. Engagement & Interactivity: o Prompting critical thinking and further exploration. o Asking questions, inviting responses, and encouraging deeper dives. 3. Use of Analogy & Comparative Illustrations: o Using parallels, metaphors, and real-world examples to explain concepts. 4. Reinforcement & Feedback: o Recognizing and affirming user input. o Correcting or building upon user understanding. o Reiteration of essential concepts. 5. Detail Depth & Content Recommendation: o The extent of detail provided, whether it's concise or expansive. o Suggesting additional learning materials and providing relevant content. 6. Personalization & Personable Touch: o Tailoring responses to individual needs. o Using emoticons, casual tones, or other elements for a more personalised feel. 7. Meta-discussion & Perspective Framing: o Addressing the broader relevance of concepts outside strict contexts. o Contextualizing abstract concepts using familiar frameworks. These categories will be used to comparatively analyse the answers of each GenAIbot to the individual participant’s questions below. This paper is available on arxiv under CC BY-SA 4.0 DEED license. Authors: (1) Renato P. dos Santos, CIAGE – Centre for Generative Artificial Intelligence in Cognition and Education. Authors: Authors: (1) Renato P. dos Santos, CIAGE – Centre for Generative Artificial Intelligence in Cognition and Education. Table of Links Abstract and Introduction Abstract and Introduction Materials And Methods Materials And Methods Results and Analyses Results and Analyses Prompts and generated texts Prompts and generated texts Conceptualizing chemical reactions Conceptualizing chemical reactions Deepening on understanding of chemical reactions Deepening on understanding of chemical reactions Question about combustion Question about combustion Question about a graph of gases turning into water over time Question about a graph of gases turning into water over time Question about the difference between atoms, molecules, and moles Question about the difference between atoms, molecules, and moles Deepening on the concept of mole Deepening on the concept of mole Question about changing of state Question about changing of state Question about an animated representation of water molecules undergoing phase changes Question about an animated representation of water molecules undergoing phase changes Question about plasma, a state of matter Question about plasma, a state of matter Question about chemical bondings Question about chemical bondings Question about illustration of chemical bonds Question about illustration of chemical bonds Question about the essence of the type of chemical bonding Question about the essence of the type of chemical bonding Further analysis Further analysis Conclusions Conclusions Limitations of the study and possible future studies Limitations of the study and possible future studies Author Contributions, Conflicts of interest, Acknowledgements, and References Author Contributions, Conflicts of interest, Acknowledgements, and References Materials And Methods This study aimed to assess the efficacy of GenAIbots, including ChatGPT, Bing Chat, Bard, and Claude, as tools to augment Chemistry learning when characterised as agents-to-think-with. Chemistry students can harness the capabilities of GenAIbots in diverse ways to elevate their academic journey. They can pose complex questions to obtain thorough explanations on difficult subjects, leverage the bot as a supplemental aid for challenging areas, or seek visual aids—including images, graphs, simulations, or even interactive games—to enhance their understanding of chemical concepts. Additionally, they can seek clarity on laboratory protocols or engage in interactive discussions to solidify their understanding. In this article, we specifically hone in on GenAIbots' utility as agents-to-think-with, virtual intelligent tutors addressing students' doubts, rectifying misconceptions, and providing in-depth insights into complex concepts. Our focus also centres on the primary challenges encountered by Chemistry students, as highlighted by Timilsena et al. (2022), encompassing chemical reactions, chemical equilibrium, phase changes, gases, stoichiometry, atoms and molecules, acids and bases, and covalent bonding. We undertook a comprehensive single case study, aligning with Yin's (2011) guidelines, to evaluate and contrast the performance of ChatGPT, Bing Chat, Bard, and Claude. In analysing the responses of these AI systems, we adopted the Content Analysis methodology according to Bardin (1977) to delve into participants' experiences and viewpoints. This method offers insightful observations on student interactions with these GenAIbots. Prompting, the process of giving instructions, is crucial in interacting with language models. The process of meticulously precise, well-crafted sentences that elicit meaningful and valuable responses, or prompt-crafting, as Mishra et al. (2023) call it, is crucial for ensuring deep iterative reflections and engagements with GenAIbots as agents-to-think-with. Mollick (2023) offers a comprehensive guide to prompting, complete with detailed annotations and expanded examples. Two illustrative examples of prompt, crafted by ChatGPT itself, is presented below. The second one adds the intention of fostering profound iterative reflections and engagement in chemistry learning: Please provide a clear and comprehensible explanation of the various types of chemical bonding (ionic, covalent, and hydrogen), covering the theoretical concepts behind them and offering real-world examples suitable for an undergraduate student’s understanding. Please provide a clear and comprehensible explanation of the various types of chemical bonding (ionic, covalent, and hydrogen), covering the theoretical concepts behind them and offering real-world examples suitable for an undergraduate student’s understanding. In the realm of chemistry, the concept of chemical bonding can be intricate, blending both theoretical principles and real-world applications. Walk me through the nuances of covalent, ionic, and hydrogen bonds. Additionally, challenge me with thought-provoking questions and analogies that can help solidify my understanding and encourage further exploration. In the realm of chemistry, the concept of chemical bonding can be intricate, blending both theoretical principles and real-world applications. Walk me through the nuances of covalent, ionic, and hydrogen bonds. Additionally, challenge me with thought-provoking questions and analogies that can help solidify my understanding and encourage further exploration. The initial prompt in the upcoming dialogue, refined through a series of AI experiments, serves as another example of engaging in a meaningful conversation with the GenAIbot rather than asking isolated questions that might merely yield standard textbook responses. Nevertheless, while this initial prompt serves to 'set the stage' for the dialogue, the subsequent interactions with the simulated student are more straightforward and devoid of any elaborate prompting. This design reflects a typical student's approach: they may lack expertise in crafting detailed prompts and simply seek answers to their chemistry doubts. The dialogue intends to capture this organic flow of inquiry rather than a meticulously structured question sequence. GenAIbots can maintain context within a particular session, drawing upon and building on topics or responses discussed earlier. Even if a user returns to a previously accessed session that has not been deleted, the chatbot can still recall and refer to past interactions from that session. This continuity lets users pick up right where they left off, facilitating a more seamless and cohesive conversation. However, it's crucial to understand that this retention is exclusive to the individual session and doesn't transfer to separate chat sessions, ensuring user privacy and data security. Dunlop et al. (2020) highlight the absence of philosophical dialogue in Chemistry Education and propose its integration into Higher Education. GenAIbots like ChatGPT, Bing Chat, Bard, and Claude can facilitate Socratic-like, philosophical dialogues, encouraging user reflection, critical thinking and a deeper understanding of Chemistry's principles, assumptions, and ethical implications. GenAIbots can bridge the gap between theory and practice by promoting meaningful discussions and providing a holistic educational experience. Using GenAIbots as agents-to-think-with is new and innovative. Consequently, finding teachers willing to incorporate it into their classrooms for experimental purposes has been challenging. Due to this, the investigation did not involve real students. Instead, the researcher, acting as a STEM teacher at a Brazilian university, assumed the role of a student (identified as P1) in the interaction sessions. Empathy was incorporated into the research methodology to enable the researchers to act like students and understand their perspectives. Four sessions were held, each emulating the learning experiences of Chemistry students using ChatGPT, Bing Chat, Bard, or Claude and focusing on their main difficulties, as discussed above. After the initial prompt to engage in a meaningful conversation with the GenAIbot and characterise the agents intervening in the dialogue, a simulated student begins in the dialogues with the widely recognised difficulty of balancing chemical reactions. The discussion then evolves in a somewhat unstructured order, where the response to the previous one inspires each new question. Data collection involved GenAIbot interaction logs, which recorded the participant's prompts and responses, and reflective journals maintained by the participant. These materials were later analysed to identify recurring themes and patterns, shedding light on the perceived impact of GenAIbots as an agent-to-think-with in Chemistry learning. From the initial readings of the participant's prompts and responses and reflective journals, the following seven categories of analysis emerged: 1. Instructional Strategies: 1. Instructional Strategies: o Methods used by AI to impart knowledge. 2. Engagement & Interactivity: 2. Engagement & Interactivity: o Prompting critical thinking and further exploration. o Asking questions, inviting responses, and encouraging deeper dives. 3. Use of Analogy & Comparative Illustrations: 3. Use of Analogy & Comparative Illustrations: o Using parallels, metaphors, and real-world examples to explain concepts. 4. Reinforcement & Feedback: 4. Reinforcement & Feedback: o Recognizing and affirming user input. o Correcting or building upon user understanding. o Reiteration of essential concepts. 5. Detail Depth & Content Recommendation: 5. Detail Depth & Content Recommendation: o The extent of detail provided, whether it's concise or expansive. o Suggesting additional learning materials and providing relevant content. 6. Personalization & Personable Touch: 6. Personalization & Personable Touch: o Tailoring responses to individual needs. o Using emoticons, casual tones, or other elements for a more personalised feel. 7. Meta-discussion & Perspective Framing: 7. Meta-discussion & Perspective Framing: o Addressing the broader relevance of concepts outside strict contexts. o Contextualizing abstract concepts using familiar frameworks. These categories will be used to comparatively analyse the answers of each GenAIbot to the individual participant’s questions below. This paper is available on arxiv under CC BY-SA 4.0 DEED license. This paper is available on arxiv under CC BY-SA 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Analyzing GenAIbots' Efficacy in Chemistry Education

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

Enhancing Chemistry Learning with ChatGPT, Bing Chat, Bard, and Claude as Agents-to-Think-With

GenAIbot Conversations in British vs. American English

Initiating Socratic Dialogues with GenAIbots in Chemistry Education

How GenAIbots Guide Students Through Balancing Equations

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

Enhancing Chemistry Learning with ChatGPT, Bing Chat, Bard, and Claude as Agents-to-Think-With

GenAIbot Conversations in British vs. American English

Initiating Socratic Dialogues with GenAIbots in Chemistry Education

How GenAIbots Guide Students Through Balancing Equations

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps