This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Linda Pigureddu, University of Turin, Italy, [email protected];
(2) Cristina Gena, Dept. of Computer Science, University of Turin, Italy, [email protected].
Table of Links
THE PROJECT
The laboratory
Pepper was placed, in the role of mediator and dialogue partner, within the therapeutic laboratory[2] to promote autonomy and functional acquisition in autistic children with low support needs (level 1 support according to DSM-5 [1]). During the laboratory, conducted between the end of February 2021 and the beginning of June 2021, the exchanges, and interactions between autistic children in rehabilitation settings were explored, supported by the robot Pepper aiming to create an environment that would stimulate the development of social and communication skills, as well as strengthen the acquisition of strategies and autonomies related to daily activities, such as snack preparation [14]–[17].
Four autistic children with low support needs (level 1 support), aged 11 to 14 years, participated in the lab. During the lab sessions, the children had the opportunity to interact both directly with the robot and to participate in group activities, guided and coordinated by Pepper itself and the specialized operators, including educators, speech therapists, and psychologists [14]–[17].
The workshop sessions (16 in total), which were held weekly with a duration of two hours each, were recorded via a fixed camera, in addition to Pepper's 2D cameras (forehead camera and tablet camera), to allow post-hoc analysis to be conducted. During the meetings, two trainee students filled out evaluation forms provided by the psychotherapists, noting the progress of the sessions in a diary. These notes were later reworked to make reports, shared within the working group, reflecting the children's evolving behavior during their interactions with the robot [14]–[17]. To ensure the children's privacy, the "PepperForAutism" application was implemented in the robot during data collection, which consists of three main modules (Android app, web platform and backend) integrated into Jumple SRL's Pyramid platform. This system enabled the Pepper robot to uniquely identify the child with whom it was interacting, allowing data to be collected anonymously through association with a random ID known only to authorized individuals [14]–[17].
1.1. The lab's sessions in details
The sessions, lasting about two hours, were conducted by one or two specialized operators (educators, speech therapists, psychologists, etc.) assisted by two or three trainees. These were held in a room furnished with table and chairs and equipped with the tools necessary to carry out the activities, inside an apartment located in the center of Turin, made available by the Paideia Foundation also equipped with a bathroom, living room and kitchen. During the sessions, each child was given the opportunity to interact directly with Pepper and perform the activities planned for the group through the directions provided by the robot, with the help of the specialized operators and in a real context setting to leverage the benefit of a real-world evaluation [23]. Each session was phased and was conducted according to the following structure:
1. Welcome: children were made to sit in the apartment and welcomed by the therapists and the robot who, at the beginning of each session, provided directions on how to prepare to participate in the workshop.
2. First social time: dialogue session with the robot on a pre-established topic of interest (e.g., music, video games, etc.). The conversation is designed to be initialized by both the children and the robot. In the first case, "Getting Acquainted", the children would stand in front of the robot and, after waiting for it to catch their presence, proceed to ask a question of any sort. In the second case, identified as "Making Friends," Pepper would call the children one at a time to suggest a topic of conversation to each, according to a script updated weekly, in the expectation of engaging the other participants in a group conversation.
3. Development of practical skills (snack preparation and/or homework execution): Pepper played the role of activity coordinator, offering snack preparation instructions and explanatory videos displayed on the built-in tablet, in to guide the children in performing the task, with the therapists ready to intervene if needed. Starting from the 11th meeting, this phase was enriched by the inclusion of the activity of doing school homework. The children, who had been previously warned about the need to come equipped with their own school supplies, were asked to consult their journal to select a subject whose homework they wished to complete with the assistance of Pepper, who, after identifying a range of topics, proceeded by providing information on what materials were necessary or optional to proceed effectively with the task they chose. Again, the use of explanatory videos was provided.
4. Second social moment: post snack (or post homework) dialogue. The conversation is guided in the mode ("Moment of Knowledge" or "Making Friends") not explored in the First Social Moment.
5. Final Feedback: Before ending the session with greetings, Pepper asks the children to give their opinions about the meeting. In this phase, children were asked to stand in front of the robot and state their mood, choosing from the illustrated faces on the integrated tablet suitable for representing sadness, neutrality, and happiness. Because the workshop took place during a period characterized by social distancing due to the Covid-19 pandemic, children were required to wear a clear visor as an alternative to the surgical mask so that Pepper could take a picture, which was later used by the trainees to assess the consistency of the emotion indicated by the child with his or her facial expressions and the mood inferred by the robot [16].
1.2. Welcome
At the beginning of each meeting, Pepper welcomed the children and provided them with instructions for getting ready for and participating in the lab. These instructions included descriptive phrases and directives similar to those found in social stories, with the purpose of guiding and supporting the children during the activities. This phase is the one that produced the most noticeably positive results. Although Pepper offered to repeat the welcome directions each time, the children demonstrated that they remembered from one encounter to the next the rules of the lab and were able to correctly follow the instructions to take off their shoes and coats and to sanitize their hands before entering the lab, while still benefiting from the psychologist's help when necessary [17].
Starting therapeutic sessions in this way is both functional for the organizational and managerial purposes of the lab itself, and in line with the natural inclination of autistic minds to seek order and behavioral patterns that facilitate schematization of routines, predictability of events, and, in general, repetition [1]. This choice offers participants a sensitive approach toward the peculiarities of their minds, allowing them to feel immediately at ease, making it easier to introduce the activities planned for the day, and encourages children to be genuine and spontaneous, immediately comforting them by implicitly declaring that they are entering a safe environment, designed specifically to support and stimulate their minds, with no prejudice about their frailties.
1.3. Practical skills training
At this stage, the robot is assigned to coordinate the activity of snack preparation, offering snacks with recipes that were from time to time more and more complex, listing ingredients, encouraging children to get organized, providing guidance on the procedures both verbally and using pictures, animations and demonstration videos played on the built-in tablet, and giving children the opportunity to review the steps if necessary. During this activity, provision was made for Pepper to enter Waiting Mode, both to allow the children time to carry out the assignments and the opportunity for the group to spontaneously chat during the meal. On most occasions, the children took advantage of this time to share their suggestion for questions to ask Pepper during the conversational moments, and about activities and topics they would like to cover and proposal to improve the robot’s dialogue structure to make it more friendly and pleasant to talk with. This was also because the time spent with Pepper was perceived as valuable, and the children were inclined to focus on it even when they were not communicating with it directly since they felt empowered with the responsibility and opportunity to contribute to the improvement of the robot.
The snack time produced good results and, in addition to being evidently stimulating for the young participants, was also one of their favorite moments; in fact, in most of the encounters they asked to start the lab session with this activity. The children showed appreciation for Pepper's help and, from the very beginning, good levels of autonomy, although the excitement associated with this time often affected the ability to maintain high concentration levels, as indicated by the scores given by the operatives [17].
Beginning in the eleventh meeting, the school homework activity was introduced, in which Pepper offered the children directions on what tools would be needed and tricks for doing the task they chose, always accompanying the verbal instructions with multimedia elements played on the integrated tablet when needed [17]. Regarding this phase, there was not sufficient data collected to assess its effectiveness, but it is still possible to consider the direct feedback from the children, who repeatedly and openly stated that they did not want to «waste» their time in the lab with ordinary activities, suggesting alternative skill-building activities (e.g., chemistry or robotics experiments), while only on one occasion one child did say that he appreciated the time devoted to doing homework, as it allowed him to associate the lab with the environment more familiar to him, of school.
1.4. Conversations
During the dialogue moments, the children were directly asked whether they wanted to "get acquainted" or "make friends" with the robot. In the former, they were given the opportunity to ask questions of their interest directly to Pepper, while in the latter, it was Pepper who asked them questions about their interests so as to encourage a collective conversation on the topic. Unfortunately, both modes did not produce satisfactory results, as they were too far removed from the human conversational style [20], consisting of simple repartee, without the possibility of in depth or direct conversation development with the robot, due to the lack of an open
conversational system that would allow the robot to use a communicative style more similar to natural language [17].
Giving children the opportunity to ask questions to the robot at a time when it lacked the ability to draw on ontologies to respond but had a dialogue system based on the manual and weekly update of its script, has produced various difficulties during the conversational moments. apart from not taking into account the difficulty of autistic people in initializing conversations [21], has produced several situations of misunderstanding, in which the robot, for technical difficulties (e.g. background noise, speech defects, etc.) or the absence of the input provided by the child within its script, it found itself unable to provide a coherent and satisfactory response [17].
After few interactions, the children began to ask themselves about Pepper with derisive attitude, deliberately asking nonsensical questions attempting to get funny answers or to evaluate how far they could go before sending cause a malfunction. An example of mocking interaction that has produced a lot of hilarity and has been revived during several meetings is related to the understanding by the children that pronounce at the beginning of the question the keywords «have you ever been in...» allowed access to the standard answer: «no, but I would like to visit the whole world», even though the question was not completed with a real location, prompting participants to ask if Pepper had ever visited the Moon, heaven or the land of the stupidity. As per the goal of stimulating the social interaction between the participants, the technical difficulties that emerged during this phase encouraged the children's curiosity about the robot, encouraging collective discussion about how Pepper works and brainstorming suggestions for improvements to make it a more enjoyable conversation partner. On the other hand, the analysis of these phases, despite not having produced satisfactory therapeutic results, is extremely interesting to conduct an evaluation of interactions aimed at improving the dialogue system, pointing out that, even before technical problems, communication with the robot is deficient because it is not fluent in the slang used by children, compromising its position as a mediator because it is unable to act as an interpreter between children and specialized adults working in the laboratory.
[2] Ethical approval for this study was obtained from the bioethical committee of the University of Turin, with approval number: 0664572.
