Researchers Work With Therapists to Use Video Games to Improve Physical Rehab

Authors: (1) Antoine Loriette, IRCAM, CNRS, Sorbonne Universite, Paris, France (antoine.loriette@gmail.com); (2) Baptiste Caramiaux, Sorbonne Universite, CNRS, ISIR, Paris, France (caramiaux@isir.upmc.fr); (3) Sebastian Stein, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (sebastian.stein@glasgow.ac.uk); (4) John H. Williamson, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (johnh.williamson@glasgow.ac.uk). Table of Links Abstract and 1 Introduction 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 4 Experimental Study 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 7 Conclusion 8 Acknowledgements and References 3 Motivation from a rehabilitation context The present work is grounded by field work with a team of occupational therapists (OTs) from the spinal injury unit of Queen Elizabeth University Hospital (Loriette, 2019). They were interested in improving their process for upper limb reach rehabilitation, which has been less explored than lower limb treatment (Barrett et al., 2016). 3.1 Observations and Brainstorming workshops The present collaboration started with a discussion related to the nature of OTs’ patients and moved onto focus on their rehabilitation practises. Their target group gathers patients with high level of spinal cord injuries (SCI), typically with spinal lesions at levels C5 and C6 (Jr et al., 1997), for whom the arm functions are severely compromised (Anderson, 2004). The induced partial or complete paralysis of specific arm muscles reduces patients’ ability to perform coordinated motions. During the first stages of rehabilitation in the weeks following injury, when fine motor control of the fingers is absent, functional rehabilitation goals were limited to whole arm motions over tabletops. In routine rehabilitation, OTs used two physical artefacts. A skateboard, onto which one arm of the patient is strapped, was used to elicit linear back and forth motions while supporting the patient’s arm, and facilitating movement through the reduced friction of the skateboard’s wheels. The other artefact was a plastic tube rolled on its long dimension with patient’s arm placed on top. A Velcro tape attached to the tube and to the table was sometimes used to increase friction as a mean to adapt task difficulty. For both devices, typical instructions from the OTs were for the patients to “move towards the window” or to “reach forward”, involving four gestures in total: forward, backward, left, right on a tabletop. However, the simplistic, repetitive and artificial nature of these exercises led to low patient motivation, and they were described as “dull, tedious and boring” by the OTs. Given the positive results reported in the literature regarding the use of games for rehabilitation (Barrett et al., 2016), this involvement of games was discussed between us (HCI practitioners) and the OTs during a series of meetings. These discussions were helpful in two ways. They helped in defining the main requirements by the OTs for a system which elicits the same types of movements as the ones exerted using the two artefacts introduced previously. And they helped in understanding the extent to which they can be parameterised in order to calibrate the level of difficulty. A two-hours participatory workshop was then organised at the hospital with five OTs and two of the authors. The workshop was split equally into two parts. We first engaged in a brainstorming session to find mappings between game actions and rehabilitation movements. The list of rehabilitation movements was matched with a list of candidates for the games. When prompted for simple games they were familiar with, the participants mentioned Whack-a-Mole, Bubble Witch, Pong and Frogger to name a few, all of which were arcade games. The controls needed to implement their gameplay were then discussed. While some required four ways directional actions (e.g. Pac-Man), for aiming at a target for example, others only required button presses for triggering specific actions (e.g. Puzzle Bubble). To simplify the workshop complexity, it was decided that games with simple 4-ways controls should be focused on. After the brainstorming, the second discussion addressed the parameterisation of specific exercises the OTs would like to see implemented by a gameplay. The movements that patients were producing when interacting with the skateboard, mentioned earlier, were used as a reference. A paper prototype of actionable controls was created for the interaction to be played out on the desk around which the workshop took place. The instructions from the OTs were to perform long reaching motions with the arm in contact with a planar surface. The goal they pursued was for their patients to reach always further. The flexibility of the paper prototype highlighted that the possibility of actionable controls whose locations on the surface could be easily modified was important for exercise adaptation. The outcome of these workshops was then threefold: • We identified that the game itself was not the focus for either parties (OTs and ourselves), as long as it is enjoyable for the patient and complies to the specifications given by OTs; • We sketched/agreed on a control input with 4-ways actionable buttons to interface with various different games; • We chose a rehabilitation parameter: the position of the gamepad’s buttons. 3.2 System description The preliminary field work presented above has led to the prototype depicted in Figure 1 (left). A patient is seated by a tabletop on top of which reaching motions are performed with one arm. An optical device both tracks the patient’s hand and creates a virtual surface with four regions arranged in a cross shape. Each virtual button is activated when the hand of the user enters its respective region. This new input control is interfaced with a digital game running in front of the patient. The gestures required by the gameplay are meant to be similar to the one’s needed for the patient’s regular exercising. As for the game, we chose Pac-Man (Namco, 1980) as test game. It is compatible with the four-way control and it is one of the most iconic arcade games where the rules are known by many. Regions of the virtual surface were mapped to directional commands (UP, RIGHT, DOWN, LEFT). Pac-Man always move forward at a given speed, the commands are only used to make directional changes. The prototype was initially set up in the authors’ lab on a 78cm wide office desk. We used an Optitrack system for the tracking of the participants hand. A single Infrared (IR) reflective marker, placed on the user’s hand between the index and middle fingers, indicated the (x, y) location of the hand on the surface. A dedicated computer ran the Optitrack system whose output was transmitted to another machine running the game in a browser. We used an opensource version of Pac-Man1 . Translating (x, y) coordinates of the hand to game commands was done with custom software on the machine running the game. The origin of the controls was indicated by a protruding tactile marker. Directional commands were triggered when the tracked hand location entered the triangular control regions, which were not visible to the user. Audio feedback indicated when commands were triggered with a bip. Users would be equipped with headphones during use to ensure they could clearly hear this feedback and the in-game audio. 3.3 Interaction parameterisation Based on the field work and the implemented system, interaction parameters have been identified in order to act upon the game difficulty and rehabilitation effectiveness. A first parameter is the spread parameter, denoted r, defined as the distance between the inner tips of the triangular actionable regions (Figure 1 (right)). This parameter serves the purpose of varying task difficulty. This parameter was actually required by the OTs as a lever to adapt the game difficulty to the patient’s needs. We conducted a first round of informal tests with members of our lab, revealing that the overall interaction difficulty was greatly increased due to the new input control modality (from button presses to reaching arm movements). We observed that the change in game difficulty incurred by the change of input control modality cannot be accommodated by the spread parameter alone. The latency and responsiveness of control is impacted by optical tracking delay and the effective reaching time necessary to move between regions. To counterbalance this effect, we identified a second interaction parameter controlling the speed of the game, called Time Rate. This is similar to what games like Tetris use to set difficulty without altering other part of the gameplay mechanics and was for example recommended by Burke et al. (Burke et al., 2009) in their guidelines. It also has the benefit to limit game alterations to a unique and controllable parameter time rate Tr. By reducing the time rate, the number of actions per unit of time required for playing are reduced, facilitating the interaction. The proposed interaction parameterisation can be represented as a block diagram (Figure 2). The perception loop includes the user, an input device and a game and produces engagement, depicted as a thick black line on the diagram. The interaction can be affected by two design parameters: the scale r, governing the distance that the participant has to travel with the arm in order to trigger a command; the time rate Tr, governing the speed of the game. Both parameters relate to game difficulty, hence a significant impact on game engagement can be expected. As a matter of fact, providing a well-calibrated challenge is key to an engaging game experience. As phrased by Przybylski et al. (Przybylski et al., 2010), the “mastery of controls plays an important role in game motivation, largely as a necessary, but not sufficient, condition for achieving psychologically need-satisfying play.” 3.4 Summary The co-design activities presented in this section led to the design of a rehabilitation system, based on a new gesturebased input device and the game of Pac-Man. We have conducted preliminary informal testing, including with participants having sustained a SCI. In the remaining of the paper, we consider the problem of building a computational model of user performance while using the presented system. As introduced earlier, our motivation is to provide a more systematic method than heuristics in order to help adjusting the interaction parameters to define the game difficulty. Our approach relies on the estimation of the distance between a current performance and a baseline performance (reference). We split the problem in two: 1. Understanding the impact of the identified parameters on non-disabled participants’ performance while using the game. The fact to perform the study on non-disabled participants is motivated by the need to set a baseline performance and how parameters affect this baseline performance (as further developed in Section 4) 2. Proposing a behavioural model able to predict participants’ performance with respect to a baseline (Section 5). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Antoine Loriette, IRCAM, CNRS, Sorbonne Universite, Paris, France (antoine.loriette@gmail.com); (2) Baptiste Caramiaux, Sorbonne Universite, CNRS, ISIR, Paris, France (caramiaux@isir.upmc.fr); (3) Sebastian Stein, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (sebastian.stein@glasgow.ac.uk); (4) John H. Williamson, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (johnh.williamson@glasgow.ac.uk). Authors: Authors: (1) Antoine Loriette, IRCAM, CNRS, Sorbonne Universite, Paris, France (antoine.loriette@gmail.com); (2) Baptiste Caramiaux, Sorbonne Universite, CNRS, ISIR, Paris, France (caramiaux@isir.upmc.fr); (3) Sebastian Stein, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (sebastian.stein@glasgow.ac.uk); (4) John H. Williamson, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (johnh.williamson@glasgow.ac.uk). Table of Links Abstract and 1 Introduction 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 4 Experimental Study 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 7 Conclusion 8 Acknowledgements and References Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 2.2 Input control performance measure and modelling and 2.3 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 4 Experimental Study 4 Experimental Study 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 7 Conclusion 8 Acknowledgements and References 7 Conclusion 8 Acknowledgements and References 8 Acknowledgements and References 3 Motivation from a rehabilitation context The present work is grounded by field work with a team of occupational therapists (OTs) from the spinal injury unit of Queen Elizabeth University Hospital (Loriette, 2019). They were interested in improving their process for upper limb reach rehabilitation, which has been less explored than lower limb treatment (Barrett et al., 2016). 3.1 Observations and Brainstorming workshops The present collaboration started with a discussion related to the nature of OTs’ patients and moved onto focus on their rehabilitation practises. Their target group gathers patients with high level of spinal cord injuries (SCI), typically with spinal lesions at levels C5 and C6 (Jr et al., 1997), for whom the arm functions are severely compromised (Anderson, 2004). The induced partial or complete paralysis of specific arm muscles reduces patients’ ability to perform coordinated motions. During the first stages of rehabilitation in the weeks following injury, when fine motor control of the fingers is absent, functional rehabilitation goals were limited to whole arm motions over tabletops. In routine rehabilitation, OTs used two physical artefacts. A skateboard, onto which one arm of the patient is strapped, was used to elicit linear back and forth motions while supporting the patient’s arm, and facilitating movement through the reduced friction of the skateboard’s wheels. The other artefact was a plastic tube rolled on its long dimension with patient’s arm placed on top. A Velcro tape attached to the tube and to the table was sometimes used to increase friction as a mean to adapt task difficulty. For both devices, typical instructions from the OTs were for the patients to “move towards the window” or to “reach forward”, involving four gestures in total: forward, backward, left, right on a tabletop. However, the simplistic, repetitive and artificial nature of these exercises led to low patient motivation, and they were described as “dull, tedious and boring” by the OTs. Given the positive results reported in the literature regarding the use of games for rehabilitation (Barrett et al., 2016), this involvement of games was discussed between us (HCI practitioners) and the OTs during a series of meetings. These discussions were helpful in two ways. They helped in defining the main requirements by the OTs for a system which elicits the same types of movements as the ones exerted using the two artefacts introduced previously. And they helped in understanding the extent to which they can be parameterised in order to calibrate the level of difficulty. A two-hours participatory workshop was then organised at the hospital with five OTs and two of the authors. The workshop was split equally into two parts. We first engaged in a brainstorming session to find mappings between game actions and rehabilitation movements. The list of rehabilitation movements was matched with a list of candidates for the games. When prompted for simple games they were familiar with, the participants mentioned Whack-a-Mole, Bubble Witch, Pong and Frogger to name a few, all of which were arcade games. The controls needed to implement their gameplay were then discussed. While some required four ways directional actions (e.g. Pac-Man), for aiming at a target for example, others only required button presses for triggering specific actions (e.g. Puzzle Bubble). To simplify the workshop complexity, it was decided that games with simple 4-ways controls should be focused on. After the brainstorming, the second discussion addressed the parameterisation of specific exercises the OTs would like to see implemented by a gameplay. The movements that patients were producing when interacting with the skateboard, mentioned earlier, were used as a reference. A paper prototype of actionable controls was created for the interaction to be played out on the desk around which the workshop took place. The instructions from the OTs were to perform long reaching motions with the arm in contact with a planar surface. The goal they pursued was for their patients to reach always further. The flexibility of the paper prototype highlighted that the possibility of actionable controls whose locations on the surface could be easily modified was important for exercise adaptation. The outcome of these workshops was then threefold: • We identified that the game itself was not the focus for either parties (OTs and ourselves), as long as it is enjoyable for the patient and complies to the specifications given by OTs; • We sketched/agreed on a control input with 4-ways actionable buttons to interface with various different games; • We chose a rehabilitation parameter: the position of the gamepad’s buttons. 3.2 System description The preliminary field work presented above has led to the prototype depicted in Figure 1 (left). A patient is seated by a tabletop on top of which reaching motions are performed with one arm. An optical device both tracks the patient’s hand and creates a virtual surface with four regions arranged in a cross shape. Each virtual button is activated when the hand of the user enters its respective region. This new input control is interfaced with a digital game running in front of the patient. The gestures required by the gameplay are meant to be similar to the one’s needed for the patient’s regular exercising. As for the game, we chose Pac-Man (Namco, 1980) as test game. It is compatible with the four-way control and it is one of the most iconic arcade games where the rules are known by many. Regions of the virtual surface were mapped to directional commands (UP, RIGHT, DOWN, LEFT). Pac-Man always move forward at a given speed, the commands are only used to make directional changes. The prototype was initially set up in the authors’ lab on a 78cm wide office desk. We used an Optitrack system for the tracking of the participants hand. A single Infrared (IR) reflective marker, placed on the user’s hand between the index and middle fingers, indicated the (x, y) location of the hand on the surface. A dedicated computer ran the Optitrack system whose output was transmitted to another machine running the game in a browser. We used an opensource version of Pac-Man1 . Translating (x, y) coordinates of the hand to game commands was done with custom software on the machine running the game. The origin of the controls was indicated by a protruding tactile marker. Directional commands were triggered when the tracked hand location entered the triangular control regions, which were not visible to the user. Audio feedback indicated when commands were triggered with a bip. Users would be equipped with headphones during use to ensure they could clearly hear this feedback and the in-game audio. 3.3 Interaction parameterisation Based on the field work and the implemented system, interaction parameters have been identified in order to act upon the game difficulty and rehabilitation effectiveness. A first parameter is the spread parameter, denoted r, defined as the distance between the inner tips of the triangular actionable regions (Figure 1 (right)). This parameter serves the purpose of varying task difficulty. This parameter was actually required by the OTs as a lever to adapt the game difficulty to the patient’s needs. We conducted a first round of informal tests with members of our lab, revealing that the overall interaction difficulty was greatly increased due to the new input control modality (from button presses to reaching arm movements). We observed that the change in game difficulty incurred by the change of input control modality cannot be accommodated by the spread parameter alone. The latency and responsiveness of control is impacted by optical tracking delay and the effective reaching time necessary to move between regions. To counterbalance this effect, we identified a second interaction parameter controlling the speed of the game, called Time Rate. This is similar to what games like Tetris use to set difficulty without altering other part of the gameplay mechanics and was for example recommended by Burke et al. (Burke et al., 2009) in their guidelines. It also has the benefit to limit game alterations to a unique and controllable parameter time rate Tr. By reducing the time rate, the number of actions per unit of time required for playing are reduced, facilitating the interaction. The proposed interaction parameterisation can be represented as a block diagram (Figure 2). The perception loop includes the user, an input device and a game and produces engagement, depicted as a thick black line on the diagram. The interaction can be affected by two design parameters: the scale r, governing the distance that the participant has to travel with the arm in order to trigger a command; the time rate Tr, governing the speed of the game. Both parameters relate to game difficulty, hence a significant impact on game engagement can be expected. As a matter of fact, providing a well-calibrated challenge is key to an engaging game experience. As phrased by Przybylski et al. (Przybylski et al., 2010), the “mastery of controls plays an important role in game motivation, largely as a necessary, but not sufficient, condition for achieving psychologically need-satisfying play.” 3.4 Summary The co-design activities presented in this section led to the design of a rehabilitation system, based on a new gesturebased input device and the game of Pac-Man. We have conducted preliminary informal testing, including with participants having sustained a SCI. In the remaining of the paper, we consider the problem of building a computational model of user performance while using the presented system. As introduced earlier, our motivation is to provide a more systematic method than heuristics in order to help adjusting the interaction parameters to define the game difficulty. Our approach relies on the estimation of the distance between a current performance and a baseline performance (reference). We split the problem in two: 1. Understanding the impact of the identified parameters on non-disabled participants’ performance while using the game. The fact to perform the study on non-disabled participants is motivated by the need to set a baseline performance and how parameters affect this baseline performance (as further developed in Section 4) 2. Proposing a behavioural model able to predict participants’ performance with respect to a baseline (Section 5). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv