Researchers Designed a Game-based Rehab System—Here's How

Authors: (1) Antoine Loriette, IRCAM, CNRS, Sorbonne Universite, Paris, France (antoine.loriette@gmail.com); (2) Baptiste Caramiaux, Sorbonne Universite, CNRS, ISIR, Paris, France (caramiaux@isir.upmc.fr); (3) Sebastian Stein, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (sebastian.stein@glasgow.ac.uk); (4) John H. Williamson, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (johnh.williamson@glasgow.ac.uk). Table of Links Abstract and 1 Introduction 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 4 Experimental Study 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 7 Conclusion 8 Acknowledgements and References 6 Discussion This paper contributes to research in interactive systems for motor rehabilitation. We presented two main contributions. First, we presented a gesture-based interactive system, interfacing with a commercial game, specifically designed such as complying with requirements from the context. These requirements were provided by occupational therapists involved in a co-design process. One of these requirements was to be able to adapt the level of the game difficulty through interaction parameters, that could be set by the OTs or automatically adapted. Our second contribution is to identify these parameters and propose a computational model estimating, in realtime, the input control performance as compared to a baseline performance, which could then be used as a proxy for game performance or optimisation function to adapt the parameters automatically. In this section, we discuss the different findings. 6.1 Design workshop and tools for modelling The co-design activities (section 3) with occupational therapists were effective in exploring interaction possibilities and establishing requirements for a gamified interaction of the upper limb for patients with SCI. The design of a new control modality was the principal outcome of the workshops with OTs. For motor rehabilitation, which targets specific motions from patients, the input control is the principal interface between the user’s body and the game. This is quite different from other types of rehabilitation targeting for example cognitive impairments (Mandryk et al., 2013). As a result, the game itself or the choice of the game did not emerge as a priority. Instead, the OTs mentioned the idea of reusing the proposed design with different games for satisfying specific patient preferences, a practice also described in the work by Hofmann et al. (Hofmann et al., 2019) as design reuse across clients. Supported by rapid design iterations with a paper prototype, an adequate mapping from desired movement space to game control space (Walther-franks et al., 2013) was identified as well as a suitable parameterisation for adapting task difficulty (Lopes and Bidarra, 2011). The physical separation between controls placed on a tabletop effectively increased the reaching motions, in accordance to the rehabilitation requirements. This was helped by the natural connection between reaching motions and targeting motions but remains an issue for other types of activity. For more complex mapping, such as the one involving pedalling on a recumbent bicycle proposed by Ketcheson et al. (Ketcheson et al., 2016), an obvious solution to the mapping task is hard to obtain. We identify here potential opportunities for linking serious game design research with research related to HCI, such as (Bachynskyi et al., 2015; Oulasvirta et al.,2013). The breadth of possibilities afforded by the design of a new input control, and the effects these have on the user’s body are not yet fully understood. As for the game adaptation, we chose not to use additional graphical overlays (Ketcheson et al., 2016). Instead, we relied on the flexibility of the control modality via the parameter spread and on the well-known game design pattern using time rate fluctuations. The OTs’ definition of the targeted motions was not very restrictive, on the contrary to systems that target more precise motion executions, such as YouMove (Anderson et al., 2013) or PhysioHome (Tang et al., 2015), wherein the use of graphical feedback to convey how well the user is performing motions seems hardly avoidable. Put together, this affords a simple parameterisation, with the spread and time rate mainly controlling to rehabilitation goals and gameplay performance, respectively. 6.2 Interaction parameters adaptation Game interaction with modified control input requires adaptation and usually results in (much) harder challenge for the player. Apart from controllers designed for performance, such as the one proposed by Kwak et al. (Kwak and Salem, 2009), new input controls can not afford users the same proficiency. This effect was measured during the experimental study (section 4) by including a control condition and evaluation against a measure of in-game score. We showed that the parameter Trate was successful in adapting the interaction difficulty: reducing Trate from its original value decreased the level of difficulty, and with a preset level fixed around 1/3 the scores were indistinguishable. This is close to the 1/4 ratio in information throughput measured by Card et al. (Card et al., 1991) between the arm and the fingers. This also quantifies the significant change in users’ control capabilities when input devices are swapped, which is thus an important effect to quantitatively measure when similar systems are proposed. However, finding approximate preset values for the interaction parameters so that users could enjoy a performance similar to the keyboard reference is time consuming when score is used as a metric due to its sparse availability and high variability. In addition, any further modifications to the design of the control input would render previous presets unusable, requiring another round of measurements. Finally, since the data collected involved only unimpaired participants, it is unlikely that these preset values transfer to the final target group. As a result, we developed a low-latency metric of baseline gameplay that could replace the reliance on score for that matter. 6.3 Modelling Gameplay We proposed modelling baseline gameplay (section 5) using game-specific features and user input, and using the normalised log-likelihood of observed gameplay under the reference model as a similarity measure. We showed that this measure is highly correlated with score, has lower variance and is available at lower latency. We relied on the reasonable assumption that unmodified play is engaging. Our model of gameplay thus captures the behaviour of users, through their input frequency or IKI, and the effect they have on the game, through the specifically designed variable PTT. Even though PTT is hand crafted feature, it is simply an indirect measure of the avatar speed. Then, we showed that this model correlates with score. This is evidence that maintaining gameplay, defined by IKI and PTT, is important to ensure game performance. The correlation between both metrics is however not perfect. The totality of the behaviours related to game outcome are not taken into account by the approach we proposed. Only the number of actions per unit of time and the speed of the Pac-Man in the game were considered. More complex models, such the one proposed by Smith et al. (Smith and Nayar, 2016) could be employed if the objective is to predict with accuracy the final score. Complex models are however hard to train. For instance, Smith et al. (Smith and Nayar, 2016) reported training duration in the order of days. The point here is that simple models, yet able to generalise, are valuable. The model we proposed relied on four parameters only, and was shown less prone to inter-user variability than in-game score. Our results showed that the spread parameter does not impact the score. We can speculate that a higher spread may have impacted the score values, or that the spread considered would have impacted the performance of participants with impairments. However the spread does impact the log-likelihood values, suggesting that the proposed model may be more suitable to assess the input control performance than game score. The spread parameter was meant to steer motions towards those targeted by OTs for rehabilitation. Having a model that is capable of leveraging changes in spread is therefore important from a rehabilitation point of view. Finally, a model linking engagement and performance is not easily established. Access to user engagement requires having specific measurements which often rely on questionnaires. Using the flow framework (Csikszentmihalyi, 2014), Limperos et al. (Limperos et al., 2011) compared the difference a Wii and a Playstation controller produced on the game experience, with the Wii controller being qualified as “natural” and requiring more involvement from the user body. They showed that the performance as defined as the end-game score was not alone explaining the enjoyment, and that the sense of control that users experienced was a rather salient indicator. Understanding what part of the gameplay makes a game engaging is an interesting research question. There is a need for models that are interpretable and that rely on quantifiable gameplay measures to further the understanding of user engagement. 6.4 Limitations and Future work We acknowledge several limitations to this work. We made several assumptions with regards to the model. One was that samples were independent of each others, and IKI and PTT were also independent. These simplification assumptions are useful to build a simple model: IKI/PTT independence allows to model them separately, otherwise a joint distribution should have been learned which would have required a more complex learning procedure. The assumed independence of consecutive samples drawn from IKI and PTT is however useful to use simple probabilistic distributions with rather good generalisation capabilities. It is justified here with regards to the variables themselves: they describe short term processes that may not vary much with time. The simplicity of the game we used is both an advantage and a disadvantage. For games using a similar control scheme, this approach is likely to be transferable. For game with more complex control commands, more work should be done on movement representation and probabilistic modelling. The distributions that were modelled in this work were simple (i.e. unimodal) and stationary which might not be always be the case. The baseline model we trained is meant to be representative of a general population of players. This is one of the reason for its simplicity and the focus on command frequency: the simpler the model, the lesser the risk of overfitting. With a total population of 12 participants however, it is likely that certain category of players would be incorrectly modelled. In that case, it could be possible that players performing well with our system while behaving out of the norm would be qualified as unlikely by our model. Finally, as future work we plan to test the model we developed to optimise online the interaction parameters. This extension to our computational approach is qualified as computational design (Oulasvirta et al.) and is a clear path forward for our research. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Antoine Loriette, IRCAM, CNRS, Sorbonne Universite, Paris, France (antoine.loriette@gmail.com); (2) Baptiste Caramiaux, Sorbonne Universite, CNRS, ISIR, Paris, France (caramiaux@isir.upmc.fr); (3) Sebastian Stein, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (sebastian.stein@glasgow.ac.uk); (4) John H. Williamson, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (johnh.williamson@glasgow.ac.uk). Authors: Authors: (1) Antoine Loriette, IRCAM, CNRS, Sorbonne Universite, Paris, France (antoine.loriette@gmail.com); (2) Baptiste Caramiaux, Sorbonne Universite, CNRS, ISIR, Paris, France (caramiaux@isir.upmc.fr); (3) Sebastian Stein, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (sebastian.stein@glasgow.ac.uk); (4) John H. Williamson, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (johnh.williamson@glasgow.ac.uk). Table of Links Abstract and 1 Introduction 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 4 Experimental Study 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 7 Conclusion 8 Acknowledgements and References Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 2.2 Input control performance measure and modelling and 2.3 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 4 Experimental Study 4 Experimental Study 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 7 Conclusion 8 Acknowledgements and References 7 Conclusion 8 Acknowledgements and References 8 Acknowledgements and References 6 Discussion This paper contributes to research in interactive systems for motor rehabilitation. We presented two main contributions. First, we presented a gesture-based interactive system, interfacing with a commercial game, specifically designed such as complying with requirements from the context. These requirements were provided by occupational therapists involved in a co-design process. One of these requirements was to be able to adapt the level of the game difficulty through interaction parameters, that could be set by the OTs or automatically adapted. Our second contribution is to identify these parameters and propose a computational model estimating, in realtime, the input control performance as compared to a baseline performance, which could then be used as a proxy for game performance or optimisation function to adapt the parameters automatically. In this section, we discuss the different findings. 6.1 Design workshop and tools for modelling The co-design activities (section 3) with occupational therapists were effective in exploring interaction possibilities and establishing requirements for a gamified interaction of the upper limb for patients with SCI. The design of a new control modality was the principal outcome of the workshops with OTs. For motor rehabilitation, which targets specific motions from patients, the input control is the principal interface between the user’s body and the game. This is quite different from other types of rehabilitation targeting for example cognitive impairments (Mandryk et al., 2013). As a result, the game itself or the choice of the game did not emerge as a priority. Instead, the OTs mentioned the idea of reusing the proposed design with different games for satisfying specific patient preferences, a practice also described in the work by Hofmann et al. (Hofmann et al., 2019) as design reuse across clients. Supported by rapid design iterations with a paper prototype, an adequate mapping from desired movement space to game control space (Walther-franks et al., 2013) was identified as well as a suitable parameterisation for adapting task difficulty (Lopes and Bidarra, 2011). The physical separation between controls placed on a tabletop effectively increased the reaching motions, in accordance to the rehabilitation requirements. This was helped by the natural connection between reaching motions and targeting motions but remains an issue for other types of activity. For more complex mapping, such as the one involving pedalling on a recumbent bicycle proposed by Ketcheson et al. (Ketcheson et al., 2016), an obvious solution to the mapping task is hard to obtain. We identify here potential opportunities for linking serious game design research with research related to HCI, such as (Bachynskyi et al., 2015; Oulasvirta et al.,2013). The breadth of possibilities afforded by the design of a new input control, and the effects these have on the user’s body are not yet fully understood. As for the game adaptation, we chose not to use additional graphical overlays (Ketcheson et al., 2016). Instead, we relied on the flexibility of the control modality via the parameter spread and on the well-known game design pattern using time rate fluctuations. The OTs’ definition of the targeted motions was not very restrictive, on the contrary to systems that target more precise motion executions, such as YouMove (Anderson et al., 2013) or PhysioHome (Tang et al., 2015), wherein the use of graphical feedback to convey how well the user is performing motions seems hardly avoidable. Put together, this affords a simple parameterisation, with the spread and time rate mainly controlling to rehabilitation goals and gameplay performance, respectively. 6.2 Interaction parameters adaptation Game interaction with modified control input requires adaptation and usually results in (much) harder challenge for the player. Apart from controllers designed for performance, such as the one proposed by Kwak et al. (Kwak and Salem, 2009), new input controls can not afford users the same proficiency. This effect was measured during the experimental study (section 4) by including a control condition and evaluation against a measure of in-game score. We showed that the parameter Trate was successful in adapting the interaction difficulty: reducing Trate from its original value decreased the level of difficulty, and with a preset level fixed around 1/3 the scores were indistinguishable. This is close to the 1/4 ratio in information throughput measured by Card et al. (Card et al., 1991) between the arm and the fingers. This also quantifies the significant change in users’ control capabilities when input devices are swapped, which is thus an important effect to quantitatively measure when similar systems are proposed. However, finding approximate preset values for the interaction parameters so that users could enjoy a performance similar to the keyboard reference is time consuming when score is used as a metric due to its sparse availability and high variability. In addition, any further modifications to the design of the control input would render previous presets unusable, requiring another round of measurements. Finally, since the data collected involved only unimpaired participants, it is unlikely that these preset values transfer to the final target group. As a result, we developed a low-latency metric of baseline gameplay that could replace the reliance on score for that matter. 6.3 Modelling Gameplay We proposed modelling baseline gameplay (section 5) using game-specific features and user input, and using the normalised log-likelihood of observed gameplay under the reference model as a similarity measure. We showed that this measure is highly correlated with score, has lower variance and is available at lower latency. We relied on the reasonable assumption that unmodified play is engaging. Our model of gameplay thus captures the behaviour of users, through their input frequency or IKI, and the effect they have on the game, through the specifically designed variable PTT. Even though PTT is hand crafted feature, it is simply an indirect measure of the avatar speed. Then, we showed that this model correlates with score. This is evidence that maintaining gameplay, defined by IKI and PTT, is important to ensure game performance. The correlation between both metrics is however not perfect. The totality of the behaviours related to game outcome are not taken into account by the approach we proposed. Only the number of actions per unit of time and the speed of the Pac-Man in the game were considered. More complex models, such the one proposed by Smith et al. (Smith and Nayar, 2016) could be employed if the objective is to predict with accuracy the final score. Complex models are however hard to train. For instance, Smith et al. (Smith and Nayar, 2016) reported training duration in the order of days. The point here is that simple models, yet able to generalise, are valuable. The model we proposed relied on four parameters only, and was shown less prone to inter-user variability than in-game score. Our results showed that the spread parameter does not impact the score. We can speculate that a higher spread may have impacted the score values, or that the spread considered would have impacted the performance of participants with impairments. However the spread does impact the log-likelihood values, suggesting that the proposed model may be more suitable to assess the input control performance than game score. The spread parameter was meant to steer motions towards those targeted by OTs for rehabilitation. Having a model that is capable of leveraging changes in spread is therefore important from a rehabilitation point of view. Finally, a model linking engagement and performance is not easily established. Access to user engagement requires having specific measurements which often rely on questionnaires. Using the flow framework (Csikszentmihalyi, 2014), Limperos et al. (Limperos et al., 2011) compared the difference a Wii and a Playstation controller produced on the game experience, with the Wii controller being qualified as “natural” and requiring more involvement from the user body. They showed that the performance as defined as the end-game score was not alone explaining the enjoyment, and that the sense of control that users experienced was a rather salient indicator. Understanding what part of the gameplay makes a game engaging is an interesting research question. There is a need for models that are interpretable and that rely on quantifiable gameplay measures to further the understanding of user engagement. 6.4 Limitations and Future work We acknowledge several limitations to this work. We made several assumptions with regards to the model. One was that samples were independent of each others, and IKI and PTT were also independent. These simplification assumptions are useful to build a simple model: IKI/PTT independence allows to model them separately, otherwise a joint distribution should have been learned which would have required a more complex learning procedure. The assumed independence of consecutive samples drawn from IKI and PTT is however useful to use simple probabilistic distributions with rather good generalisation capabilities. It is justified here with regards to the variables themselves: they describe short term processes that may not vary much with time. The simplicity of the game we used is both an advantage and a disadvantage. For games using a similar control scheme, this approach is likely to be transferable. For game with more complex control commands, more work should be done on movement representation and probabilistic modelling. The distributions that were modelled in this work were simple (i.e. unimodal) and stationary which might not be always be the case. The baseline model we trained is meant to be representative of a general population of players. This is one of the reason for its simplicity and the focus on command frequency: the simpler the model, the lesser the risk of overfitting. With a total population of 12 participants however, it is likely that certain category of players would be incorrectly modelled. In that case, it could be possible that players performing well with our system while behaving out of the norm would be qualified as unlikely by our model. Finally, as future work we plan to test the model we developed to optimise online the interaction parameters. This extension to our computational approach is qualified as computational design (Oulasvirta et al.) and is a clear path forward for our research. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv