Celebrating 30 years in VR: Professor Robert J. Stone on Human Factors and the Future of VR and AR

Professor Robert J. Stone, who leads the VR/Human Interface Technologies team at the University of Birmingham, celebrates his 30-year anniversary in the field of Virtual Reality this year (2017). Among his VR ‘hits’ that he had developed include: one of the first & Simulation (VRS) initiative in 1993, invention of one of the first (if not first) tactile feedback gloves back in 1992, and the design and development of the world’s first keyhole surgery skills trainer, MIST_VR_ in 1995. MIST_VR_ became the world’s first part-task surgical skills trainer, adopted and evaluated on an international scale, and adopted as the skills trainer by the European Surgical Institute in Germany. Virtual Reality the de facto Vanessa Radd caught up with Stone, who is also a founding member of the XR Alliance. Vanessa Radd: Bob, first of all, congratulations on your 30th anniversary in VR! Thank you so much for sharing your experience with us and thoughts on how you see the future of VR (and AR). I believe all of us have a lot to learn from your experience especially in implementing human-centered design and getting VR projects out of the lab to real life deployments. Could you start by sharing with us about your background? Bob Stone: My academic background is in Applied Psychology and Human Factors (a.k.a. Ergonomics). In brief, upon graduating from UCL, I started my career as a Human Factors specialist at what was British Aerospace in Bristol, UK, undertaking work in the defence, subsea oil and gas, and space domains, and then, following an amazing experience at NASA’s Research Centre in California in 1987, I joined the UK’s National Advanced Robotics Research Centre in 1979 to undertake research and development in the field of VR and “telepresence”. Then, following the successful launch and completion of a totally industry-funded Virtual Reality & Simulation initiative between 1993 and 1995, I became the Director of a small spin-out company called VR Solutions. This company went through many roller coaster transitions over a 7-year period, including being purchased by two US organizations, and I finally decided to make the “jump” to academia in 2003. Today, I lead a small group of researchers and students within the School of Engineering at the University of Birmingham called the Human Interface Technologies (HIT) team. Vanessa Radd: Please tell us about the work you do. In what industry verticals are you and your team focusing on? Bob Stone: As for industry sectors, my team and I tend to get involved wherever we think there is a particular technical challenge or where we can help — from an independent perspective — potential adopters to make the right choices of VR or AR for their organizations and end users. However, in terms of the sectors we “specialize” in, based on where the main — and sustainable — projects are evolving, I would say we are particularly active in defense, healthcare and heritage. technology In the defense area, for example, we have a particularly strong track record in areas such as simulation for submarine, counter-terrorist and bomb disposal robot training, Mixed Reality, or “MxR” techniques for command and control, and, last but not least, defense healthcare — MxR training for future Armed Forces paramedics, for example. In healthcare generally, we have, in the past, developed VR and AR for surgical training and education; today, we’re investigating the delivery of virtual scenes of nature into hospital intensive care wards to help improve patient sleep quality and post-operative rehabilitation. Finally, in the heritage sector, which is my personal “favorite” (because it enables us to work closely with schoolchildren and members of the public), we undertake projects as wide-ranging as using drones to recreate VR models of remote areas of historical significance, subsea wreck visualizations, and — in collaboration with colleagues in the US — the development of VR and AR content to support the educational aspects of the commemorations scheduled for 2020. Mayflower 400 Vanessa Radd: How much of your work delve into VR vs AR? Bob Stone: Historically, my work and that of my team has focused on VR, but today, especially with many of our project areas focusing on defence, healthcare and heritage, we’re probably expanding as much effort into AR and MxR as we are in VR. I predict a strong growth in MxR over the coming years, as I believe that blending the “best of the real” with the “best of the virtual, or augmented” will deliver the most credible and affordable solutions in training and real-time visualization. Having said that, I have to confess that my personal technical implementation skills when it comes to VR, AR or MxR are quite limited! I have a small team of very competent specialists providing expertise in topics like 3D modelling, physics-based rendering and VR toolkit exploitation. My main area of expertise is, and always has been, human factors and human-centred design — trying to ensure that we deliver appropriate interactive hardware technologies and appropriate content to real people undertaking real tasks in the real world. For VR, AR and MxR to succeed, especially given the academic nature of my team, we cannot simply develop our solutions and demonstrators in a cosy academic laboratory, publishing papers in journals that those who might benefit from our research simply will not read!! We MUST work with the end users from the beginning, and throughout each and every stage of the project. Vanessa Radd: Tell us more about what you have developed in VR and AR. Bob Stone: Over the years, both my commercial and academic teams have developed a huge range of what we call “concept capability demonstrators”. What this means is that we take on small, self-contained projects over short periods of time, working closely with sponsors to develop something they and their colleagues or employees can evaluate, prior to making significant and highly risk-laden investments in otherwise unproven technologies (we’re seeing a lot of that in the VR and AR community today, sadly). My team’s portfolio is huge, with far too many projects to summarize here. But they are very diverse indeed, ranging from submarine safety training to future AR and MxR concepts for military/security service command and control and next-generation aircraft concepts; from recreating historical wrecks and subsea habitats in Plymouth Sound to abandoned railway lines and archaeological sites on Dartmoor; and from future autonomous vessels to novel ideas for AR exhibits with the UK’s National Marine Aquarium. Very occasionally we are fortunate to be in a position to take our developments to market, or to the end users for trials and evaluation at the very least. At the end of the 1990s, a close-range gunnery VR training system, developed by my previous commercial VR team, was installed at the Royal Navy’s land base at HMS Collingwood, and was used for basic weapons training for many years. Similarly, helicopter rear-crew VR systems were delivered to, and regularly used for training by the RAF at their search-and-rescue bases in Shawbury (England) and Valley (in Wales). An avionics maintenance training system, based on an extensive VR model of an RAF aircraft was also installed at a British RAF base and used very effectively for many years. One particular success in respect of taking solutions to “market” is the remote driving and manipulation simulator for the UK’s current bomb disposal robot, . After some 3 to 4 years of development and testing using VR software toolkits, we (uniquely for a British University) were sponsored to join forces with a small local company to deliver 42 simulator units to Explosive Ordnance Disposal (EOD) squadrons across the whole of the UK, and to Cyprus and Gibraltar as well. Tornado CUTLASS Vanessa Radd: What was your biggest achievement in VR to date? What are you most proud of? Bob Stone: Difficult question, as practically every month has brought with it a new challenge, enabling us to test some of today’s VR and AR technologies to their limits. We don’t always succeed, of course, but when we do, it’s always a huge and exciting achievement for the team! Looking back over the past 30 years, there are a number of achievements I am very proud of. Launching the Virtual Reality & Simulation, or initiative, at VR Solutions Ltd back in 1993, when we were still part of the UK’s National Advanced Robotics Research Centre. The response we had from industry, following a chance appearance of my team on the BBC’s in January 1993, was tremendous. So much so, that we were able to pull together companies large and small to sponsor a 3-year research and demonstrator programme without any UK Government or Research Council support at all! Just some of the names we had onboard the programme were British Nuclear Fuels, GEC Alsthom, ICI, Rolls-Royce PLC, and Vickers Shipbuilding and Engineering Limited (today part of the BAE Systems empire). VRS 9 O’Clock News VRS Another achievement, albeit rather short-lived, unfortunately, was inventing one of the first (if not first) tactile feedback gloves back in 1992. was a pneumatic prototype, initially consisting of 20 small “airpockets” distributed throughout the lower part of a Lycra glove and fed by micro-capillary tubes. It was originally designed to be used in conjunction with VPL’s . In fact, a commercial agreement along these lines was under development when, sadly, VPL filed for bankruptcy in 1990. the Teletact DataGlove I guess the one achievement I’m most proud of (as it has influenced practically every VR, AR and MxR project since) was the design and development of the world’s first keyhole surgery skills trainer, MIST_VR_ in 1995. Briefly, MIST_VR_ evolved from a project sponsored in 1994 by the UK’s Wolfson Foundation and Department of Health, the overarching aim of which was to assess the potential of emerging VR technologies to deliver cost effective keyhole surgery training for future medics. MIST_VR_ became the world’s first part-task surgical skills trainer, adopted and evaluated on an international scale, and adopted as the skills trainer by the European Surgical Institute in Germany. de facto More importantly, however, the MIST_VR_ project demonstrated conclusively that, by adopting a strong human-centered design approach from the outset of a VR project, it was all too easy to be influenced by the availability of expensive, glamorous and unproven technologies and to invest very heavily to achieve the highest fidelity (and equally expensive) solution. not Spending four afternoons in the operating theaters observing surgeons performing a variety of keyhole surgeries enabled us to design and develop a much simpler — and affordable — solution, ultimately (as the medical community confirmed) delivering exactly what the trainee surgeons required. Today, I am very proud to lead a team of very capable research scientists and VR specialists. We’re also very lucky to have attracted in the past — and, indeed, continue to attract — some incredibly keen and able students to undertake projects with us, at both undergraduate and postgraduate levels. The HIT team is very small indeed, and our funding levels are tiny, certainly in contrast to the crazy sums of money changing hands in the VR world at the moment (although we live in hope that, one day, we might find an enthusiastic benefactor who might sponsor us to do more of what we do — and work with us to take our solutions to market!). However, despite our size, we very much “punch above our weight” and people often surprised at the wide range of activities we undertake and the speed and quality with which we deliver. Vanessa Radd: What are the most successful and practical applications of VR and emerging technologies going forward — where do you see the game-changing developments? And in what industries? Bob Stone: Commentators on the VR and AR circuit are always harping on about the absence of a “killer app” for VR and AR. It’s been like that for years! In some respects, though, I would agree that there is a perception that either technology has yet to make a significant — and properly documented (i.e. not “over-glossed” with PR hype) — impact in a non-gaming sector. Indeed, VR and AR have some way to go before being mainstream in the gaming arena too. Nevertheless, since the end of VR’s second meteoric rise and fall (i.e. the early-to-mid 2000s), the affordability and accessibility of VR and, to a lesser extent (currently) AR, in terms of available hardware devices and software toolkits, has opened up a raft of opportunities in a growing number of applications. Training is an obvious example, and there are numerous case studies now becoming evident where the exposure of trainees to virtual recreations of tasks and contexts in such sectors as defense and healthcare (which includes surgical training, “training to challenge” (i.e. the decisions of one’s seniors) and various aspects of patient rehabilitation) is delivering a positive and proven advantage over and above other forms of training delivery. Other key sectors, including aerospace, automotive, maritime, to mention but three, are also prime beneficiaries for the technology, but really need to do more to demonstrate a real impact on their employees and organizations — PR comments from senior corporate personnel claiming improved time to market as a result of significant investments in VR studios are not enough. I believe that VR and AR will — indeed are — revolutionizing the education sector, at all age levels and in many, many topics, from science and history, to foreign language training and humanities. However, much more needs to be done to ensure that the confidence of teaching personnel — many of whom are still hopelessly outperformed by their students when it comes to real-time gaming and online competence — is adequately dealt with during the early stages of design! I also believe that Virtual Heritage, although not an “industry” as such, is a highly rewarding sector for applications of VR and AR technology. Although I accept (and speak from hard experience!) that heritage pursuits are never going to make significant amounts of money for anyone in the VR community, the impact the projects make, in terms of public engagement and outreach to schoolchildren, is tremendous. The demos we have staged at urban and rural school, small churches, beach clubs and for small village historical societies, not to mention the follow-up contacts such demos stimulate, constitute the most rewarding experiences I have ever had in my 30 years of involvement in VR and AR! **What are your thoughts about VR of yesteryear and the VR ‘hype’ and innovation of today?**The VR innovators of yesterday are people like Scott Fisher and Steve Ellis, Tom Furness, Fred Brooks, Myron Krueger, Morton Heilig, Bob Jacobson, Ben Delaney (editor of ), and the (sadly) late Mike Bevan (editor of the original )…that’s just a handful. CyberEdge Journal VR News Sadly, I’ve not really witnessed any one personality during the recent 3 or 4 years — in other words during the “resurrection” of VR and AR — who come anywhere close to what the VR “veterans” of yesteryear achieved. Sure, there are those animated individuals who do very well at convincing others by their animated and over-hyped on-stage appearances at the ever-increasing number of same-old, same-old VR and AR conferences. I’d like to think that my past 30 years (and the fact that I was there at the very start of VR — including the incredibly important Conference in Santa Barbara in 1990), and the fact that I’m still here (!), has, perhaps, put me on a par with these good folks, especially in terms of “flying the flag” for the UK’s role (at least) in the development of the global VR community. I just hope that the VR and AR sectors mature quickly and that we see strong examples of real benefit being reported by commercial and industrial organizations. Teleoperators & Virtual Environments We currently find ourselves in a similar situation to that experienced during the 1990s, when the hype and false promises put out by the fledgling worldwide VR “movement” resulted in many examples of premature investment in technologies and non-business-savvy start-ups that were destined to fail. Worse still, many, many potential adopters during that period had their “fingers burned” as a result of believing that the technology could deliver much more than was the truth. Hopefully we can ride the inevitable storm that approaches, so that VR and AR can, once again, progress — albeit much more slowly than most would like — to another historical period of consolidation and development. Vanessa Radd: Thank you Bob and Happy 30th VR birthday! Twitter About the Author Vanessa Radd is ranked the global top 3 augmented reality influencer and is a founding member of the XR Alliance. Vanessa is also the President of the global VR AR Association Singapore chapter. @vanradd Radd’s background comprises +17 years of internet and broadcast content & technology across the media value chain (from production, post production and content delivery, to hardware, software, image processing and cloud/virtualization – utilized as the broadcast industry standard for the Olympics) experience representing international media and technology brands in Asia. Radd also worked for a broadcast industry organization that oversaw the Olympics broadcast rights for over 3 billion viewers in Asia-Pacific and the setting of broadcast technical standards. Radd’s international executive liaison includes Tier-1 broadcasters, corporate CXOs, media organizations, non-profits, governmental agencies and the tech and developer communities. **About the XR Alliance** We at the XR Alliance work with industry partners to develop quality programs that deliver value to the ecosystem and the industry as a whole. We strongly believe in fostering good partnerships and collaborations and are selective in what we work on. We also take on suitable consulting projects, including helping VR/AR companies in the U.S. and Europe to enter Asia-Pacific markets. Please enquire about availability. E-mail: v @ xralliance.org Twitter: @xrforce