X-R Solutionis in the Virtual Reality Industry

X-R Solution

This is the fifth piece in a series on developing XR applications and experiences using Oracle and focuses on XR applications of healthcare, vision AI, training and collaboration, and messaging, including other topics such as multi-platform development, etc. Find the links to the first four articles below: Develop XR With Oracle, Ep 1: Spatial, AI/ML, Kubernetes, and OpenTelemetry Develop XR With Oracle, Ep 2: Property Graphs and Data Visualization Develop XR With Oracle, Ep 3: Computer Vision AI, and ML Develop XR With Oracle, Ep 4: Digital Twins and Observability As with the previous posts, I will specifically show applications developed with Oracle database and cloud technologies using HoloLens 2, Oculus, iPhone, and PC and written using the Unity platform and OpenXR (for multi-platform support), Apple Swift, and WebXR. Throughout the blog, I will reference the corresponding demo video below. https://www.youtube.com/watch?v=QTe2trT2ZHM&embedable=true Extended Reality (XR) and Healthcare I will refer the reader to the first article in this series (again, the link is above) for an overview of XR, and I will not go in-depth into the vast array of technology involved in the healthcare sector but will instead focus on the XR-enablement of these topics and the use of Oracle tech to this end particularly as Oracle has an increased focus in this area via the Cerner acquisition and other endeavors. It is well known that telehealth has grown tremendously since the pandemic, peaking at 78 times the rate just a month before the pandemic and even now still leveling off at 38 times that rate. These and other numbers and their impact are well documented in numerous publications, such as this and the , which will be referenced in this article. One only needs to have heard of the to understand the extent to which XR will aid in this industry, and so again, I will not delve too deeply into attempting to prove this, but proceed to give some examples and ideas involving Oracle database and cloud technologies. McKinsey report VR/AR Associations Healthcare Forum use of XR for live surgeries by Johns Hopkins back in 2021 Vision AI and Contextual Intelligence: Real-time analysis and diagnosis The first example involves the use of the Hololens mixed-reality headset (though other headsets/devices could be used) and Oracle’s Vision AI service. It should not be much of a stretch of the imagination to envision healthcare workers wearing XR devices that provide additional functionality and information to them (and indeed, many already are). Today doctors, dentists, etc. constantly cross-reference a computer screen with a patient to check for information and orient what they see (being x-ray, MRI, etc.) and often where head-mounted lights, monocular magnifying glasses, etc. The XR solutions being discussed here are mere evolutions of that. More details on this application can be found in the research blog , but the basic details of how it works are described here. here The application running on the Hololens takes pictures with its built-in camera using the wearer’s field of vision as a reference and does so at regular intervals. This provides a hands-free experience that is, at the very least, more convenient. Still, in many situations, like in the operating room, it’s necessary for users’ hands to interact with the real (or XR) world, thus making it a more optimal solution. This also means the system can pick up contextual information that the practitioner may not be aware of or have access to while quickly gathering and processing such information without the practitioner having to explicitly instruct it, thus saving time as well. This leads to the optimization that mixed reality provides, an exciting byproduct of immersion. via secure REST calls, where they can be conveniently accessed by directly and also stored and/or accessed in the database. The Hololens then sends these pictures to OCI Object Storage the OCI Vision AI Service There are several different approaches and architectures that can be used from this point to conduct the logic and calls to the Oracle Vision AI APIs for processing the images sent by the XR device. For example, as far as language used, initial versions were written using the OCI CLI, Java, and Python, with a final Java GraalVM-native image version being used. Also, as the Java GraalM native image starts almost instantaneously when the service conducts a particular short-lived routine, it is a good candidate for a serverless function. Optionally, the OCI Notification Service can listen to Object Store changes and call serverless functions as well. The Java service receives a notification of image upload to object storage and conducts a series of actions as follows: Conducts an OCI Vision AI service API call backed by an x-ray object detection model and provides the location of the image sent by the Hololens. Receives a reply from the object detection model with the percentage chance of the x-ray being in the image and the bounding coordinates of it. Crops the original image using the bounding coordinates. Conducts another OCI Vision AI service API call backed by an x-ray classification model and provides it the cropped image of the x-ray. Receives a reply from the image classification model with percentage change of x-ray containing signs of abnormalities/pneumonia. The Hololens application receives this reply and notifies the wearer with an audible notification – this is configurable and can also be visual. In the case of this application, the information includes the picture of the cropped x-ray with its discovered details listed and stored in a virtual menu located on the wrist and viewable by the wearer only. This approach prevents the interruption of the wearer though it is also possible to overlay the results on the real-life x-ray from which they were derived. Vision AI and XR are a natural match for solutions in a number of areas, including both healthcare workers, as shown here, and people with conditions such as Parkinson’s, autism, Alzheimer’s, vision and hearing impairment, etc.. (e.g., applications/solutions that can be used in this space is described in ) this blog Motion and Sentiment Study: Real-time face and body tracking analytics Today, 95% of healthcare facilities provide remote treatment and rehabilitation. XR technologies can be used to help patients better understand their conditions and treatment options. This can help patients feel more informed and empowered in their healthcare decisions. The next example was implemented using the iPhone and Swift using Apple’s for body tracking movements. The joint coordinates of a model movement are recorded and sent to the Oracle database via Rest calls to ORDS. A person (whether it be a patient, athlete, etc.) using the application attempts to conduct the same movement, and their joint coordinates are compared to those of the model movement stored in the database. If the allowable deviation/delta of the movements is exceeded, the corresponding joints and bones are shown in red rather than green. RealityKit This feedback is given in real-time to measure progress (e.g., after a rotator cuff or other surgery) and/or to allow the user to modify their movement to match the control model as well as work on balance and coordination. These movements can, in turn, be analyzed (more on XR and Oracle Analytics in an upcoming blog), replayed, manipulated to carry out simulations, etc. These use cases, of course, extend into the sports, entertainment, etc. sectors as well. Digital Twins + Doubles and Multiplayer/Participant Messaging: Training and Collaboration A recent study reported by Unity shows that approximately 94% of companies using real-time 3D technology find it valuable for staff training and are using it to create interactive guided learning experiences. This applies across industries, including manufacturing, transportation, healthcare, retail, and many more, and some trends in the area (many of which coincide with what is shown in this blog) are discussed . here XR technologies can be used to create realistic simulations of medical scenarios, allowing healthcare professionals to practice procedures and techniques in a safe and controlled environment. For example, medical students and surgeons can use XR simulations to practice surgeries with tactile sensations giving the feedback of real surgery, and nurses can use AR simulations to practice administering medications. This can help healthcare professionals gain valuable experience without risking patient safety. In addition to teaching, XR provides a unique ability for collaboration between individuals in different locations and specialties, creating a shared virtual space where multiple users can interact with each other and with virtual objects. There are several different techniques and software options available for creating XR multiplayer/participant training and collaboration, including XR conferencing software and metaverses, Photon and others like it, Rest, WebSockets, and different types of messaging. XR conferencing software and platforms, such as Alakazam, are becoming increasingly popular and allow multiple users to participate in virtual meetings, events, training sessions, etc. Photon is a platform that allows developers to create multiplayer games and applications using Unity, Unreal Engine, and other game engines and can easily be installed on and take advantage of Oracle Cloud compute (including ). It is perhaps the most famous of such platforms, but there are others. NVIDIA GPU WebSockets are a protocol for real-time communication between web clients and servers and can be a faster and more efficient method for such use cases than Rest though Rest calls are a simple, viable option in many cases as well, and more APIs are available via Rest in general than any of the other methods. Messaging is another technique that can be used to create XR training and collaboration experiences. Systems such as Kafka and JMS have both pub/sub (multi-consumer topics) and producer/consumer (single-consumer topics) and are very flexible for different training and collaboration needs. In the training application shown in the video, I used a number of the techniques above but focused on using Oracle’s TxEventQ messaging engine (formerly known as AQ). This is a very powerful and unique offering for a number of reasons, one key one being its ability to conduct database work and messaging work in the same local transaction. This is unique and and also exactly-once message delivery so that there is no message loss and no need for the developer to write deduplication logic. perfectly suited for microservices as it provides a transaction outbox pattern This may not be necessary for conventional gaming or movie streaming but is a must for mission-critical systems and provides some extremely interesting and unique aspects for XR where the ability to reliably store a shared (3D) object (especially those that are dynamic or created via generative AI, e.g.) and the interactions/messages that have been made upon it by various participants is a very powerful tool. That is precisely what is done in the app shown in the video. The collaborative session is not only recorded in 3d for viewing as a video but the actual objects and their interactions by participants are recorded in the database and available for playback so that they can be intercepted and manipulated later on. This allows for the ability to do deeper learning and run further models (AI or otherwise), simulations, scenarios, etc., by tweaking the objects and interactions like/with playbooks. With the development of more advanced technology and more powerful devices, XR training will become more prevalent in the near future, making the training experience more immersive and interactive. One other small note is that the app in the video has healthcare workers in full 3d volumetric video captures made professionally in a studio but also one that is a simple 2d video with the green screen alpha-channel removed via a Unity shader providing a similar holographic effect with minimal effort (the quality could be better with more effort re lighting, etc.). A Zoom virtual greenscreen could be used to this end as well as free assets with animation done in Blender. Unity, Zoom, and Blender all being free and use the Oracle always-free Autonomous database, making creating the solution very accessible as far as software and cloud costs. Data EcoSystem: Telehealth and Virtual Health Centers A virtual healthcare center or hospital is a healthcare facility that provides medical services through digital channels such as video conferencing, online chat, remote monitoring, and, increasingly, XR technologies. These digital tools allow patients to access healthcare services from the comfort of their homes or remote locations without the need to physically visit a hospital or clinic. The concept of a virtual healthcare center is becoming increasingly popular due to the growing demand for telemedicine services, especially in the wake of the COVID-19 pandemic. A virtual healthcare center can provide a wide range of medical services such as primary care, specialty consultations, diagnostic tests, and prescription refills and offers convenience and flexibility to patients who can access medical services from anywhere and at any time. This can be especially beneficial for patients with mobility issues, those living in rural areas, or patients who need to consult with a specialist who is not available in their local area. It can also reduce healthcare costs for both patients and healthcare providers. By using remote consultations and monitoring, healthcare providers can reduce the need for expensive in-person consultations and hospital stays. Patients can also save money on transportation costs and time off from work. A virtual healthcare center can improve the quality of care for patients. With remote monitoring, patients can receive personalized and continuous care, with healthcare providers able to monitor their health in real-time and intervene quickly if necessary. This can result in better health outcomes and reduced hospital readmissions. However, there are also some challenges to implementing virtual healthcare centers. These include issues around data privacy and security, as well as the need for adequate internet access and digital literacy among patients. In addition, some patients may still prefer traditional in-person care, and, though decreasing in number, there are limitations to what medical services can be provided remotely. The concept of a virtual healthcare center has the potential to revolutionize the way healthcare is delivered, with benefits for both patients and healthcare providers. However, careful consideration and planning are needed to ensure that the implementation of virtual healthcare centers is safe, effective, and equitable for all patients. Oracle, with its acquisition of Cerner and increased focus on the future of healthcare, is in a one-of-a-kind position to facilitate such XR solutions by its ability to “Deliver better health insights and human-centric experiences for patients, providers, payers, and the public. Oracle Health offers the most secure and reliable healthcare solutions, which connect clinical, operational, and financial data to improve care and advance decision-making around health and well-being.” ( ). Larry Ellison made this clear at his Oracle OpenWorld presentation, where healthcare was the priority for Oracle’s future, and earlier in the year when he , “Together, Cerner and Oracle have all the technology required to build a revolutionary new health management information system in the cloud.” Oracle Health page stated Mental Health Over 20% of American adults experience mental illness, while over 2.5M youth struggle with severe depression and 800,000 people commit suicide each year in the world due to mental illness. Simultaneously, there is a growing deficit of mental health professionals, which is a nationwide issue but is particularly severe for minors and adolescents. According to the U.S. Department of Health and Human Services, the country is expected to have a shortage of 10,000 mental health professionals by 2025. Mental health is one of the areas where XR has been most widely researched and proven extremely effective as a way to deliver therapies and treatments for mental health conditions such as anxiety, depression, and PTSD. For example, VR exposure therapy can be used to help patients confront and overcome their fears by exposing them to simulations of the things they are afraid of in a controlled and safe environment. Cognitive Behavioral Therapy (CBT), XR can be used to create immersive, meditative experiences to promote mindfulness, relaxation, and stress reduction. For example, VR environments can simulate peaceful and calming natural environments, such as beaches, forests, mountains, or space. Children, the elderly, and others that may spend prolonged periods in hospitals, etc., are using XR to explore the world and socialize. Using XR and teletherapy, therapists can create virtual environments that can simulate in-person therapy sessions, providing a more immersive and personalized experience while reducing the inhibitions of the patient and even allowing the use of AR lenses and avatars for privacy. NLP (Natural Language Processing) sentiment analysis, such as that provided by the corresponding Oracle AI service, can interpret emotions and intent from verbal communication, and with advances in facial recognition and facial sentiment analysis, it is now possible with great accuracy to detect the emotions a person is feeling and/or expressing non-verbally (the same follows for body/gesture tracking). Neuro-technology and neural interfaces in coordination with XR can now interpret and give insight into human intent and measure emotion, providing even greater insight into intent and emotion. And, of course, the world is quite familiar with the greater and simpler accessibility of AI interfaces such as ChatGPT, etc., to facilitate this. As with the X-ray example given earlier, these can, in some situations, provide greater and more consistent accuracy and speed than a human but can also act as assisting technologies for healthcare workers. More examples of this will be in upcoming blogs. Additional Thoughts I have given some ideas and examples of how healthcare and XR can be used together and facilitated by Oracle. I look forward to putting out more blogs on this topic and other areas of XR with Oracle Cloud and Database soon. Please see for more information on XR and Oracle cloud and database, as well as various topics around microservices, observability, transaction processing, etc., as well as this . Also, please feel free to contact me with any questions or suggestions for new blogs and videos, as I am very open to suggestions. Thanks for reading and watching. my other publications recent blog about what AR is

Develop XR with Oracle, Ep 5 Healthcare, Vision AI, Training/Collaboration, and Messaging

Oracle, Ep 5 Healthcare, Vision AI, প্রশিক্ষণ/সহযোগিতা, এবং মেসেজিং সহ XR বিকাশ করুন

Entwickeln Sie XR mit Oracle, Ep 5 Healthcare, Vision AI, Training/Zusammenarbeit und Messaging

Desarrolle XR con Oracle, Ep 5 Healthcare, Vision AI, capacitación/colaboración y mensajería

Développer XR avec Oracle, Ep 5 Healthcare, Vision AI, Formation/Collaboration et Messagerie

Oracle, EP 5 हेल्थकेयर, विज़न AI, प्रशिक्षण/सहयोग, और मैसेजिंग के साथ XR विकसित करें

Oracle、Ep 5 Healthcare、Vision AI、トレーニング/コラボレーション、およびメッセージングを使用した XR の開発

Oracle, Ep 5 Healthcare, Vision AI, 교육/협업 및 메시징을 사용하여 XR 개발

Desenvolva XR com Oracle, Ep 5 Healthcare, Vision AI, Treinamento/Colaboração e Mensagens

Oracle, Ep 5 Healthcare, Vision AI, Eğitim/İşbirliği ve Mesajlaşma ile XR'yi geliştirin

Phát triển XR với Oracle, Ep 5 Chăm sóc sức khỏe, Vision AI, Đào tạo/Cộng tác và Nhắn tin

使用 Oracle、第 5 集医疗保健、Vision AI、培训/协作和消息传递开发 XR

At WWDC 2023, we witnessed ’s biggest product launch since 2015. The biggest tech company on the planet launched a revolutionary spatial computer that seamlessly blends digital content with the physical world. With , users will get to experience a fully immersive infinite canvas for apps that scales beyond the boundaries of a traditional display. Apple Apple Vision Pro With the world’s first spatial operating system, , and the groundbreaking features loaded into it, Vision Pro will change the way users interact with technology. Seeing someone with a VR headset at a coffee shop won't be a novelty in a few years. The revolutionary also unlocks countless exciting opportunities for enterprises. Vision OS extended reality (XR) headset In this blog, let us take a brief look into how Vision Pro can offer enterprise VR training solutions that are more engaging, effective, and scalable than traditional methods. Warehousing and logistics Vision Pro can help workers in the warehousing and logistics sector to improve their efficiency and accuracy in tasks such as picking, product assembly, inventory management, and delivery tracking. It can help large-scale organizations to keep up with surging demands by drastically improving worker efficiency and operational performance Workers can also get real-time information, guidance, and feedback, and collaborate with supervisors or colleagues remotely. Locating and tracking inventory in real-time using visual cues will expedite operations and reduce errors. Construction Businesses in the construction industry can benefit from Vision Pro’s enterprise training solutions, which can help them improve their safety and productivity on the job. They can learn and practice their skills in realistic and interactive VR simulations. Whether they need to inspect a site, review a design, check the quality, or perform maintenance, they can do it in a safe and immersive way with Vision Pro. Vision Pro can also help workers in challenging and dangerous professions, such as hyperbaric welding. They can learn how to weld various materials in a VR simulation that mimics real-world conditions. Engineers can also explore and inspect the floor plan in VR, and spot any problems or defects easily. Industrial manufacturing Vision Pro can take enterprise training for the industrial manufacturing sector to the next level. They will be able to create realistic and interactive VR simulations that help workers learn and practice their skills safely and engagingly. This can result in better safety, productivity, and business outcomes. For instance, machinists can learn how to test different types of materials in VR and get instant feedback on their performance. They can also simulate an assembly line and practice putting together the parts and spotting any issues or defects that might affect the product quality. Healthcare and medical Vision Pro can transform the medical and healthcare sector by offering new possibilities for learning and practicing medicine. It can enhance the efficiency and convenience of medical professionals and improve the safety, access, and outcomes of patients. With Vision Pro, enterprises in this sector can leverage the power of spatial computing to improve collaboration and communication among healthcare providers and enhance decision-making and patient care. They can also use VR to develop innovative solutions and procedures that can improve the health and well-being of people around the world. Conclusion Enterprise XR training solutions are set to be revolutionized by Vision Pro, offering heightened engagement, effectiveness, and scalability compared to traditional methods. Vision Pro can help enterprises achieve their goals and objectives by providing immersive and personalized experiences that suit the user’s needs and preferences. The future holds immense potential as Vision Pro paves the way for a truly immersive and interconnected world.

Apple Vision Pro and The Future of Enterprise Training

What Will the Spring of Technology Bring? Part 2 — Virtual Human

The term was coined by Neal Stephenson in his 1992 sci-fi novel . It has since been adopted to describe a certain vision of the future of computing and our relationship with technology. The metaverse is now often viewed as a profit-making asset in industries such as and . Stephenson has an advisory role in Jeff Bezos’ Blue Origin aerospace company and a position as “Chief Futurist” for Magic Leap, a company that specializes in producing technology for virtual reality (VR) and augmented reality (AR) applications (together known as XR or “extended reality”). predicted widespread XR adoption by the 2010s, but, despite Stephenson’s technological optimism, the 2022 metaverse craze is already dissipating. The metaverse failed to be widely adopted, and now the bubble is about to burst. metaverse Snow Crash Web3 Industry 4.0 Snow Crash Luis Bravo Martins and Samantha G. Wolfe intend as “a #responsible #business-oriented book for the #metaverse.” Martins is a chief marketing officer at UK start-up , which is in the business of “worker augmentation” and calls itself “an industrial AR solution to enhance workers on the manufacturing shop floor.” Wolfe is an adjunct professor at NYU and the Founder of PitchFWD, “a boutique strategic advisory firm focused on the emerging technologies ecosystem.” Together, they want to sell you something. In a previous self-published work entitled , written with Cathy Hackl under the “The Marketing Futurists,” Wolfe endorsed a vision of the metaverse that makes it seem more like a dystopian advertising hellscape. The foreword by Lisa Buyer ends: “The journey is just beginning, and you are here. Let’s immerse in the future of marketing today. Namaste.” Metaversed KIT-AR identified on LinkedIn as Marketing New Realities nom de plume —the use of technology to increase worker efficiency—is not a new concept. One example is the introduction of onboard computers in the trucking industry, which—as any trucker will tell you—was never primarily intended to increase the drivers’ safety and comfort but to ensure compliance and increase productivity at all costs. The book, then, is designed for bosses, not workers. Worker augmentation Metaversed Since 2022, the prophecies of a consumer-driven metaverse have given way to efforts to rebrand it as a business tool for collaborative productivity. This is because consumers have rejected the services and products offered by the metaverse for obvious reasons: they entail hardware more expensive than the latest iPhone for an experience worse than the one provided twenty years ago. And in addition, the reputation of the Web 2.0 brands that offer these products has been tainted. Despite the employee-first philosophy many of them once touted, they have been . Conditions of work at the tech giants are beginning to seem much less attractive just as those companies are trying to get us all to slip into their avatars—not for entertainment purposes but as part of a gruesome Second Life hemorrhaging employees officeverse. Martins’ and Wolfe’s book has arrived late to this party, and much of its thrust seems to be based on now outmoded early-2022 consumer optimism. The book predicts that XR will become bigger than the internet, connecting every object, place, and person on the planet, and will have significant effects on various industries that require new roles and skills. They also believe that it will alter the nature of human interactions and dramatically impact society since it will change how we define what we consider to be . (Amid these lofty but vague ambitions, they never explain how we currently define what is .) Even metaverse enthusiast Matthew Ball that this imaginary future offers little that might persuade people to trade in their smartphones for the more costly and cumbersome equipment that is not that different from just being online. But that hasn’t stopped diehard defenders of the metaverse from claiming that they will be able to sell us an alternative reality. real real agrees the metaverse requires to facilitate a user experience For Wolfe and Martins, one of the main selling points of the —especially the officeverse—is that the use of avatars can promote diversity and inclusion since they mask some of the user’s real-life characteristics, such as gender, age, race, and religion. Realistically, however, a culture in which people are known only by their avatars is unlikely to emerge in the current social climate. The widespread adoption of pronouns in bios and email signatures—primarily by people to whom pronoun misattribution is less likely—and the annual corporate scramble to proclaim support for LGBT initiatives during Pride Month suggests that people are more likely to voluntarily try to make these aspects of themselves more salient, rather than concealing them. metaverse don’t ask, don’t tell If employees are forced to exchange material reality for a digital environment, they are going to want some part in the reality-creation process. And at a time of what Guy Rundle as “the rise of the knowledge class,” companies are scrambling to adopt the “new progressivist” values of workers. has described Employers could use these new values to incentivize employees in a metaverse workplace. Moral authority to change workplace conditions could be delegated to individual workers as a reward for productivity. Plus, the ability to choose your own avatar could be presented as a perk: digital self-actualization could be an employment benefit—the metaverse could become a kind of corporate version of the pay-to-play games many of us already have on our phones. Major companies may soon be competing to be the most furry-friendly workplace in the metaverse. Achieve these goals for the firm, and you can join the metaverse on your own terms. But whether we are speaking of individual consumers or company workspaces, there must be widespread adoption of metaverse-related technologies to make it worth investing in such technologies in the first place. That puts those who are selling the technology today—like Wolfe and Martin—in the difficult position of pitching a thing that simultaneously offers to be a marketing paradise with a captive audience and a collective voluntary endeavor: an impossible repeat of the Facebook formula, dependent on hardware that has not yet been adopted. The tech simply does not—and may never—sell itself, regardless of the engineering achievements that it represents. The history of the fax machine provides a cautionary tale here. Although faxes existed for over a century, the technology failed to gain ground in the commercial marketplace until it had reached the very end of its usefulness, despite technological improvements and decreasing costs. Predictions that the metaverse is likely to be adopted by Gen Z, based on their use of platforms like Roblox that include an XR element, feel like a marketing Hail Mary. Setting aside the question of , the use of Roblox is only as indicative of the metaverse’s future as the incessant virtual feeding demanded by Tamagotchis in the 1990s has been predictive of today’s so-called trend. Roblox’s exploitative labor practices quiet quitting As was the case during the dotcom bubble, many companies outside Silicon Valley today are heavily invested in as yet unproven tech. ’s publisher, John Wiley & Sons, is one example. It surpassed $2 billion in revenue in 2022 while it was trying to expand from being merely an educational publisher to providing “tech-enabled” educational services. CEO Brian Napack has stated that his plan for Wiley is to “bridge the widening talent gap with career-connected education products and education services” (i.e., he wants to break into the market). If they want to succeed in that, Wiley would be well advised to maintain its integrity as a scientific publisher. In the past, both Wiley and its subsidiary Hindawi have a number of scientific publications retracted. These were the result of Chinese (organizations selling research paper authorship), aided by collusion rings and tools like SCIgen. Metaversed augmented work paper mills The IEEE (Institute of Electrical and Electronics Engineers), which has collaborated with Wiley since 2001, is also plagued by such problems. The 2022 IEEE Conference on Metaverse in Haikou, China, was funded by various sponsors connected to the Chinese state. That conference featured multiple keynote speakers whose contributions to the IEEE conference seem to have been later retracted on ethical grounds (as a search indicates). Before purchasing Hindawi in 2021, Wiley acquired IT educational service provider, mthree, in 2020, rebranding it as “Wiley Edge” and promising that it would be “future proof” and that clients’ onboarding costs would be minimized. Wiley is, therefore, intimately connected with the kind of worker augmentation the book advocates—so much so that it may sort out its research publishing issues in order to ensure that the growth of its “tech-enabled” solutions continues unimpeded. here In August 2022, the Biden administration acted on an Obama-era , that all federally-funded publications and research be made open access by the end of 2025. The White House has also and has promised more reforms to come. Perhaps Wiley has made its recent retractions in preparation for exploring alternative sources of funding, such as those acquired by the IEEE in Haikou. There are certainly incentives for Wiley to explore augmented work realities that allow new forms of outsourcing (i.e., worker exploitation). Brian Napack, Wiley’s president and CEO since 2017, also serves on the board of Burning Glass, a labor market analysis company. He likes to envision an “ ”—a buzzy manner of describing a labor market trapped in permanent precarity by the lack of open access to professional educational tools. memorandum mandating declared 2023 the Year of Open Science opportunity hyperloop If the by IT and related managerial positions, this could provide the captive audience for Wiley’s educational services, leaving the company ideally placed to lead the predicted —aka . In fact, Wiley-IEEE has just published a Chinese edition of their book . However, it is hard to imagine that the metaverse will really catch on—especially when compared to its recent competitor, AI. It is becoming increasingly clear that the concept of the metaverse signals little more than the hopes of multinational corporations who want to manipulate labor markets in order to generate profit for investors. metaverse were adopted fourth industrial revolution Industry 4.0 Industry 4.1 (edited by Zheng Fangtian) Systems theorist and Silicon Valley idol Buckminster Fuller once said, “We are called to be architects of the future, not its victims.” Ironically, author Robert Kiyosaki and others have borrowed this idea from Fuller’s 1981 book to justify their endorsement of cryptocurrency (in reality, Fuller something more akin to Universal Basic Income). Even if the metaverse is more than just a novelty that today’s tech giants are using to try to stay relevant, it is unlikely that you or I would want to experience the “reality” that Wolfe, Martins, Wiley, and Meta want us to buy. It’s just too obvious that this is about profits, not about people. And that is probably the biggest obstacle to its widespread voluntary adoption by consumers. Though perhaps businesses may be able to force it upon their employees. Rich Dad Poor Dad Critical Path was proposing Also published . here

The Metaverse is Now Corporate and Depressing

90 Stories To Learn About Ar

182 Stories To Learn About Augmented Reality

Understanding Image Types: A Developer's Guide

Comprender los tipos de imágenes: una guía para desarrolladores

Présentation des types d'images : guide du développeur

छवि प्रकारों को समझना: एक डेवलपर की मार्गदर्शिका

画像タイプを理解する: 開発者ガイド

Compreendendo os tipos de imagem: um guia do desenvolvedor

Hiểu các loại hình ảnh: Hướng dẫn dành cho nhà phát triển

了解图像类型：开发人员指南

Every June, tech bros, pension portfolio managers and Twitter threadbois the world over turn their eyes to Santa Clara as hosts its annual Worldwide Developer Conference (WWDC, or ‘dub-dub’). Apple Among other things, dub-dub is best known as being the birthing ground of almost all of Apple’s suite of ubiquitous consumer technology. The original iPhone, iPad, Apple Watch, and Apple TV and subsequent upgrades have all been launched at the event. This year, this stable of consumer devices is joined by an entirely new product for the first time since 2015. That new product is the VisionPro. There is nothing I can tell you about the VisionPro or Apple’s vision for the future of extended reality that a quick browse of , or the internet at large cannot. The point of this article is instead to try and observe: Twitter LinkedIn how we got here; why people are building computers for your face and the many divergent paths this may lead us on. Content Overview XR: A Brief History Headsets Today: Code New Worlds The Next Frontier for Spatial Computers Requests for Startups XR: A Brief History For the uninitiated, XR refers to - a catch-all blend of (AR, e.g Pokemon Go), virtual reality (VR, e.g. Oculus Rift), and ‘capital R’ Reality (e.g. Google Street View). Mixed reality is another term that exists largely on the same dimension as AR and thus will mostly be ignored here. extended reality augmented reality The modern vision for XR begins around 1935. This was the year in which published . In this short story, protagonist Dan Burke is disillusioned with the reality in which he finds himself. To remedy this, he puts on a pair of glasses that transport him to a world of eternal youth and happiness. The only catch is that he obeys its rules (_T&C_s, in modern parlance). Stanley Weinbaum Pygmalion’s Spectacles Fast forward a few decades and we begin to see the first builds of XR in real environments. This begins with Morton Heilig’s ‘ ’ (excluded from the timeline below), a sort of 4D cinema experience that provided the viewer with smells and a vibrating chair in addition to the sounds & sight of the film. Sensorama Also excluded from the timeline below is the first military application of XR. In 1961, became the first headset with motion-tracking technology. Philco Headsight Though it has since been discontinued, the Oculus Rift was the first headset to gain real commercial appeal. Following Oculus, many of big tech’s household names joined the action. Curiously, most had different visions for who the primary customer base for these headsets would be. The dominant expected use case for most was in immersive entertainment applications, more specifically in gaming. This was the path followed by Oculus, Sony through their , and with the Meta Quest line of products. PlayStation VR Meta Others chose to focus predominantly on enterprise. Early pioneers in this segment included and These early movers focused their aims on things like enterprise collaboration platforms (much like some of the metaverse plays described below), emergency response training, and medical education. HTC Vive Magic Leap. Then, the hype cycle. Corporations the world over began to envision what it would look like if we took our lived environment but made it *virtual*. The vision was essentially , but this time with less porn and more corporate work stuff. metaverse SecondLife In the wake of COVID-19, the rise of remote work created a supposed need for collaborative digital environments. joined the party with its own industrial metaverse. Accenture launched a ‘metaverse services’ division. Facebook changed its name to Meta and began spending $1bn per month on this vision, culminating in this brilliant . Microsoft presentation I have had fun panning the metaverse, but the hype cycle around it was an important moment in raising awareness for the arrival of what Apple would refer to this week as spatial computing. As this space evolves, there is every chance that many of these ideas come back into vogue. Other use cases for commercial XR varied from the practical to the benign. Some of them are beginning to become commonplace in popular web back-ends. You take your pick as to which fits which description from the below: Virtual tourism Virtual home inspections Virtual clothing fitting (If you’re going to watch any of these videos, do yourself a favour and make it this one) Corporate training & education Surgical simulations Product design & prototyping Disney-fied sporting telecasts. Most of the existing devices listed above tried to achieve some sort of balance with the use cases they were targeting. The most successful to date by sales, the Meta Quest, did so by positioning hard as a gaming device. Normally, this would be a lesson. However, if anyone is to buck this trend it would be Apple. Apple has a broad suite of already ubiquitous products (1.6bn active iPhones, to name one) that can and will serve as secondary endpoints for VisionPro applications. While Apple’s inaugural demos for the VisionPro were all hosted within working environments, it is more than likely that it will end up being a general-purpose VR. I foresee its role as being a form of spatial augmentation for Apple’s existing ecosystem of products (e.g Maps, FaceTime, Siri et al) as its vision for computers shifts modalities. More on other potential applications later in this piece. Headsets Today: Code New Worlds Spatial Computing Rather than ride the wave of today’s tech buzzwords, Apple used its informational market-making power to stamp authority on a new one at WWDC. https://twitter.com/tim_cook/status/1665806600261763072?embedable=true While it sounds intuitive enough on the surface, let’s delve a little bit deeper into what this term may actually convey going forward. Besides the obvious branding benefits of coining a buzz phrase, the use of the term ‘computing’ feels extremely deliberate. It represents a logical next step from previous eras of ‘desktop computing’ and ‘mobile computing’. Just as Apple has done with the previous generations of personal computers, it will aim to make this concept and terminology ubiquitous. To quote Tim Cook from WWDC: “In the future, you’ll wonder how you led your life without augmented reality” Besides the change in hardware, how is spatial computing functionally different from its predecessors? Arguably the most important immediate shift will be from working in 2D to working in 3D. Tasks such as visualization, workspace navigation (i.e. clicking and moving things), and collaboration will all be enabled in ways not seen under previous modalities. In the longer term, and are more likely to prove the groundbreaking features of spatial computing. contextual awareness interconnectivity refers to the ability of the computer to; Contextual awareness i) adapt existing recommendations and notifications to a user’s sensory context (i.e recommendations based on location and movement patterns, automatic changes to screen presentations based on user habits, etc) and ii) provide customized (possibly agentic) assistance to users as they complete tasks in extended reality. refers to the user’s ability to extend extended reality beyond the device. Beyond just working and collaborating on holographic interfaces, users can control and interact with other devices in their environment. To take a boring example, users can change smart home settings from within the device. Interconnectivity In the longer run, interconnectivity may see Siri’s role as a virtual assistant evolve from voice commanded search aggregator to a living assistant that can help complete tasks from within the device based on the user’s previous habits and preferences. As the baseline capabilities of autonomous agents advance, the combination of Apple’s distribution network and the familiarity of Siri may make it the killer technology for bringing personal agents to the world en masse. Information Presentation As mentioned above, Apple’s demos of the VisionPro focused almost exclusively on working environments. Make no mistake, this first generation is very much geared toward information workers. How will they use it? Firstly, there is obvious and trodden ground. Apple will re-do and re-design all the enterprise XR solutions we have seen before. They will provide resources for institutions to train their employees, students, and leaders. They will provide a bevy of try-on solutions, gaming applications, and cloud TV solutions that we have seen before. One relatively novel example that was showcased as part of the launch was the ‘Minority Report’ style computer interfaces. Instead of having to boot up a physical device every time you want to work on a file, access it from anywhere in the cloud in an instant. But how does this make the core experience of personal computing itself different? Some early ideas: at any given point in time Unlimited monitors with information work (speech, physical movement etc) Multimodal forms of interacting Intelligent browser navigation through sensory (e.g eye) tracking Entirely automatically tailored to user preference custom 3D screen presentations (boring but important) 3D data visualization Each of these will present its own universe of custom ideas that will take the place of extensions and app stores before it. AppleLM This tweet from founder Sulaiman Omar feels prescient. Apple does indeed have a history of sacrificing first-mover advantage for the sake of developing a better product over a longer timeframe. The VisionPro is a perfect example of this, being released 13 years after the first commercially available Oculus Rift. Cognosys AI Integrating language models into its headsets will open up a number of possibilities for Apple that aren’t available to incumbents without their own proprietary LLMs (note: Meta’s LlaMa may put them in the same boat - Balaji Srinivasan has published a highlighting their potential role in the spatial computing race). great thread https://twitter.com/balajis/status/1666506719084843009?embedable=true&thread=true Among the possibilities that this may usher in: trained on users’ real-life habits. Virtual gaming experiences Immediate through the facial ‘browser’. queries built into the headset and trained on user habits. Agentic virtual assistants that recommend actions at a point in time. Spatial recommendation engines Real-time translation. allowing users to re-arrange objects in their space. Environmental augmentation that plays out immediately as the user speaks it. Generative content creation But could it be done more elegantly… The Next Frontier for Spatial Computers Many a commentator proclaimed the death of Google Glass upon seeing it. Who would want to wear a computer on the face? Why would you opt into looking a bit like a dork? Many have noted that not a single Apple executive even tried a VisionPro on as part of the launch. There has to be a better reason for strapping one on than “because everyone else is”. The key question in the long run then becomes: Why would anyone want to strap an XR computer to their face when you can have one inside your brain? The classic counter to this would be the idea of ‘ ’. When everyone is wearing an XR mask, we can reproject photons to make it look as though no one is. Because reprojection involves creating entirely new photons altogether, people can even make themselves look entirely different from what they actually do without the mask. This would become a killer app for catfishing. reprojection The techno-optimist’s concept of living in augmented reality is akin to living in a perpetual lucid dream. Sleep mask when you go to bed, extended reality mask when you get out of it. Examine exhibit A below: The current paradigm of XR relies on these exoskeletal aids for users to navigate alternative realities. The reality is that we are already so close to no longer needing to depend on these aids. Enter the brain-machine interface. BMIs In the long run, the envisioned reality is likely to trump any form of mixed reality that exists today. I use the term envisioned reality here to describe any form of environmental alteration that is; a) completely customizable to the user’s requirements or demands and b) requires no wearable aids. The most obvious conduit for envisioned reality today is (BMIs). For a longer primer, I cannot recommend Tim Urban’s piece on ‘ ’ from as far back as 2017. brain-machine interfaces Wizard hats for the brain BMIs offer many of the same ‘spatial computation’ benefits as tools like the VisionPro but without the need for strapping a weighty device to the face. existing prototypes for the first commercial BMIs involve invasive implants that, understandably, make many uncomfortable (even if the reward is superhuman intelligence and memory). What about when these become less invasive, more accessible, and thus more palatable to the human public? Neuralink’s Extended reality abounds. Requests for Startups Mentions of batteries were oddly absent from Apple’s inaugural launch of the VisionPro. It is less obvious for ‘in the moment’ demos and usage than other design features like interfaces, compute, and weight (as referenced in ) but equally important when used day-in-day-out. Better battery systems for today’s XR devices. this thread by Kyle Samani Not since first dates moved from cafés to online chat rooms has the core wiring of how humans interact with one another been so fundamentally rocked. How can behavior be moderated in a world where people can make themselves appear to be anything? (note: Apple already seems to be working on some form of protocol which may protect against this). What kind of content becomes more pervasive when the range of presentation methods expands so broadly? How do people come together online? Social infrastructure for an XR world. proof-of-identity SecondLife, Minecraft, and Roblox all created huger-than-expected businesses off the back of secondary marketplaces for digital assets. These marketplaces were enabled by creative tools baked into the very fabric of the platforms themselves. Arm the Rebels: Creative Tools for XR. While there are many developers today already working on designs for new experiences etc for headsets and metaverse-style projects, how can we make this process more accessible for the layman (more specifically the 8-year Roblocker) to create worlds of their own imagination? What will be the second coming of Minecraft for the augmented reality or ‘envisioned reality’ paradigm? A large part of the appeal of virtual/augmented/envisioned reality is the idea of living a different life from the one you have now. Much like Dan Burke in anyone will now be able to optimize their lived experiences within virtual worlds. Designs for Lived Experiences (i.e Qualia). Pygmalion’s Spectacles, In the event that such technologies do become ubiquitous, this creates a giant universal market for synthetic instances of subjective experience. The design scope is theoretically infinite. How many ways can you experience pure ecstasy? Nostalgia? The pool of rewards for those who can bring these kinds of experiences to the world through some form of extended reality are enormous. qualia - Also underappreciated may be the development of tools for users to capture or develop these qualia themselves. Is there a market for the ‘recording’ of one’s own lived experiences to share with others? How can people engineer new experiences to take to market and compose atop other open-source qualia? How can we provide guarantees of privacy for personal qualia if they begin to be used in recommendation engines? For the curious, check out the to learn more. Qualia Research Institute One thing that XR still has in common with other computing paradigms before it is its top-down nature. Users are at the whims of Apple for how they want to interact with their phones. If they want to protest this, they can select another one of the depressingly finite options on the market. To date, composability and customisability in hardware have been extremely limited. Open-Source Hardware & XR Software Marketplaces. But what if we were to open-source the development of headsets themselves through decentralized labs? Active participants could work to modify and iterate upon different hardware specifications to their preference. Modular designs could allow for customization at the layman's level. Just as importantly, these open-source labs would have no incentive to create closed-loop systems for software development. Developers could experiment with, ship, and deploy code that could be available to all headsets at the rate that they can build it. Such open-source development would represent a step change in the way people interact with both software and hardware. For some inspiration, check out the great work the team at is already doing. Auki Labs Linked to the point above, the top-down software ecosystem of today means that incentives built into software tend to be warped. Rather than going all out on user utility, market incentives corner developers into building sub-optimal applications that aim to maximize metrics like ‘average session time’ or ‘clickthrough rates’. Private User Agents. The opportunity that open-source development studios would have to outcompete legacy tech companies by building in private, natural language user agents that act as functional assistants to their users is immense. Also published here. The lead image for this article was generated by HackerNoon's AI Image Generator via the prompt "Augmented reality headsets"

Facial Computing: A Brief History, and the Promising Future, of Personal XR

ফেসিয়াল কম্পিউটিং: একটি সংক্ষিপ্ত ইতিহাস, এবং ব্যক্তিগত XR এর প্রতিশ্রুতিশীল ভবিষ্যত

Facial Computing: Eine kurze Geschichte und die vielversprechende Zukunft von Personal XR

Cómputo facial: una breve historia y el futuro prometedor de Personal XR

L'informatique faciale : une brève histoire et l'avenir prometteur du Personal XR

फेशियल कम्प्यूटिंग: ए ब्रीफ हिस्ट्री, एंड द प्रॉमिसिंग फ्यूचर, ऑफ पर्सनल एक्सआर

フェイシャル コンピューティング: パーソナル XR の簡単な歴史と有望な未来

얼굴 컴퓨팅: 개인용 XR의 간략한 역사와 유망한 미래

Computação facial: uma breve história e o futuro promissor do Personal XR

Yüz Bilişimi: Kişisel XR'nin Kısa Tarihi ve Umut Veren Geleceği

Điện toán khuôn mặt: Lịch sử tóm tắt và Tương lai đầy hứa hẹn của Personal XR

面部计算：个人 XR 的简史和充满希望的未来

X-R SOLUTION

x-rsolutions.com

-- employees

Since n.d.

COMPANY RANKING

X-R SOLUTION

EVERGREEN INDEX #12163

Representing this company? Click here to claim and customize this page!

X-R Solution WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

X-R SOLUTION

x-rsolutions.com

-- employees

Since n.d.

COMPANY RANKING

X-R SOLUTION

EVERGREEN INDEX #12163

RELATED COMPANIESRANK

STORIES ABOUT X-R SOLUTION

Representing this company? Click here to claim and customize this page!

X-R Solution WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!