The first metaverse experiments? Check out what’s going on in medicine
Surgeon Shafi Ahmed poses for a picture with a Microsoft HoloLens headset in his operating room at Royal London Hospital on Thursday January 11, 2018.
Bloomberg | Bloomberg | Getty Images
The Metaverse, the next big thing of the digital world, is touted as the internet domain where animated avatars of our physical selves can virtually perform all kinds of interactions, from shopping to games to traveling – someday. Wonks say it could take a decade or more for the technologies needed to catch up with the hype.
For now, however, the healthcare industry is leveraging some of the essential components that will ultimately comprise the metaverse – Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Artificial Intelligence (AI) – such as, as well as the software and hardware, to implement operate their applications. For example, medical device companies are using MR to assemble surgical instruments and design operating theaters, the World Health Organization (WHO) is using AR and smartphones to train Covid-19 responders, and psychiatrists are using VR to treat post-traumatic stress (PTS). among combat soldiers and medical schools are using VR for surgical training.
Facebook, Oculus and Covid
Since Facebook – now Meta Platforms – acquired Oculus and its VR headset technology for $ 2 billion in 2014, numerous healthcare applications have been developed. One of the most recent was a collaboration with Facebook Reality Labs and Nexus Studios, as well as the WHO Academy. The organization’s R&D incubator has developed a mobile learning app for health workers fighting Covid-19 around the world. One of the training courses includes AR to simulate the correct techniques and processes for putting on and taking off personal protective equipment on a smartphone. With content in seven languages, the app is geared towards the needs of 22,000 health workers worldwide, which was surveyed by the WHO last year.
Oculus technology is being used at UConn Health, the University of Connecticut’s medical center in Farmington, Connecticut, to train residents in orthopedic surgery. Educators have partnered with PrecisionOS, a Canadian medical software company that offers VR training and educational modules in orthopedics. By wearing Oculus Quest headsets, residents can visualize a range of surgical procedures in 3D, such as inserting a needle into a broken bone. Because the process is performed virtually, the system allows students to make mistakes and get feedback from faculty to consider on their next attempt.
While the Metaverse is under construction, “we see a great opportunity to continue the work of Meta, which is already supporting the health effort,” said a Meta spokesperson. “As Meta’s experiences, apps, and services evolve, it’s safe to assume that health strategies will play a role, but it’s far too early to say how those might intersect with third-party technology and providers.”
When Microsoft launched its HoloLens AR data glasses for commercial development in 2016, the first users were Stryker, the medical technology company in Kalamazoo, Michigan. In 2017, the AR device began to be used to improve processes in the design of operating rooms for hospitals and operation centers. Because operating rooms are shared by different surgical services – from general surgery to orthopedics, cardiac surgery, and others – lighting, equipment, and surgical instruments vary depending on the procedure.
Recognizing the potential of the HoloLens 2 in advancing OR design from 2D to 3D, Stryker engineers are able to create shared ORs using holograms. The MR experience visualizes all people, devices and facilities without the need for physical objects or people to be present.
Zimmer Biomet, a medical technology company based in Warsaw, Indiana, recently unveiled its OptiVu Mixed Reality Solutions platform, which uses HoloLens devices and three applications – one uses MR to make surgical instruments, and another collects and stores data related to the Advance and track patient’s progress. after surgery and a third one that allows doctors to share an MRI experience with patients before a procedure.
“We are currently deploying the HoloLens in pilot form with remote support in the US, EMEA and Australia,” said a Zimmer Biomet spokesman. The technology has been used for remote case coverage and training programs, and the company is developing software applications on the HoloLens as part of data solutions that focus on pre- and post-treatment, the spokesman said.
Microsoft’s holographic vision of the future
In March, Microsoft unveiled Mesh, an MR platform powered by its Azure cloud service that allows people in different physical locations to participate in 3D holographic experiences on different devices, including HoloLens 2, a range of VR headsets, Smartphones, tablets and PCs. In a blog post, the company featured avatars of medical students learning about human anatomy, gathering around a holographic model, and peeling muscles to see what’s underneath.
Microsoft sees many opportunities for its MR technology and secured a $ 20 billion contract with the US military in March for use with soldiers.
In real-world applications of AR medical technology, Johns Hopkins neurosurgeons performed the institution’s very first AR surgeries on live patients in June. During the first procedure, doctors placed six screws in a patient’s spine during a spinal fusion. Two days later, a separate team of surgeons removed a cancerous tumor from a patient’s spine. Both teams wore headsets from Augmedics, an Israeli company equipped with a see-through eye display that projects images of a patient’s internal anatomy, such as bones and other tissues, based on CT scans. “It’s like having a GPS navigator in front of your eyes,” said Timothy Witham, MD, director of the Johns Hopkins Neurosurgery Spinal Fusion Laboratory.
At the University of Miami’s Miller School of Medicine, instructors from the Gordon Center for Simulation and Innovation in Medical Education use AR, VR, and MR to train emergency first responders to manage trauma patients, including those who have had a stroke, heart attack, or gunshot wound . Students practice life-saving cardiac procedures on Harvey, a lifelike mannequin that realistically simulates almost any heart disease. With VR headsets, students can “see” the underlying anatomy that is graphed on Harvey.
“In the digital environment, we are not tied to physical objects,” said Barry Issenberg, MD, professor of medicine and director of the Gordon Center. Prior to developing the virtual technology curriculum, students had to be physically on-site and train on real trauma patients. “Now we can guarantee that all learners will have the same virtual experience regardless of their geographic location.”
Since its inception in 1999, the University of Southern California Institute for Creative Technologies (ICT) has developed VR, AI, and other technologies to treat a wide variety of medical and mental illnesses. “When I first got involved, technology was the stone age,” says Albert “Skip” Rizzo, psychologist and director of medical virtual reality at ICT, and remembers his tinkering with an Apple IIe and a Game Boy handheld Console. Today he uses VR and AR headsets from Oculus, HP and Magic Leap.
Rizzo helped develop a VR exposure therapy called Bravemind, which aims to alleviate PTS, particularly among veterans of the wars in Iraq and Afghanistan. During exposure therapy, a patient, under the guidance of a trained therapist, confronts their trauma memories through simulations of their experiences. With a headset, the patient can be immersed in various virtual scenarios, including a city in the Middle East and environments with desert roads.
“Patients use a keyboard to simulate people, insurgents, explosions, even smells and vibrations,” said Rizzo. And instead of relying solely on imagining a particular scenario, a patient can experience it in a safe, virtual world as an alternative to traditional talk therapy. Evidence-based Bravemind Therapy is now available in more than a dozen Veterans Administration hospitals where it has been shown to significantly reduce PTS symptoms. Other randomized controlled trials are ongoing.
While Big Tech continues to build the metaverse together with software and hardware companies, universities and other R&D partners, the healthcare industry remains a real testing ground. “Although the Metaverse is still in its infancy, it has enormous potential for transforming and improving healthcare,” wrote Paulo Pinheiro, Head of Software at Sagentia Innovation, based in Cambridge, UK, on the consultancy’s website. “It will be fascinating to watch the situation evolve.”
Stay connected with Healthy Returns
For a front row seat at CNBC Events, hear directly from the visionary leaders, innovators, executives and influencers who stand on the stage on The Keynote Podcast. Now, listen whichever way you get your podcasts.
For more exclusive insights from our reporters and speakers, sign up for our Healthy Returns newsletter to get the latest news straight to your inbox every week.