Benjamin Warf, a renowned neurosurgeon at Boston Children's Hospital, at MIT.nano Immersion Lab. More than 3,000 miles away, his virtual avatar stands alongside Matheus Vasconcelos in Brazil as the resident performs delicate surgery on a model resembling a doll of a baby's brain.
Wearing a pair of virtual reality goggles, Vasconcelos can watch Warf's avatar demonstrate a brain surgery procedure before replicating the technique himself and while asking questions of Warf's digital twin.
“It's almost an out-of-body experience,” Warf says of watching his avatar interact with residents. “Maybe that's what it's like to have an identical twin?”
And that's the point: Warf's digital twin shortened the distance, allowing him to functionally be in two places at once. “It was my first training with this model and it performed excellently,” says Vasconcelos, a neurosurgery resident at the Faculty of Medical Sciences of Santa Casa de São Paulo in São Paulo, Brazil. “As a resident, I now feel more confident and comfortable applying the technique on a real patient under the guidance of a professor.”
Warf's avatar arrived through a new project launched by a medical simulator and augmented reality (AR) company EDUCATION. The company is part of the 2023 cohort of HOME.nanothe MIT.nano deep tech accelerator offering early-stage startups discounted access to MIT.nano labs.
In March 2023, Giselle Coelho, scientific director of EDUCSIM and pediatric neurosurgeon at Santa Casa de São Paulo and Sabará Children's Hospital, began working with technical staff at the MIT.nano Immersion Lab to create Warf's avatar. In November, the avatar was training future surgeons like Vasconcelos.
“I had the idea of creating Dr. Warf's avatar as a proof of concept and asked, 'Where would be the place in the world where they would be working on technologies like that?'” Coelho says. “Then I found MIT.nano.”
Capturing a surgeon
As a neurosurgery resident, Coelho was so frustrated by the lack of hands-on training options for complex surgeries that she built her own baby brain model. The physical model contains all the brain structures and can even bleed, “simulating all the steps of surgery, from incision to skin closure,” she says.
He soon discovered that simulators and virtual reality (VR) demonstrations shortened the learning curve for his own residents. Coelho launched EDUCSIM in 2017 to expand the variety and scope of training for residents and experts looking to learn new techniques.
Those techniques include a procedure to treat childhood hydrocephalus pioneered by Warf, director of neonatal and congenital neurosurgery at Boston Children's Hospital. Coelho had learned the technique directly from Warf and thought his avatar could be a way for surgeons who couldn't travel to Boston to benefit from his experience.
To create the avatar, Coelho worked with Talis Reks, the IT AR/VR/gaming/big data technologist at the Immersion Lab.
“For startups, access to a lot of technology and hardware can be very expensive when they begin their business journey,” explains Reks. “START.nano is a way to allow them to use and afford the tools and technologies we have in the MIT.nano immersion lab.”
Coelho and his colleagues needed high-fidelity, high-resolution motion capture technology, volumetric video capture, and a variety of other VR/AR technologies to capture Warf's dexterous finger movements and facial expressions. Warf visited MIT.nano on several occasions to be digitally “captured,” including performing an operation on the physical model of the baby while wearing special gloves and clothing with embedded sensors.
“These technologies have been used primarily for entertainment or VFX (visual effects) or CGI (computer generated imagery),” says Reks, “but this is a unique project, because now we are applying it to real medical practice and real learning. “. .”
One of the biggest challenges, Reks says, was helping develop what Coelho calls “holoportation”: transmitting Warf's 3D volumetric video capture in real time over the Internet so his avatar can appear in transcontinental medical training.
Warf avatar has synchronous and asynchronous modes. The training that Vasconcelos received was in an asynchronous mode, where residents can observe the avatar's manifestations and ask questions. The answers, delivered in a variety of languages, come from artificial intelligence algorithms that are based on previous research and an extensive bank of questions and answers provided by Warf.
In synchronous mode, Warf operates his avatar remotely in real time, Coelho says. “I could walk around the room, he could talk to me, he could guide me. It's amazing.”
Coelho, Warf, Reks and other team members demonstrated a combination of modes in a second session in late December. This demonstration consisted of a volumetric capture of live video between the Immersion Lab and Brazil, spatialized and visible in real time through AR headsets. It significantly expanded on the previous demonstration, which only transmitted volumetric data in one direction through a two-dimensional display.
Powerful impacts
Warf has a long history of training desperately needed pediatric neurosurgeons around the world, most recently through his nonprofit organization. neurochildren. Remote and simulated training has been an increasingly important part of training since the pandemic, she says, although he doesn't think it will ever completely replace in-person hands-on instruction and collaboration.
“But if we could actually one day have avatars, like this one of Giselle, in remote places, showing people how to do things and answering questions, without the cost of travel, without the cost of time, etc., I think it would be better . could be really powerful,” says Warf.
The avatar project is especially important for surgeons serving in remote and underserved areas such as Brazil's Amazon region, Coelho says. “This is a way to give them the same level of education they would get elsewhere and the same opportunity to be in contact with Dr. Warf.”
According to Coelho, a baby recently treated for hydrocephalus at an Amazonian clinic traveled by boat 30 hours for the surgery.
Training surgeons with the avatar, he says, “can change the reality of this baby and change the future.”
