Streaming video for medical training: it’s all about trade-offs
On Thursday 14th of April, 2016, the world’s first operation on a British cancer patient was live-streamed using 360-degree cameras. The event was organised by VRinOR, a team put together by Medical Realities—a healthcare company specialised in virtual reality—and Mativision, who provided the cameras enabling live-streaming. Their idea is that “a single operation could be used to teach not only a couple of students present in the limited-spaced operating theatre, but thousands wherever they are in the world” explains Shafi Ahmed, the cancer surgeon and co-founder of Medical Realities who performed the operation at the Royal College Hospital in London. More than 54,000 individuals worldwide watched the 3-hour operation from their desktop, smartphone, or virtual-reality headset.
During the UCL MedTech Conference 2016, Shafi Ahmed reveals that two thirds of the world’s population do not have access to safe and affordable surgery due to a global shortage of surgical skills. He estimates that 2.2 million additional surgeons are required. A key challenge in addressing this issue is the training of surgeons. The current model—limiting the number of students present in the operating theatre—is simply not scalable. He believes that global connectivity is the right path toward a change in surgeons’ education. The goal of the company is to use cutting-edge technology to attract and train the large number of medical students required to respond to the current shortage of surgical skills.
The company had already made a first attempt at capturing a surgical operation to post online. In November 2015, Medical Realities produced a 360-degree video, recording a surgical operation at St Bartholomew’s Hospital in London. To reproduce a 360-degree view, the team used the GoPro Stereo Array, assembled from four GoPro cameras. This method is today widely recognised; those cameras now provide 4K resolution, and the resulting footage can be posted to Youtube (via the 360-degree video option). Even more conveniently, thanks to the cameras’ small size, they can be closer to one another, thereby minimising parallax issues. The resulting quality is considered high enough for viewing details in sensitive surgical cases. However, using several cameras involves a complex image-stitching process in post production that necessitates higher costs and makes online streaming unfeasible.
By using the Mativision cameras, however, the company was able to stream online – a step that will make a big difference in helping the company reach their goal. Steve Dann, co-founder of Medical Realities, points out that being the first to live-stream a surgical operation is a technological breakthrough that will excite and attract the massive audience of medical students they are looking to train. For example, universities organised special lectures for which students were provided with low-cost VR headsets and reunited to watch the event together. In addition, the Mativision cameras use a real-time stitching process that removes the costs and time associated with the manual stitching process used previously.
Surprisingly, the technology is not new. Mativision has been developing 360-degree cameras that are able to record and transmit in real-time since 2004. What changed, however, is the quality of the images produced. As mentioned with GoPro cameras, the major improvements observed in recent years have been the increasing resolution, specifically the ability to record at 4K. Resolution is currently seen as the benchmark of quality: the higher the resolution, the higher the number of pixels included in the space of the image, the more details that can be observed in the video.
However, there are issues to consider when increasing the resolution. Higher resolution leads to higher bitrate, which means there is more information to process and a larger filesize. Bigger files have two associated complications: they increase the time required to process the information, and quality is decreased due to the encoding algorithm (which treats each pixel better if there are less pixels in the space).
Furthermore, managing the size of the file when live-streaming is particularly challenging. Being able to produce higher resolution video does not necessarily mean that we are able to transmit it today. Additionally, there is a difference between the quality of image recorded and transmitted. Anthony Karydis, founder of Mativision, explains that although they can control the quality of images produced and transmitted from their server to anywhere worldwide, they cannot control the quality of images that are received. This will depend on the device on which the event is viewed, and on the internet connection of the participants. Video files recording and transmitting at 4K consist of a large amount of data, which cannot be processed by current average devices. This could lead to artefacts (i.e., a failure of quality), which occur when the amount of data in the file of the video is too large to be treated. The file then has to be compressed at a higher rate to reduce data, leading to a significant loss in quality. Because they cannot control the reception, the company has to work with a certain limit of data that will enable participants to view the video without artefacts.
For this reason, Anthony Karydis believes that “the resolution is not the absolute benchmark of success to deliver high quality when live-streaming”. For him, good quality after transmission is a trade-off between the resolution, the bitrate, and the frame rate. If we want to increase the resolution while managing the amount of data being treated, one has either to decrease the complexity of the encoding algorithm, or decrease the frame rate. Similarly, increasing the frame rate generates more data, so one has to decrease either the resolution or the complexity of the encoding algorithm. Finally, increasing the complexity of the encoding algorithm means more information to be treated in each pixel, so one has either to decrease resolution or frame rate. Achieving good quality after transmission is therefore a compromise between how much information can be encoded, as opposed to how good the quality of the image is.
The choice of the trade-off will vary according to the application. When it comes to the VRinOR application, the two most important things are reaching the highest number of individuals possible while providing a quality of image that allows participants to view the details of the surgical operation. According to these priorities, Mativision defined its trade-off. Firstly, the team tried to adapt a resolution that would fit any device. They therefore restricted it to 1200 x 1600 pixels, i.e., the number of pixels that can be displayed on a smartphone. A higher resolution is unnecessary, as it would create more information that would need to be processed but could not be displayed on smartphones. The team then restricted the frame rate to 25 frames per second. In this case, the frame rate is particularly low as there is not much in the way of large or fast movements during an operation. By managing a low resolution and a low frame rate, the team was able to maximise the complexity of the encoding algorithm required to perceive the details of the operations.
Mativision successfully managed the size of the video file by providing a quality that is high enough to view the details of the operation and low enough to be transmitted, so that everybody can access it. By redefining the benchmark of success for quality and adapting such trade-offs to their own application, Mativision has been able to help Medical Realities reach their goal: educating and training the largest audience possible. Karydis estimates that in a year, we will be able to transmit 4K everywhere. The lesson: managing quality is about adapting the trade-off between resolution, bitrate, and frame rate to a specific application – not just about attaining the highest resolution possible.