Streaming video for medical training: it’s all about trade-offs

image medtech2
Video captured, for the purposes of training, of a medical surgery in process. Credit: VRFocus.

 

On Thursday 14th of April, 2016, the world’s first operation on a British cancer patient was live-streamed using 360-degree cameras. The event was organised by VRinOR, a team put together by Medical Realities—a healthcare company specialised in virtual reality—and Mativision, who provided the cameras enabling live-streaming. Their idea is that “a single operation could be used to teach not only a couple of students present in the limited-spaced operating theatre, but thousands wherever they are in the world” explains Shafi Ahmed, the cancer surgeon and co-founder of Medical Realities who performed the operation at the Royal College Hospital in London. More than 54,000 individuals worldwide watched the 3-hour operation from their desktop, smartphone, or virtual-reality headset.

During the UCL MedTech Conference 2016, Shafi Ahmed reveals that two thirds of the world’s population do not have access to safe and affordable surgery due to a global shortage of surgical skills. He estimates that 2.2 million additional surgeons are required. A key challenge in addressing this issue is the training of surgeons. The current model—limiting the number of students present in the operating theatre—is simply not scalable. He believes that global connectivity is the right path toward a change in surgeons’ education. The goal of the company is to use cutting-edge technology to attract and train the large number of medical students required to respond to the current shortage of surgical skills.

The company had already made a first attempt at capturing a surgical operation to post online. In November 2015, Medical Realities produced a 360-degree video, recording a surgical operation at St Bartholomew’s Hospital in London. To reproduce a 360-degree view, the team used the GoPro Stereo Array, assembled from four GoPro cameras. This method is today widely recognised; those cameras now provide 4K resolution, and the resulting footage can be posted to Youtube (via the 360-degree video option). Even more conveniently, thanks to the cameras’ small size, they can be closer to one another, thereby minimising parallax issues. The resulting quality is considered high enough for viewing details in sensitive surgical cases. However, using several cameras involves a complex image-stitching process in post production that necessitates higher costs and makes online streaming unfeasible.

By using the Mativision cameras, however, the company was able to stream online – a step that will make a big difference in helping the company reach their goal. Steve Dann, co-founder of Medical Realities, points out that being the first to live-stream a surgical operation is a technological breakthrough that will excite and attract the massive audience of medical students they are looking to train. For example, universities organised special lectures for which students were provided with low-cost VR headsets and reunited to watch the event together. In addition, the Mativision cameras use a real-time stitching process that removes the costs and time associated with the manual stitching process used previously.

Surprisingly, the technology is not new. Mativision has been developing 360-degree cameras that are able to record and transmit in real-time since 2004. What changed, however, is the quality of the images produced. As mentioned with GoPro cameras, the major improvements observed in recent years have been the increasing resolution, specifically the ability to record at 4K. Resolution is currently seen as the benchmark of quality: the higher the resolution, the higher the number of pixels included in the space of the image, the more details that can be observed in the video.

However, there are issues to consider when increasing the resolution. Higher resolution leads to higher bitrate, which means there is more information to process and a larger filesize. Bigger files have two associated complications: they increase the time required to process the information, and quality is decreased due to the encoding algorithm (which treats each pixel better if there are less pixels in the space).

Furthermore, managing the size of the file when live-streaming is particularly challenging. Being able to produce higher resolution video does not necessarily mean that we are able to transmit it today. Additionally, there is a difference between the quality of image recorded and transmitted. Anthony Karydis, founder of Mativision, explains that although they can control the quality of images produced and transmitted from their server to anywhere worldwide, they cannot control the quality of images that are received. This will depend on the device on which the event is viewed, and on the internet connection of the participants. Video files recording and transmitting at 4K consist of a large amount of data, which cannot be processed by current average devices. This could lead to artefacts (i.e., a failure of quality), which occur when the amount of data in the file of the video is too large to be treated. The file then has to be compressed at a higher rate to reduce data, leading to a significant loss in quality. Because they cannot control the reception, the company has to work with a certain limit of data that will enable participants to view the video without artefacts.

For this reason, Anthony Karydis believes that “the resolution is not the absolute benchmark of success to deliver high quality when live-streaming”. For him, good quality after transmission is a trade-off between the resolution, the bitrate, and the frame rate. If we want to increase the resolution while managing the amount of data being treated, one has either to decrease the complexity of the encoding algorithm, or decrease the frame rate. Similarly, increasing the frame rate generates more data, so one has to decrease either the resolution or the complexity of the encoding algorithm. Finally, increasing the complexity of the encoding algorithm means more information to be treated in each pixel, so one has either to decrease resolution or frame rate. Achieving good quality after transmission is therefore a compromise between how much information can be encoded, as opposed to how good the quality of the image is.

The choice of the trade-off will vary according to the application. When it comes to the VRinOR application, the two most important things are reaching the highest number of individuals possible while providing a quality of image that allows participants to view the details of the surgical operation. According to these priorities, Mativision defined its trade-off. Firstly, the team tried to adapt a resolution that would fit any device. They therefore restricted it to 1200 x 1600 pixels, i.e., the number of pixels that can be displayed on a smartphone. A higher resolution is unnecessary, as it would create more information that would need to be processed but could not be displayed on smartphones. The team then restricted the frame rate to 25 frames per second. In this case, the frame rate is particularly low as there is not much in the way of large or fast movements during an operation. By managing a low resolution and a low frame rate, the team was able to maximise the complexity of the encoding algorithm required to perceive the details of the operations.

Mativision successfully managed the size of the video file by providing a quality that is high enough to view the details of the operation and low enough to be transmitted, so that everybody can access it. By redefining the benchmark of success for quality and adapting such trade-offs to their own application, Mativision has been able to help Medical Realities reach their goal: educating and training the largest audience possible. Karydis estimates that in a year, we will be able to transmit 4K everywhere. The lesson: managing quality is about adapting the trade-off between resolution, bitrate, and frame rate to a specific application – not just about attaining the highest resolution possible.

Books

Jobs

Quality Officer at Minnesota Department of Human Services Recruitment
Expires: 01/29/2021 Employer: Minnesota Department of Human Services Recruitment
Intelligence Analyst at Minnesota Department of Human Services Recruitment
Expires: 01/21/2021 Employer: Minnesota Department of Human Services Recruitment
Systems and Architecture Engineer, Entry Level at NXP Semiconductors
Expires: 02/15/2021 Employer: NXP Semiconductors
Plant Health Safeguarding Specialist, PPQ Technician, and various other entry-level positions at U.S. Department of Agriculture's Animal and Plant Health Inspection Service (APHIS)
Expires: 01/20/2021 Employer: U.S. Department of Agriculture's Animal and Plant Health Inspection Service (APHIS)
Behavioral Health Specialist - Qualified Mental Health Professional (QMHP) at Oregon Youth Authority
Expires: 01/29/2021 Employer: Oregon Youth Authority
Renewable Energy Engineer at California Public Utilities Commission
Expires: 01/24/2021 Employer: California Public Utilities Commission
Associate Toxicologist at California Department of Pesticide Regulation
Expires: 01/30/2021 Employer: California Department of Pesticide Regulation
Informatics Liaison & Public Health Informatician(Job Id 15513) at South Dakota State Government
Expires: 01/29/2021 Employer: South Dakota State Government - Department of Health
IC Design Engineer Internship at Power Integrations
Expires: 02/27/2021 Employer: Power Integrations
Physical Security Command Center Technician at Van Andel Institute
Expires: 02/14/2021 Employer: Van Andel Institute
Epidemiologist I(Job Id 15512) at South Dakota State Government
Expires: 01/22/2021 Employer: South Dakota State Government - Department of Health
Management Analyst I - Non-Merit at Baltimore County Government
Expires: 01/22/2021 Employer: Baltimore County Government
Assignment Assistant for Central Assignment Office - Non-Merit at Baltimore County Government
Expires: 01/27/2021 Employer: Baltimore County Government
Environmental Specialist - Science and Engineering Division at City of Tacoma
Expires: 02/02/2021 Employer: City of Tacoma
Probation Officer I - Adult Probation at Franklin County PA
Expires: 01/29/2021 Employer: Franklin County PA
Legal Secretary at California Department of Toxic Substances Control (DTSC)
Expires: 01/26/2021 Employer: California Department of Toxic Substances Control (DTSC)
IT Associate / Project Analyst at California Department of Toxic Substances Control (DTSC)
Expires: 01/22/2021 Employer: California Department of Toxic Substances Control (DTSC)
Manager Braille & Talking Book(Job Id 15491) at South Dakota State Government
Expires: 02/04/2021 Employer: South Dakota State Government - Department of Education
Assistant Electrical Chief (Operations & Policy Analyst 3) at Oregon Department of Consumer and Business Services
Expires: 01/27/2021 Employer: Oregon Department of Consumer and Business Services
Behavioral Health Senior Clinician at Fairfax County Government
Expires: 01/23/2021 Employer: Fairfax County Government - Fairfax County Human Resources
Behavioral Health Specialist II - Residential Treatment & Detox Services at Fairfax County Government
Expires: 01/23/2021 Employer: Fairfax County Government - Fairfax County Human Resources
Deputy Director Community Services Board / Administrative Operations at Fairfax County Government
Expires: 01/30/2021 Employer: Fairfax County Government - Fairfax County Human Resources
Occupational Safety Consultant (Occupational Safety Specialist 3) at Oregon Department of Consumer and Business Services
Expires: 02/02/2021 Employer: Oregon Department of Consumer and Business Services
Deputy Director, Department of Public Works and Environmental Services, Capital Facilities at Fairfax County Government
Expires: 01/30/2021 Employer: Fairfax County Government - Fairfax County Human Resources
Human Resources Intern at Virginia State Corporation Commission
Expires: 02/26/2021 Employer: Virginia State Corporation Commission
Member and Committee Vacancies at U.S. Congress
Expires: 01/27/2021 Employer: U.S. Congress
Grants and CJAB Coordinator at Franklin County PA
Expires: 01/28/2021 Employer: Franklin County PA
Engineering Technician I at City of Carlsbad
Expires: 02/09/2021 Employer: City of Carlsbad
Postdoctoral Fellow- Biologics Optimization: CHO Cell Engineering for Targeted Integration at Amgen
Expires: 02/26/2021 Employer: Amgen - Fulltime Recruiting
Associate, Data Strategy at Tempus, Inc.
Expires: 01/29/2021 Employer: Tempus, Inc.
Engineer I - II (Region Bridge Engineer) (Job Id 15468) at South Dakota State Government
Expires: 02/05/2021 Employer: South Dakota State Government - Department of Transporation
Trauma Program Manager/EMS Specialist (Registered Nurse)(Job Id 15477) at South Dakota State Government
Expires: 01/26/2021 Employer: South Dakota State Government - Department of Health