创新背景

医疗行业是电子信息高科技在现实领域应用的先行者，随着科学技术的突飞猛进，数字医学得到了不断发展，混合现实技术为外科疾病精准化、个性化治疗提供了新的途径，在中国“十四五”规划和2035年远景目标纲要中均提出，要营造良好的数字生态，推广远程医疗，推进医学影像辅助判读、临床辅助等应用。混合现实（mixed reality，MR）是将虚拟现实（virtual reality，VR）和增强现实（augmented reality，AR）结合，通过对传统影像数据进行3D重建，再将3D模型通过3D眼镜投射到现实场景中，实现了虚拟世界和现实世界的融合，将其运用在医疗过程中可使诊断更加直观。

创新过程

在远程手术服务的场景中，基于5G、视频融合技术、AR技术等能力，将远程会诊专家的指导意见（手势或路径）实时融合叠加至手术现实画面中。同时采用H.264视频和AAC音频编码技术标准，双向隧道视频传输方案，使音视频流传输流畅自然。应用多源视频显示技术，支持多个视频源接入，具有高扩展性的优势，能够兼容开放手术、腔镜手术、达芬奇机器人和超声等检查设备的视频流信号。
将影像数据通过云服务器进行集中处理，最后分布给各终端，满足不同科室业务场景的需求，实现海量的影像数据按需存储和调取，呈现出的影像可以在二维和三维之间自由切换，保证医生会诊时所需要影像的各个维度，同时还为医生提供人工智能建模、高性能图像处理、三维影像重建等应用功能。

通过混合现实显示设备营造全新3D可视化环境，可在物理空间1:1显示真实环境和虚拟模型结合的全息影像，为医生同时呈现三维虚拟医学影像数据与真实空间环境，从更高的维度应用数据，深度挖掘影像数据信息，提高医生对影像数据的空间认知能力。另外，建立起全新的医患沟通模式，把传统的二维片子，通过三维全息方式展现给家属及患者。借助展现立体、直观、逼真、可视化等优势，使医患沟通和术前方案讨论更加顺利、便捷。
为各类内镜、显微镜等医疗设备提供手术可视化信息管理及实时业务协同，提供影像采集与处理、数据输出与备份、术中远程协作等功能，见图2。以多模态影像为基础，用于骨科、神经外科、胸外科、耳鼻喉科等外科手术，利用光学和磁学导航实现目标导航和路径导航。其中，在微创骨科手术导航中的应用，有望既不增加软组织损伤又可安全、精准、快速地辅助手术医生置钉。在3D打印模型上实现了高精度定位，模型位置偏差最小可达1 cm以内，为后期精准的临床手术提供安全保障。

2018年1月，本院骨科叶哲伟教授带领团队成功实施全球首例混合现实技术三地远程会诊，实现将混合现实用于多地远程会诊和远程手术指导。目前本院所构建的5G混合现实云平台已经与全国多家医院开展联合会诊手术，最远距离达4 500 Km。2021年2月24日-2月25日，本院与距离3 600 Km外的中国人民解放军新疆军区总医院及4 500 Km外的新疆叶城县解放军第九五〇医院通过混合现实云平台技术，成功为前线战士及边疆居民开展3台远程会诊手术，这也是国家平战结合、军民融合方针导向下边疆、边境前线地区首次实现的混合现实云平台远程手术，为破解偏远地区战创伤、骨疾病进行手术救治提供了解决思路。应用过程中，远程手术指导中借助5G，网络速度基本稳定在900 M/s，实现效果如同零距离面对面手术。同时，利用成像技术生成的全息影像与实际人体结构在手术部位的平均偏差为0.4 cm，支持高分辨率成像使医生能够对手术视野完整掌控，术中所见与术前计划完全相符。平台从投入使用至今已开展近千台手术，术中运行稳定，未出现异常。

创新价值

1.提升患者就医体验，改善医患沟通。
2.提高了医生诊疗效率并节省了人力投入。
3.改善了医学生的学习方式并降低了学习成本。
4.实现了院内院外数据实时共享。

创新关键点

结合人工智能图像处理、混合现实交互、增强现实音视频融合合、云计算等技术应用，构建了基于5G的混合现实影像云平台。实现院外远程医学会诊场景中影像数据的实时共享和精准远程手术指导以及院内手术场景中三维影像重建和精准手术辅助。

Application of mixed reality image cloud platform based on 5G in remote surgery

In the scenario of remote surgery service, based on 5G, video fusion technology, AR technology and other capabilities, the guidance opinions (gestures or paths) of remote consultation experts are fused and superimposed into the realistic picture of surgery in real time. At the same time using H.264 video and AAC audio coding technology standard, two-way tunnel video transmission scheme, so that audio and video stream transmission smooth and natural. The application of multi-source video display technology supports the access of multiple video sources, which has the advantage of high scalability and can be compatible with the video streaming signals of open surgery, endoscopic surgery, Da Vinci robot and ultrasound inspection equipment.
The image data are processed centrally through the cloud server and finally distributed to each terminal, which can meet the needs of business scenarios of different departments and realize the storage and retrieval of massive image data on demand. The images presented can be freely switched between 2D and 3D to ensure the various dimensions of the images required by doctors in consultation. At the same time, it also provides doctors with artificial intelligence modeling, high-performance image processing, three-dimensional image reconstruction and other application functions.
By mixing reality display device build new 3 d visualization environment, can be in physical space 1-1 shows real environment and the virtual model combined with the hologram, as the doctors at the same time rendering 3 d virtual medical image data and the real space environment, application data from the higher dimension, depth image data mining information, improve the space cognitive ability of doctors to image data. In addition, a new doctor-patient communication mode was established to show the traditional two-dimensional films to the families and patients through three-dimensional holography. With the help of three-dimensional, intuitive, realistic, visual and other advantages, doctor-patient communication and preoperative plan discussion are more smooth and convenient.
It provides surgical visual information management and real-time business collaboration for various endoscopes, microscopes and other medical devices, and provides functions such as image acquisition and processing, data output and backup, and intraoperative remote cooperation, as shown in Figure 2. Based on multimodal imaging, it is used in orthopedics, neurosurgery, thoracic surgery, otorhinolaryngology and other surgical operations, and uses optical and magnetic navigation to achieve target navigation and path navigation. Among them, the application in minimally invasive orthopedic surgery navigation is expected to not only not increase soft tissue injury, but also safely, accurately and quickly assist surgeons in nail placement. High precision positioning is achieved on the 3D printing model, and the minimum deviation of the model position can be less than 1 cm, which provides safety guarantee for accurate clinical surgery in the later stage.
In January 2018, Professor Ye Zhewei of the Department of Orthopedics of our Hospital led a team to successfully implement the world's first three-place remote consultation with mixed reality technology, realizing the application of mixed reality in multi-place remote consultation and remote surgical guidance. At present, the 5G mixed reality cloud platform built by our institute has carried out joint consultation surgery with many hospitals across the country, with the farthest distance reaching 4 500 Km. From February 24 to February 25, 2021, our hospital, the PLA General Hospital of Xinjiang Military Command located 3 600 Km away and PLA 950th Hospital located 4 500 Km away in Yecheng County, Xinjiang successfully carried out three remote consultation surgeries for frontline soldiers and border residents through the mixed reality cloud platform technology. This is also the first remote surgery on a mixed reality cloud platform in frontier and frontier areas under the guidance of the national policy of combining peacetime and war and civil-military integration, which provides solutions for the surgical treatment of war wounds and bone diseases in remote areas. In the application process, with the help of 5G in the remote surgery guidance, the network speed is basically stable at 900 M/s, achieving the same effect as zero-distance face-to-face surgery. At the same time, the average deviation between the hologram generated by imaging technology and the actual human body structure at the surgical site was 0.4 cm, which supported high-resolution imaging so that doctors could fully control the surgical field, and the intraoperative vision was completely consistent with the preoperative plan. Nearly 1,000 operations have been performed on the platform since it was put into use, and the operation has been stable without any abnormalities.