560 research outputs found
MURPHY: Relations Matter in Surgical Workflow Analysis
Autonomous robotic surgery has advanced significantly based on analysis of
visual and temporal cues in surgical workflow, but relational cues from domain
knowledge remain under investigation. Complex relations in surgical annotations
can be divided into intra- and inter-relations, both valuable to autonomous
systems to comprehend surgical workflows. Intra- and inter-relations describe
the relevance of various categories within a particular annotation type and the
relevance of different annotation types, respectively. This paper aims to
systematically investigate the importance of relational cues in surgery. First,
we contribute the RLLS12M dataset, a large-scale collection of robotic left
lateral sectionectomy (RLLS), by curating 50 videos of 50 patients operated by
5 surgeons and annotating a hierarchical workflow, which consists of 3 inter-
and 6 intra-relations, 6 steps, 15 tasks, and 38 activities represented as the
triplet of 11 instruments, 8 actions, and 16 objects, totaling 2,113,510 video
frames and 12,681,060 annotation entities. Correspondingly, we propose a
multi-relation purification hybrid network (MURPHY), which aptly incorporates
novel relation modules to augment the feature representation by purifying
relational features using the intra- and inter-relations embodied in
annotations. The intra-relation module leverages a R-GCN to implant visual
features in different graph relations, which are aggregated using a targeted
relation purification with affinity information measuring label consistency and
feature similarity. The inter-relation module is motivated by attention
mechanisms to regularize the influence of relational features based on the
hierarchy of annotation types from the domain knowledge. Extensive experimental
results on the curated RLLS dataset confirm the effectiveness of our approach,
demonstrating that relations matter in surgical workflow analysis
- …