SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Tracking

Video Demonstrations

These videos demonstrate SurgSLOT's robust long-term tracking across unseen surgical datasets.

EV18: Zero-shot transfer to nephrectomy, a procedure type unseen during training.

RARP50: Semantic long-term tracking distinguishes the prompted target from visually similar instruments after prolonged absences.

NUH-Hyst: Long-term tracking on in-house clinical hysterectomy videos, each up to 30 minutes.

Annotation Platform: This supplementary demo presents our dedicated platform for efficient large-scale masklet annotation. Using the in-house NUH-Hyst data, annotators create and refine temporally consistent masklets through interactive prompting.

Abstract

Surgical scene understanding demands temporally consistent tracking of instruments and tissues. For clinical use, such tracking should generalize to new centers and procedure types, yet retraining for each of them is costly and not scalable. Interactive video object segmentation offers a way toward this generalization: the target is specified at inference by a first-frame visual prompt, so a model can generalize to unseen categories and new scenarios without retraining. However, training such a generalizable model demands spatio-temporal masklet annotations at a scale and procedural diversity that existing surgical benchmarks lack.

We fill this gap with iSurg, the largest surgical segmentation benchmark to our knowledge, spanning six procedure types with over 170k frames, 410k object masks, and 2.4k masklets, including an in-house clinical dataset of four 30-minute videos. Yet tracking over such long videos remains challenging: the target needs to be re-identified among visually similar objects after long absences, while long-term memory has to judge whether each stored frame genuinely depicts the target, both demanding a stable object-level semantic identity.

To this end, we propose SurgSLOT, a surgical segmentation generalist that segments any prompted target through two coupled modules built on this semantic identity: Temporal Semantic Learning learns it for re-identification, and Semantic-driven Long-term Memory reuses it to select reliable memory frames, suppressing identity drift over long procedures. On the SAM2 and SAM3 backbones, SurgSLOT reaches 81.0 and 82.8 Macro Average J&F under cross-dataset evaluation, surpassing their fine-tuned counterparts by 5.1 and 5.3 points and transferring zero-shot to an unseen procedure type and unseen object categories, with the SAM2 version running in real time at 68 FPS.

Introduction

Overview of the iSurg benchmark and SurgSLOT

Surgical videos vary substantially across clinical centers, imaging systems, and procedure types. A clinically useful model should therefore segment a user-prompted instrument or tissue in previously unseen settings without retraining. Interactive video object segmentation makes this possible with a first-frame visual prompt, but its development requires large-scale, temporally consistent masklet annotations that existing surgical datasets do not provide.

iSurg supplies this foundation and SurgSLOT addresses the remaining long-term tracking challenge: after a target leaves the field of view, the model must re-identify it among visually similar distractors while keeping reliable references in memory. SurgSLOT builds and reuses a stable object-level semantic identity to recover the correct target and suppress identity drift throughout extended procedures.

iSurg Benchmark

A large-scale, multi-procedure benchmark for cross-dataset and zero-shot surgical video segmentation.

735

Videos

176K

Frames

414K

Object Masks

2.45K

Masklets

6

Procedure Types

Covered Surgical Procedures
                Cholecystectomy: Endoscapes, CholecSeg8k, and CIS
Colonoscopy: BKAI-IGH, Kvasir-SEG, ClinicDB, and PolypGen
Gynecology: AutoLaparo, Hyst-YT, NUH-Hyst, and SurgAI3.8k
Rectal resection: DSAD-I and DSAD-V
Prostatectomy: GraSP and RARP50
Nephrectomy: EndoVis17 and EndoVis18

            

Proposed Method

Architecture overview of SurgSLOT. A semantic token forms a stable object-level anchor through Temporal Semantic Learning (TSL); Semantic-driven Long-term Memory (SLM) uses this anchor to choose reliable long-term references.

Key Innovations

iSurg: A Generalization-oriented Data Foundation

iSurg unifies 16 surgical datasets into temporally consistent masklets across six procedure types. Its evaluation protocol holds every test source out from training, with additional zero-shot settings for an unseen procedure type and unseen object categories. This scale, diversity, and split design make cross-dataset surgical segment-anything learning measurable and trainable.

Temporal Semantic Learning (TSL)

TSL decodes a per-frame semantic token and aggregates reliable target appearances into a slowly evolving semantic anchor. Temporal anchor coherence pulls reliable tokens toward this object identity and pushes away invalid tokens that carry no identity evidence, enabling re-identification after long absences. A vision-language contrastive objective further separates visually similar instruments.

Semantic-driven Long-term Memory (SLM)

SLM uses long-range sampling during training so memory attention learns to associate references across extended temporal gaps. At inference, anchor-aligned selection admits only buffered frames most consistent with the semantic anchor into long-term memory, choosing references by target identity instead of mask quality and suppressing drift from distractors.

Performance Highlights

Cross-dataset Generalization

81.0
SurgSLOT-SAM2
+5.1 over fine-tuned SAM2

82.8
SurgSLOT-SAM3
+5.3 over fine-tuned SAM3

Macro Average J&F with 3-click initialization

Real-time SAM2 Performance

68
FPS (Frames Per Second)

SurgSLOT-SAM2 on an A6000 GPU
81.0 Macro Average J&F

Long-term Tracking

SurgSLOT-SAM2 versus fine-tuned SAM2

Reliable recovery after long target absences
Average across long-duration subsets: 85.1 J&F (+5.4)
NUH-Hyst (30 min): 87.6 J&F

Zero-shot Procedure-type Generalization

J&F with 3-click initialization; nephrectomy is absent from training

SurgSLOT-SAM2

EV17: 86.6
EV18-I: 85.3
EV18-T: 74.8

SurgSLOT-SAM3

EV17: 87.6
EV18-I: 90.9
EV18-T: 76.2

Evaluation protocol: Every iSurg test subset is drawn from a source unseen during training. SurgSLOT further transfers to an unseen procedure type and to object categories never observed during training.

Related Work

Explore our related work on surgical video segmentation foundation models

Surgical Segmentation Foundation Models: Our research group develops methods that adapt promptable segmentation foundation models to surgical video understanding. Surgical SAM 2 achieves real-time performance (86 FPS) through efficient frame pruning, ReSurgSAM2 introduces text-guided referring segmentation with credible tracking initialization for language-driven surgical scene understanding, and UniSurgSAM extends promptable surgical segmentation to unified visual, textual, and audio interaction. SurgSLOT (this work) is a general semantic long-term tracking framework that runs on both SAM2 and SAM3. Its Temporal Semantic Learning and Semantic-driven Long-term Memory use a shared object-level identity to support cross-dataset, zero-shot, and long-duration surgical tracking.

Acknowledgement

We would like to thank National University Hospital (NUH) for providing the external hysterectomy videos. These resources were valuable for verifying the effectiveness and robustness of our model in real-world surgical scenarios.

Citation

        @article{liu2025surgslot,
          title={SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Tracking},
          author={Liu, Haofeng and Wang, Ziyue and Mishra, Sudhanshu and Gao, Mingqi and
                  Qin, Guanyi and Low, Chang Han and Kong, Alex Y. W. and Jin, Yueming},
          journal={arXiv preprint arXiv:2511.16618},
          year={2025}
        }