Latent retrieval-augmented generation

ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation

Yiran Wang^1*, Zeyu Zhang^2*†, Ling Shao³, Hao Tang^2‡

¹USYD ²School of Computer Science, Peking University ³UCAS

^*Equal contribution. ^†Project lead. ^‡Corresponding author.

Comparison

Same prompt across five models

MoGenTS

ReMoDiffuse

TMR

ReMoMask

ReMoMask-2

01 /A man walks forwards and then stops.

MoGenTS

ReMoDiffuse

TMR

ReMoMask

ReMoMask-2

02 /A person is balancing on something.

MoGenTS

ReMoDiffuse

TMR

ReMoMask

ReMoMask-2

03 /A man walks forward in a clumsy way.

MoGenTS

ReMoDiffuse

TMR

ReMoMask

ReMoMask-2

04 /A person walks forward rather slowly.

MoGenTS

ReMoDiffuse

TMR

ReMoMask

ReMoMask-2

Qualitative Visualization

Caption-guided motion samples

Sample 01

A person walks backward in a straight line.

Sample 02

A person takes several sideways steps while keeping their body facing forward.

Sample 03

A person performs jumping jacks, opening and closing both arms and legs.

Sample 04

A person jumps upward from both feet and lands in the same spot.

Sample 05

A person waves their right hand above shoulder height while standing still.

Sample 06

A person bends forward at the waist with their arms hanging down.

Sample 07

A person gestures with both hands while explaining something.

Sample 08

A person points forward with their right arm.

Sample 09

A person makes a small jump forward from both feet.

Sample 10

A person takes slow sideways steps to the right.

Sample 11

A person reaches upward with the right arm.

Sample 12

A person jogs at an easy pace.

Sample 13

A person runs forward quickly, pumping both arms.

Sample 14

A person bends forward and reaches both hands toward their toes.

Sample 15

A person warms up by swinging both arms and bending their upper body forward.

Sample 16

A person gives a short bow.

Method

Retrieval, masking, and evaluation

Comparison between ReMoMask and ReMoMask-2 — ReMoMask vs ReMoMask-2.(a) ReMoMask retrieves motion clips from a database outside the generator, so every reference crosses the boundary between the two regions. (b) ReMoMask-2 rebuilds the database from the generator's own latents, so store, references and mask tokens share one substrate and nothing crosses.

Framework of ReMoMask-2 — Framework of ReMoMask-2. (a) The retrieval database is built offline by the frozen RVQ-VAE encoder E: each motion becomes a pre-quantisation latent grid, pooled and ℓ₂-normalised into a key. (b) A query projector φ, distilled from the frozen HBM retriever, maps text into that same key space, so cosine retrieval returns evidence already expressed in the generator's representation. (c) Topology Structured Masking (TSM) and (d) Semantic Spatial–Temporal Attention (SSTA) are inherited unchanged from ReMoMask, except that R_t may now enter the Value pathway.

Comparison on FID and inference cost — Comparison on FID and Inference Cost.

Effect of retrieval database coverage on HumanML3D — Effect of retrieval database coverage on HumanML3D.

Motion quality user study — Motion Quality User Study.

Text-motion correspondence user study — Text-Motion Correspondence User Study.

Database coverage protocol. Subsets of the latent-aligned database are drawn uniformly at random with a shared seed across points and are swapped in at inference time only: the generator, the query projector, and the training recipe are those of the deployed model throughout, so these points share the intervention protocol of the ablation ladder. The dashed line marks the no-retrieval setting; 100% is the deployed configuration.