Carnegie Mellon University · 2026

SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation

Saksham Singh, Zheyuan Hu, Max Sobol Mark, Jeffrey Yu, Zackory Erickson, Aviral Kumar
Carnegie Mellon University
Paper Code BibTeX

SeeQ (Subtask-elicited Q-functions) learns Q-values for the active subtask rather than the entire task. It first predicts the subtask in natural language, one token at a time, then estimates its value.

We build SeeQ on a pretrained PaliGemma backbone, pretrain it on open-source robot manipulation data, and fine-tune it on downstream tasks. Using these values for best-of-N action selection improves policy performance on real-world, long-horizon bimanual manipulation tasks.

SeeQ overview. Left: diverse robot data with observations, actions, sparse rewards and subtask labels. Middle: a VLM-backed Q-function pretrained to predict the subtask and its value, then fine-tuned on a downstream task. Right: at test time the policy samples action chunks and SeeQ scores each one for the predicted subtask.
SeeQ overview. We train a generalist Q-function on diverse robot data using a pretrained vision-language model (VLM), then fine-tune it on a downstream task to steer a base policy at test time. Temporal-difference (TD) learning estimates Q-values for the active subtask, shortening the prediction horizon. The Q-function first predicts the subtask in natural language, then estimates its value, so a human does not need to specify the subtask at deployment.
  1. Q-functions can improve generalist policies by ranking candidate actions. But with sparse task-level rewards, they must assign credit over long horizons, learn through difficult Bellman backups, and rely on broad data coverage. Direct regression to Monte Carlo returns avoids TD learning, but limits the value function to the returns represented in its training demonstrations.
  2. SeeQ learns Q-values for the active subtask. The shorter prediction horizon makes TD learning more effective. Subtask span annotations already present in offline robot data define this decomposition, allowing SeeQ to learn from diverse datasets that may include suboptimal behavior.
  3. The Q-function predicts the active subtask in natural language, one token at a time, and conditions its value estimate on that prediction. At test time, it needs no human subtask annotations or separate subtask predictor. Without the subtask next-token prediction loss, steering performs worse than the base policy.
  4. SeeQ improves best-of-N policy steering on four real-world tasks across two bimanual robot platforms. We initialize it from PaliGemma, pretrain it on RoboCOIN's bimanual robot data, and fine-tune it on each target task. Steering a fine-tuned π0.5 policy increases successes on shirt-hang from 10/24 to 22/24 and improves results on the other three tasks.

Policy steering on real-world tasks

Each pair compares π0.5, fine-tuned on the target-task data, with the same policy steered by SeeQ using best-of-N action selection. Both use the same policy weights, scene, and task instruction. Each row pairs BC and SeeQ rollouts; SeeQ’s predicted subtask and value follow its video.

Hang a shirt

10/24 base → 22/24 with SeeQ

Remove a hanger from a rod, insert it into both sleeves of a T-shirt, and hang the shirt back on the rod. Guiding the hanger into each sleeve requires precise positioning and coordination between the arms.

Seal a container with a lid

10/24 base → 15/24 with SeeQ

Pick up a food-storage container lid, place it on the container, and close all four latching flaps. The lid must align precisely with the container rim.

Pack items into a box

9/24 base → 17/24 with SeeQ

Pack grocery items that vary in shape, size, and deformability into boxes. Instructions vary across episodes, specifying which objects to pack and where each should go. When an object starts on the opposite side from its target box, the robot must pass it between arms.

The task description specifies the target box for each object on the table. SeeQ follows this object-to-box mapping correctly in the successful rollouts below.

Take apart and sort Lego blocks

5/24 base → 10/24 with SeeQ

Separate assembled Lego blocks of different colors and place each in its instructed target tray.

Each action that removes one or more blocks from the assembly, as well as each individual pick and place operation, is treated as a separate subtask. Consecutive subtasks may share the same text during training, and we observe the same behavior during inference. The task description specifies which tray receives each block color: episode 1 uses red → red, green → green, and blue → blue; episode 2 uses red → green, green → blue, and blue → red (block color → tray color).

Results

We compare successes over 24 trials for the base policy and the same policy steered by SeeQ using best-of-N action selection.

Policy steering results successes out of 24 trials
π0.5 base policyπ0.5 + SeeQ
shirt-hang
lid-sealing
packing
lego-disas­sembly

BibTeX

@article{singh2026seeq,
  title  = {SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation},
  author = {Singh, Saksham and Hu, Zheyuan and Mark, Max Sobol and Yu, Jeffrey and Erickson, Zackory and Kumar, Aviral},
  year   = {2026}
}