Q-OIL: Optimistic Exploration in Learning from Human Interventions

Pranav Teegavarapu1,* Sriyash Poddar1,* Rico Qi1 Cleah Winston1 Krisha Khandelwal1 Aykut Onol2 Natasha Jaques1,‡ Abhishek Gupta1,‡

1University of Washington   2Toyota Research Institute

*Equal contribution   ‡Equal advising
Conference on Robot Learning (CoRL) 2026

Humans are suboptimal at correcting robot actions and provide delayed reactions. We analyze real human data to build better benchmarking tools and a better algorithm. With improved benchmarking, we find more reliable rankings in simulation. Our method, Q-OIL, uses these corrections to guide exploration rather than as supervision, reaching 10–50% higher success on real robots (including VLA post-training) with almost 30% less human effort.

Humans can intervene to correct robot actions.
Try Q-OIL in your browser

The robot learns to reach the pink ring from scratch. Take over to help it: click a cell to steer it there, or click the maze and use the arrow keys.

Algorithm
Show
In control
–
Episodes
0
Success, last 20
0%
Takeovers, last 20
0%
SuccessSteps under takeoverRobot's best route

The problem

Simulated humans don't act like real ones

SolutionBuild better benchmarking tools.

Existing methods are fast or optimal, not both

SolutionBuild an algorithm that is fast and can handle suboptimal corrections.

The fix

How do people intervene?

We asked 8 participants to supervise a simulated robot as it learned 3 manipulation tasks, collected about 2,000 episodes (200k steps) of their interventions, and analyzed when and how they stepped in.

Left: progress over a 12-step window separates intervention states from other states, while the instantaneous action value does not. Right: value curves where takeovers follow plateaus or drops.

Takeovers follow plateaus or drops in value, not single bad actions.

DesignSo Q-OIL keeps the bonus from leaking back to the states before a takeover.

Bar chart: intervention actions are about three times more suboptimal than the reference policy.

Takeover actions are about 3× more suboptimal than an expert policy.

DesignWe use interventions to guide exploration rather than as optimal supervision targets.

At intervention states, the converged policy's actions have higher value than the human's actions.

At intervention states, the converged policy selects different, higher-value actions than the human.

DesignThe influence of corrections decays as interventions become rare, so learning shifts back to the task reward.

Rank correlation with real-user algorithm rankings: PAM 1.00, action-suboptimality 0.77, random 0.37.

PAM, our progress-aware intervention model, matches real users' ranking exactly. Its coefficients k, δ, α, β are learned from real intervention data.

P(intervene) = α  if  V*(st) − V*(st−k) < δ,  else  β

BenchmarkSo we compare every method with PAM in simulation.

Since human corrections are not always optimal, use them for exploration guidance rather than as supervision

We use these insights to design Q-OIL (Q-Optimistic Intervention Learning): a simple change to any off-policy RL algorithm: each takeover earns an optimism bonus that guides exploration.

HIL RL with interventions
  1. for each step:
  2. act: the person if they take over, else the robot
  3. store (s, a, r, s')
  4. update Q toward r + γ Q(s', a')
  5. update the policy to maximize Q

Interventions are treated like any other data.

Q-OIL
  1. for each step:
  2. act: the person if they take over, else the robot
  3. store (s, a, r, s'), plus a bonus b if it was a takeoverFinding 2
  4. update Qtask toward r + γ Qtask(s', a')
  5. update Qopt toward r + b + γ Qtask(s', a')Finding 1
  6. update the policy to maximize Qopt, plus BC on takeoversFinding 3

Interventions guide exploration, and the bonus stays on the intervention.

addedchanged

Our algorithm is based on data-driven insights

How does Q-OIL work?

Intuitively, Q-OIL treats each intervention as a signal for where to explore rather than a label to imitate. We add an optimism bonus to intervention transitions, which increases the value of the states and actions that humans guide the robot toward. However, naively adding this bonus to a single critic causes it to propagate backward through bootstrapping, assigning high values to the very states where the robot was failing. To prevent this, we decouple the critic into a task critic, trained only on the task reward, and an optimistic critic that receives the bonus but bootstraps from the task critic. As the robot improves and interventions become less frequent, the bonus is sampled less often, and the optimistic critic converges to the task critic, leaving no asymptotic bias.

Why two critics?

One critic Q-OIL: two critics Bonus leaks back: states before takeovers look best Value rises toward the goal low value high value green dot: start · red dot: goal
Q-OIL training: the task critic Q_TD learns from the task reward and bootstraps from itself; the optimistic critic Q_opt adds the bonus but bootstraps from Q_TD, and the policy improves on Q_opt.
Only Qopt sees the bonus. Its targets come from QTD, which never sees it.

How does it compare to other methods?

Q-OILThe bonus stays on the corrected path, and the policy solves the maze.
No decouplingWith one critic, the bonus leaks to even the bad states before the intervention, and the policy collapses.
HILCorrections are just more data. The policy does not find the way in time.

Results

Simulated and real human interventions

  • Q-OIL robustly outperforms all methods with both simulated and real human interventions.
  • Our updated simulated intervention model (PAM) keeps the ranking of all algorithms that real people produce, so simulation gives more reliable insights.
Simulated interventions (PAM)
Lift task: pick up a cube.
Success rateSuccess rate on Lift with simulated interventions.Environment steps
Can task: move a can into a bin.
Success rateSuccess rate on Can with simulated interventions.Environment steps
Square task: fit a square nut onto a peg.
Success rateSuccess rate on Square with simulated interventions.Environment steps
Real human interventions
Success rate (%)Success rate on Lift with real human interventions.Environment steps
Success rate (%)Success rate on Can with real human interventions.Environment steps
Legend: Q-OIL, Q-OIL without BC, HG-DAgger, HIL, BC+RL, SIRIUS, PVP, RLIF.

Real robots: Franka

  • Q-OIL reaches 10–50% higher success on real Franka tasks, with about 30% less human effort.
  • Q-OIL is the most efficient in terms of interventions across tasks, i.e., it makes maximal use of each human correction to adapt its behavior.
Pen in bowl
Success rateSuccess rate on pen in bowl.Environment steps
Intervention steps (×10²)Human effort (intervention steps) on pen in bowl.Environment steps
Peg insertion
Success rateSuccess rate on peg insertion.Environment steps
Intervention steps (×10²)Human effort (intervention steps) on peg insertion.Environment steps
Unbutton shirt
Success rateSuccess rate on unbuttoning a shirt.Environment steps
Intervention steps (×10²)Human effort (intervention steps) on unbuttoning a shirt.Environment steps
Legend: Q-OIL, HG-DAgger, HIL, BC+RL.

Pretrained VLAs, bimanual robot

  • Q-OIL drops straight into a VLA post-training pipeline. We fine-tune a pretrained VLA on 25 demonstrations, freeze it, and train a small residual policy online with Q-OIL from the operator's interventions.
Bag assembly evaluation: BC+RL (left) vs. Q-OIL (right).
Base VLA BC+RL Q-OIL

Ablation: why two critics?

Success averaged over training versus optimism bonus size. Q-OIL stays near 0.7 from 0.01 to 0.5; without decoupling, success falls from 0.3 to near 0.
Success vs. bonus size, averaged over the Robomimic tasks.

Q-OIL is stable across a 50× range of bonus values. With one critic, the bonus leaks backward (see Why two critics?), and performance collapses as the bonus grows.

BibTeX

@inproceedings{qoil2026,
  title     = {Q-OIL: Optimistic Exploration in Learning from Human Interventions},
  author    = {Teegavarapu, Pranav and Poddar, Sriyash and Qi, Rico and
               Winston, Cleah and Khandelwal, Krisha and Onol, Aykut and
               Jaques, Natasha and Gupta, Abhishek},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026},
}