Reinforcement learning has shown strong potential for learning control policies in robotics
and continuous-control tasks, but its practical use is often limited by the cost of collecting
real-environment data. In many realistic settings, collecting target-environment rollouts is
slow, expensive, and potentially unsafe. Simulation offers a natural alternative because it
can generate large amounts of data cheaply, but simulated dynamics are usually biased
and cannot be treated as a perfect replacement for the target environment. This project
studies how to use limited trusted target data together with abundant biased simulation
data in an offline reinforcement learning setting.
The project extends a simulation-aided reinforcement learning method, based on control
variates ideas, into a pure offline RL pipeline. In this setting, the algorithm receives a fixed
target replay buffer and cannot collect additional target-environment data during training.
Since offline RL is vulnerable to distribution shift, the learned policy may choose actions
that are poorly represented in the dataset, causing the critic to assign overly optimistic Q
values to unsupported actions. To address this, Conservative Q-Learning (CQL) was
integrated into the critic objective.
In the project setup, no physical robot was used. The “real” environment refers to the
target MuJoCo environment with nominal physical parameters, while the “sim”
environment refers to a shifted MuJoCo simulator with biased dynamics. This controlled
setting makes it possible to study sim-to-real dynamics mismatch in a reproducible way.
The offline pipeline loads a fixed target replay buffer, loads the behavior policy that
generated it, rolls out the same behavior policy in the biased simulator to generate a
matched simulated replay buffer, and then trains an actor-critic agent using a control
variate critic loss with CQL regularization.
A central requirement of the method is that the real and simulated datasets must be
generated by the same behavior policy. If the datasets come from different policies, the
real/sim classifier may detect policy mismatch instead of transition-dynamics mismatch.
This makes the correction weights unstable and can damage the control-variate estimator.
Therefore, the implemented workflow explicitly generates the simulator dataset using the
same behavior policy that produced the target dataset.
The main technical contribution of the project is the implementation of offline SAVER-style
training with CQL support. The system includes offline replay-buffer loading, behavior
policy-based simulator data generation, a real/sim classifier with validation-based
stopping, separate CQL terms for target and simulator minibatches, optional CQL
Lagrangian tuning, and experiment logging. Several variants were compared, including
real-only CQL, naive sim+real mixing, and the proposed control-variate sim+real method.
Experiments were performed on MuJoCo Hopper-v5 in a low-target-data regime. The final
evaluation used a small target replay buffer and a much larger simulated replay buffer
generated under shifted simulator parameters. The results show that simulation data can
improve offline RL performance when used carefully. In the tested setting, the control
variate method achieved better mean target-environment performance than real-only CQL
and regular sim+real mixing. The experiments also showed that CQL is necessary for
offline stability, that classifier overfitting must be controlled, and that separate treatment
of real and simulated losses is important for reliable tuning.
Overall, this project demonstrates a working offline simulation-aided RL pipeline that uses
biased simulation as a correction signal rather than as ground truth. The results support
the usefulness of combining control variates with conservative offline RL in a controlled
low-data sim-to-real setup. Future work should extend the evaluation to more
environments, perform stronger ablations, improve offline model selection, and eventually
test the method on a real robotic system.
The final experiment used Hopper-v5 as the target environment, 1.5K target-environment
steps, 100K simulator steps, a simulator with gravity, friction, and angular-momentum
related parameters shifted to 85% of the original values, 100K training steps, and
evaluation over 10 runs with seeds 1-10 and 10 evaluation episodes. The improved offline
control-variate method should be interpreted as a successful simulation-based
implementation and experimental validation of the offline component in this tested low
data setup, not as a claim of fully general offline sim-to-real performance.