This project compares Proximal Policy Optimisation (PPO), an on-policy actor–critic algorithm, with Soft Actor-Critic (SAC), an off-policy method, on continuous-control problems. The central question is how their different uses of experience affect learning speed, stability, final performance, and practical training cost.
We evaluated both algorithms in two MuJoCo environments: HalfCheetah-v4, a locomotion task requiring coordinated joint torques, and Pusher-v4, a manipulation task in which a robotic arm must move an object toward a target. Together, these environments test smooth locomotion and precise interaction with an object.
Our evaluation used hyperparameter sweeps followed by five-seed final runs. PPO was trained for one million environment steps, while SAC was trained for 400,000 steps to keep runtime manageable. We therefore interpret sample efficiency and practical runtime separately rather than treating the experiment as a perfectly matched compute benchmark.
PPO trains from trajectories generated by the current policy. Its clipped objective limits the size of each policy update using the importance-sampling ratio
This constraint reduces destructive updates, but the algorithm cannot extensively reuse older experience. Our implementation used separate two-layer actor and critic networks with 64 hidden units and Generalized Advantage Estimation.
SAC stores transitions in a replay buffer and learns from them repeatedly. Its critic target includes an entropy term:
Entropy regularization encourages continued exploration. We used twin Q-networks, Polyak-averaged target networks, two 256-unit hidden layers, and automatic entropy tuning.
For PPO, we swept the learning rate and clipping coefficient. A learning rate of $3\times10^{-4}$ worked best in both environments, with clip coefficients of $0.2$ for HalfCheetah and $0.3$ for Pusher. PPO was more sensitive to learning rate than clipping and was comparatively robust across the sweep.
For SAC, we varied the critic learning rate and target-network update rate $\tau$. The best setting depended on the environment: $q\_lr=10^{-3}$ and $\tau=0.01$ for HalfCheetah, but $q\_lr=3\times10^{-3}$ and $\tau=10^{-3}$ for Pusher. Strong SAC performance occupied a smaller, more environment-dependent part of the search space.
Figure 1. Grid-search results for PPO and SAC on both environments. Each cell reports mean return over the last 50 episodes for one hyperparameter setting; for Pusher-v4, values closer to zero are better.
SAC outperformed PPO in sample efficiency on both tasks. In HalfCheetah, SAC approached a return of 10,000 by 400,000 steps, whereas PPO remained near 2,000 after one million steps. In Pusher, SAC rapidly improved from roughly $-150$ to $-40$ and converged near $-20$ to $-25$; PPO improved more slowly and finished near $-60$.
The five-seed results also favored SAC. Its final policies achieved higher return in both environments, and its Pusher performance was more tightly clustered across seeds. PPO showed more variability, particularly in HalfCheetah, where several policies failed to discover a stable forward-running gait.
Figure 2. Mean episodic return versus environment steps for PPO and SAC on HalfCheetah-v4 and Pusher-v4 across five seeds. Shaded regions indicate $\pm1$ standard deviation.
Figure 3. Distribution of final performance across five seeds. Each boxplot summarizes mean return over the last 50 episodes, and individual points denote seeds.
SAC’s ability to replay experience improved sample efficiency, but each training step required more computation. On our setup, the PPO sweeps took approximately five hours, while the SAC sweeps took approximately 9.5 hours. PPO was therefore cheaper in wall-clock time even though it needed more environment interaction and produced weaker final policies.
Rendered rollouts reinforced the quantitative results. SAC learned smooth locomotion and controlled pushing across seeds. PPO was less reliable: some HalfCheetah policies flipped or moved with unstable gaits, and most Pusher seeds failed to move the object to the target through deliberate, controlled contact.
For data-intensive continuous-control problems, our results favor SAC when interaction efficiency and exploration are priorities. The report applies this lesson to a proposed multi-robot formation-control task with local observations, dynamic obstacles, and continuous velocity commands. A centralized-training, decentralized-execution version of SAC could learn when to adapt or deform a formation while each robot acts from local information.
Reinforcement learning alone is not enough for safety-critical deployment. A practical system would use RL for high-level adaptation and coordination while retaining structured low-level tracking and safety enforcement, such as Control Barrier Functions or Model Predictive Control. The broader conclusion is that learned policies and classical control serve complementary roles in real robotic systems.