Model-based Methods

NPC

Performance Gap Issue

We want good exploration to gather more data to address the distribution shift, but with a not-so-good early model we cannot deploy good exploration techniques

Basically the model early on overfits an function approximator and the exploration policy is stuck on a local minima / maxima

Uncertainty Estimation

Uncertainty estimation helps us reduce performance gap.

Under this trick, the expectation of reward under high-variance prediction is very low, even though the mean is the same

However:

1. We still need exploration to get better model

2. Expected value is not the same as pessimistic value

But how to build it?

1. Use NN?
   1. Cannot express uncertainty of unseen data accurately ⇒ when querying out-of-sample data, the uncertainty output is arbitrary
   2. “The model is certain about data, but we are not certain about the model”

2. Estimate Model Uncertainty
   1. Estimate $\argmax_\theta \log \mathbb{P}(\theta|D) = \argmax_\theta \log \mathbb{P}(D|\theta)$
   2. Can we instead estimate the distribution $\mathbb{P}(\theta|D)$?
      1. We can use this to get a posterior distribution over the next state
         1. $\int p(s_{t+1}|s_t,a_t,\theta) p(\theta|D) d\theta$
      2. Bayesian Neural Nets
         1. Every weight, instead of being a number, is a distribution
         2. But it’s difficult to join parameters into a single distribution
         3. So we do approximation
            1. $p(\theta|D) = \prod_i p(\theta_i|D)$
               1. Very Crude approximation
            2. $p(\theta_i|D) = N(\mu_i, \sigma_i^2)$
               1. Instead of learning numerical value learn mean value and std/variance
            3. Bllundell et al. Weight Uncertainty in Neural Networks
            4. Gal et al. Concrete Dropout

3. Bootstrap Ensembles
   1. Train several different neural nets and bootstrap samples($D_i$ sample with replacement from $D$) to feed to those networks
      1. So each network learns slightly differently
      2. In theory each NN learns well on their own training data but makes different mistakes outside of training data
   2. Then estimate $p(\theta|D) \approx \frac{1}{N} \sum_i \delta(\theta_i)$
   3. $\int p(s_{t+1}|s_t,a_t,\theta) p(\theta|D) d\theta \approx \frac{1}{N} \sum_i p(s_{t+1}|s_t,a_t,\theta_i)$
   4. Very crude approximation
      1. Because the number of models is usually small(expensive)
      2. But works
      3. Resampling with replacement is usually unnecessary, because SGD and random initialization usually makes the models sufficiently independent

With Complex Observations (images)

1. High dimensionality with observation prediction model
   1. Redundancy

2. Partial Observability

Can we seperately learn $p(o_t | s_t)$, which is high-dim but not dynamic, and $p(s_{t+1}|s_t,a_t)$, which is low-dim but dynamic?

Now we need to maximize

Note that the expectation is conditioned on

So can we learn an additional approximate posterior (encoder):

Also we have other choices:

1. $q_\psi(s_t,s_{t+1}|o_{1:T},a_{1:T})$ - full smoothing posterior
   1. most accurate
   2. most complicated to train
   3. Preferred when the state is less fully-observed

2. $q_\psi (s_t | o_t)$
   1. single-step encoder
   2. simplest but less accurate
   3. stochastic case requires variational inference
   4. Deterministic: $q_\psi(s_t|o_t) = \delta(s_t = g_\psi(o_t))$

To use uncertainty

Before uncertainty is introduced, we have (assuming $s_{t+1} = f(s_t,a_t)$

Now(With bootstrap):

Other options:

1. Moment matching

2. more complex posterior estimation with BNNs

Further Readings

1. Classic
   1. Deisenroth et al. PILCO: A Model-Based and Data-Efficient Approach to Policy Search.

2. Recent papers:
   1. Nagabandi et al. Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning.
   2. Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models.
   3. Feinberg et al. Model-Based Value Expansion for Efficient Model-Free Reinforcement Learning.
   4. Buckman et al. Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion.

Can we just backprop through policy / dynamics model?

1. Similar parameter sensitivity problems as shooting methods
   1. But no longer have convenient second order LQR-like method, because policy parameters couple all the time steps

2. Similar problems to training long RNNs with BPTT
   1. Vanishing and exploding gradients

Let’s compare policy gradient:

with backprop (pathwise) gradient:

Policy gradient might be more stable (if enough samples are used) because it does not require multiplying many Jacobians

Parmass et al. “18: PIPP: Flexible Model-Based Policy Search Robust to the Curse of Chaos”

It we use policy gradient to backprop through our model dynamics model, there would still be a problem (although generating samples does not require interacting with physical world or simulator)

But this does not solve the problem because the point of using model-based RL is being sample efficient and using sampled real world data and do short-rollout open-loop planning restrict how much we can change our policy.

Model-based RL with Policies

1. Model-Based Acceleration (MBA)
   1. Gu et al. Continuous deep Q-learning with model-based acceleration. ‘16

2. Model-Based Value Expansion (MVE)
   1. Feinberg et al. Model-based value expansion. ’18

3. Model-Based Policy Optimization (MBPO)
   1. Janner et al. When to trust your model: model-based policy optimization. ‘19

Good Idea:

1. Sample Efficient

Bad Idea:

1. Additional Source of bias if model is not correct
   1. Reduce this by using ensemble methods

2. If not seen states before then we still cannot perform valid action

Multi-step Models and Successor Representations

Therefore

Where

is the “successor representation”

Dayan Improving Generalization for Temporal Difference Learning: The Successor Representation, 1993.

Hmm…looks like a bellman backup with “reward” $r(s_t) = 1-\gamma \delta(s_t = i)$

But issues:

1. Not clear if learning successor representation is easier than model-free RL

2. how to scale to large state spaces?

3. How to extend to continuous state spaces?

To scale to large state spaces, use successor feature of $s_t$

If we can express the reward function as combination of features of state s

Turns out we can express the value function as

The dimensionality of $\phi$ and $\psi$ would be a lot lower than the dimension of possible states

Can also construct a Q-function like version

(when $r(s) = \phi(s)^\top w$)

Using Successor Features

1. Recover a Q-function very quickly
   1. Train $\psi^\pi(s_t,a_t)$ via Bellman backups
   2. Get some reward samples $\{s_i, r_i \}$
   3. Get $w \leftarrow \argmin_w \sum_i ||\phi(s_i)^\top w - r_i||^2$
   4. Recover $Q^\pi(s_t,a_t) \approx \psi^\pi(s_t,a_t)^\top w$
      1. $\pi’(s) = \argmax_a \psi^\pi(s,a)^\top w$
         1. This is only equivalent to one step of policy iteration because Q is with respect to the actor

2. Recover many Q-function
   1. Train $\psi^{\pi_k}(s_t,a_t)$ for many policies $\pi_k$ via Bellman backups
   2. Get some reward samples $\{s_i, r_i \}$
   3. Get $w \leftarrow \argmin_w \sum_i ||\phi(s_i)^\top w - r_i||^2$
   4. Recover $Q^{\pi_k}(s_t,a_t) \approx \psi^{\pi_k}(s_t,a_t)^\top w$ for every $\pi_k$
      1. $\pi’(s) = \argmax_a \max_k \psi^{\pi_k}(s,a)^\top w$
         1. This is only equivalent to one step of policy iteration because Q is with respect to the actor
         2. Finding to highest reward policy in each state

Continuous Successor Representations (C-Learning)

Positive set during training:

Negative set just pulled randomly.

To train (On-policy algorithm, can also derive an off-policy algorithm that is also very efficient):

1. Sample $s\sim p^\pi(s)$ ⇒ run policy, sample from trajectories

2. Sample $s \sim p^\pi(s_{future}|s_t,a_t)$ ⇒ pick positive future pairs

3. Update $p^\pi(F=1|s_t,a_t,s)$ using SGD with cross entropy loss

Eysenbach, Salakhutdinov, Levine. C-Learning: Learning to Achieve Goals via Recursive Classification. 2020.