Supervised Learning of Policies (Behavior Cloning)

Imitation Learning

In practice, after a long time of training it actually worked.

“tightrope walker scenerio”: has to follow exact same behavior otherwise the model does not know what to do

Three angles supervised to steer differently! This trick helped a lot

More like in training data we have scenerios to train the machine how to correct its little mistakes.

Instead of being clever about $p_{\pi_\theta}(o_t)$, let’s be clever about $p_{data}(o_t)$ ⇒ Do DAgger(Data Aggregation)

DAgger (Data Aggregation)

Idea: To collect data from $p_{\pi_\theta}(o_t)$ instead of $p_{data}(o_t)$

How: Just run $\pi_\theta(o_t)$ but we need labels $a_t$

1. train $\pi_\theta(a_t|o_t)$ on human dataset $D = \{o_1,a_1,\dots,o_N,a_N\}$

2. run $\pi_\theta(a_t|o_t)$ to obtain $D_\pi = \{o_1,\dots,o_M\}$

3. Ask humans to label $D_\pi$ with actions $a_t$

4. Aggregate $D \leftarrow D \cup D_\pi$

But:

1. Lots of the problem is in step 3
   1. We as humans have to watch for feedback and then give optimal actions, cannot just watch a state and give output

2. What if the model does NOT drift?

3. Need to mimic expert behavior very accurately

Why might we fail the expert?

1. Non-Markovian Behavior
   1. Unnatural for humans to do perfect Markovian actions: Humans more like $\pi_\theta (a_t|o_1, \dots, o_t)$ ⇒ based on previous observations
   2. Solve this by adding RNN / LSTM cells to the network (Fig. 1)
      1. May still work poorly (Figure 2)

2. Multimodal Behavior (Multiple Modes/Peaks in real distribution)
   1. If discrete actions, then this is not an issue
      1. But continuous requires exponential discrete bins (for softmax)
   2. Solved by
      1. Output mixture of Gaussians
         1. $\pi(a|o)=\sum_i w_i N(\mu_i,\Sigma_i)$
         2. Tradeoffs:
            1. need more output parameters
            2. The ability to model with this in high dimensions is challenging (in theory the number of mixture elements rises exponentially with dimension increase)
      2. Latent variable models
         1. The output is still Gaussian
         2. In addition to inputting an image into NN, we also input a latent variable(may be drawn from a prior distribution) into the model.
            1. Conditional Variational Autoencoders (VAEs)
            2. Normalizing Flow/realNVP
            3. Stein Variational Gradient Descent
         3. Complex to train
      3. Audoregressive discretization
         1. discretizes one dimension at a time using an NN trick ⇒ never incur the exponential cost
         2. Adds a softmax for one dimension, does a discrete sampling (and obtains a dim 1 value), feed the dim 1 value into another NN and softmax layer, do sampling, obtain dim 2 value, so on.
         3. Have to pick bins

Questions:

1. Does including history info (LSTM/RNN) mitigate causal confusion?
   1. My guess is no since history has no correlation with confusion

2. Can DAgger mitigate causal confusion?
   1. My guess is yes since the model will confuse and then this the part of data that the model is confused on will then be manually labeled.

Whats wrong with imitation learning:

1. Humans need to provide data, which is limited
   1. DL works best when data is plentiful

2. Humans are not good at providing some actions

3. Humans can learn autonomously, can we do the same on machines?
   1. Unlimited data from own experience
   2. continuous self-improvement

Analysis of Imitation Learning

Ross et al. “Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning”

We assume:

then…

with DAgger

Without DAgger

We cannot assume anything for $p_{mistake}$

But we can bound the total variation divergence of training and testing

Since $(1-\epsilon)^t \ge 1-\epsilon t$ for $\epsilon \in [0,1]$,

So finally

Another way to imitate (Goal Conditional Behavioral Cloning)

After clarification from class:

1. Sometimes we have bad “demonstrations” that may lead to different end states

2. Those demonstrations may be close to each other enough that we can train some “shared policy” for each of those different end states (in a sense that most of previous states can be obtained by a shared weight)

3. During production / test, specify the end state you want to achieve and then we can predict $\pi_\theta(a|s,g)$

“Learning to Reach Goals via Iterated Supervised Learning” - Dibya, Abhishek