Neural Program Synthesis from Diverse Demonstration Videos

Abstract

Interpreting decision making logic in demonstration videos is key to collaborating with and mimicking humans. To empower machines with this ability, we propose a neural program synthesizer that is able to explicitly synthesize underlying programs from behaviorally diverse and visually complicated demonstration videos. We introduce a summarizer module as part of our model to improve the network’s ability to integrate multiple demonstrations varying in behavior. We also employ a multi-task objective to encourage the model to learn meaningful intermediate representations for end-to-end training. We show that our model is able to reliably synthesize underlying programs as well as capture diverse behaviors exhibited in demonstrations.

Model Architecture

A brief overview of our model.

• Demonstration Encoder receives a demonstration video as input and produces an embedding that cap- tures an agent’s actions and perception.
• Summarizer Module discovers and summarizes where actions diverge between demonstrations and upon which branching conditions subsequent actions are taken.
• Program Decoder represents the summarized under- standing of demonstrations as a code sequence.

Paper

@inproceedings{sun2018neural,
  title = {Neural Program Synthesis from Diverse Demonstration Videos},
  author = {Sun, Shao-Hua and Noh, Hyeonwoo and Somasundaram, Sriram and Lim, Joseph},
  booktitle = {Proceedings of the 35th International Conference on Machine Learning},
  year = {2018},
}

Related Work

Check out some other recent work in program synthesis:

• R. Bunel et al., Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis, in ICLR 2018
• J. Devlin et al., RobustFill: Neural Program Learning under Noisy I/O, in ICML 2017
• M. Balog et al., DeepCoder: Learning to Write Programs, in ICLR 2017
• E. Parisotto et al., Neuro-Symbolic Program Synthesis, in ICLR 2017