Abstract: 

Deep learning has been tremendously successful, especially for solving problems in natural language understanding and computer vision. This is a great achievement, but the field still has a long way to go toward achieving the versatility of humans. After training on large amounts of data, conventional deep neural networks are frozen in time. Unlike humans, they cannot easily continue to learn more information. If this is attempted, conventional models will suffer from catastrophic forgetting, resulting in the loss of previously acquired skills. However, continual learning systems overcome this limitation. In this tutorial, I discuss the applications for continual learning, including privacy protection, user customization, GPU-free learning on embedded devices, and more. I then review state-of-the-art methods for continual learning in neural networks with application to computer vision and natural language processing tasks. The topic is of interest to researchers and practitioners interested in the topic.

Session Outline
This session assumes familiarity with linear algebra and supervised machine learning, with a basic understanding of backpropagation for neural networks. This hands-off tutorial will focus on applications, problem setup, and algorithms. There will not be any coding, but there will be pseudocode.

Lesson 1: Why Continual Learning is Needed
You will understand what continual learning is, the problem of catastrophic forgetting, how to set up a continual learning system, and the applications that continual learning can solve that the traditional train once then test paradigm cannot.

Lesson 2: Paradigms for Mitigating Catastrophic Forgetting
You will learn about variations of the continual learning problem, including continual batch learning and online streaming learning. We will discuss regularization, replay, and other methods that attempt to tackle the problem. We will focus on classification, but also briefly discuss continual learning in reinforcement learning and for generative adversarial networks.

Lesson 3: Algorithms for Online Streaming Learning
You will study algorithms for online streaming learning, how to implement them, and their pros and cons in terms of classification performance, memory, and compute. This includes learning about the streaming deep linear discriminant analysis algorithm, which can learn 900 categories with 900,000 examples in about 30 minutes compared to a week with alternative algorithms.

Background Knowledge
This session assumes familiarity with linear algebra and supervised machine learning, with a basic understanding of backpropagation for neural networks. This hands-off tutorial will focus on applications, problem setup, and algorithms. There will not be any coding, but there will be pseudocode.

Bio: 

Christopher Kanan is an Assistant Professor at the Rochester Institute of Technology (RIT), a Visiting Assistant Professor at Cornell Tech, and a Senior AI Scientist at Paige. At RIT, his lab works on lifelong machine learning and language driven computer vision, which has been supported by awards from NSF, AFOSR, ONR, DARPA, Adobe Research, and other industrial partners. He is also Associate Director of RIT's Center for Human-aware AI and he is a member of RIT's McNair Scholars advisory board. At Paige, a startup that has raised $95M to improve the diagnosis of cancer, he led the AI R&D team during its first 1.5 years and continues to advise its AI teams. He received a PhD in computer science from the University of California at San Diego, where he worked on brain-inspired algorithms for object recognition, neural networks, active vision, and cognitive modeling. He received an MS in computer science from the University of Southern California. Before joining RIT, he was a postdoctoral scholar at the California Institute of Technology, and later worked as a Research Technologist at NASA’s Jet Propulsion Laboratory, where he used deep learning to develop vision systems for autonomous ships. He is the recipient of the 2016 Rising Star Award and the 2019 Distinguished Scholarship Award in RIT's College of Science. He is an IEEE Senior Member and has published over 50 refereed papers, many of which are in top venues across AI such as CVPR, ICCV, NeurIPS, AAAI, ICLR, ACL, etc.