Abstract: 

Building data science and machine learning into a product does not mean automating all human work away. Instead of aiming for a flawless artificial intelligence, we can accept that the technology is only somewhat intelligent and make it work together with human domain experts to fill in the gaps in the system’s capabilities and train it to improve.
This combination allows to to build full stack vertical applications in a particular domain.
We use as an example a system that learns from expert users how to answer frequently asked questions sent to customer support automatically. Dealing with natural language and question answering provides a good illustration of challenges of defining which parts should be done by machines and which are best left to humans.
We describe the algorithms and the the data pipelines, but go deeper into the considerations beyond algorithms that are required to make the product successful:
* How to choose evaluation metrics that capture the business objectives
* How to make the users feel good about interacting with the system through managing perceptions and expectations
* How to anticipate the AI to make mistakes and design the system to minimize the impact of these errors on user experience
* How to plan the handoff between the machine and the human workers
* How to build an integrated workflow where work is divided between the machine and the human workers
* How to align the interests of the human workers with the machine
* How to design the machine/human interface to maximize clear signals for machine learning
* How the probabilistic part of the system works with the deterministic one

This session should be of interest to engineers, product managers and designers who want to know more about building machine learning into their products."

Bio: 

Eugene leads data science at Directly, a startup in San Francisco that helps companies scale excellent customer service by letting expert users help other users on demand. He builds augmented intelligence systems that allow humans and machines work together to make customer support better. Previously Eugene was creating data stories and data products in the fitness wearables space at Jawbone. Prior to Jawbone, he was CTO/Co-founder of survey startup Qualaroo.