Generative NLP models in customer service. How to evaluate them? Challenges and lessons learned in a real use case in banking.


Daily communication via text between customer service agents and clients is rapidly increasing day by day, and banking is not an exception. In this talk we will explain how we experimented with generative NLP models to assist financial advisors in their daily interactions with clients.

For this work we have used a seq2seq deep learning neural network architecture based on two LSTM acting as encoder and decoder.

When using generative methods, one of the main challenges is evaluation. In this aspect many questions arise. What does correct mean? Could an answer be syntactically and grammatically correct but does not reply to customer needs? What about a suggestion that saves time to the manager but needs editing? How can the whole performance of the system be assessed? We will share how we tackled this problem in a specific use case in banking along with other lessons learnt that can be helpful for practitioners thinking of using these types of models in customer service. The originality and main contribution of our talk is the learnings obtained by the overall evaluation process and how these metrics correlate to human evaluation, which is an agnostic part to the NN architecture or NLG technique used.


María is Senior Data Scientist and Data Product Owner at BBVA AI Factory, with ten years of experience in the Data Science field, she was one of the first Data Scientists in BBVA, taking part in the Big Data ecosystems set up in the bank. Graduated in Mathematics and Computer Engineering, she holds a MSc in Computational Intelligence from Universidad Autónoma de Madrid (UAM), specialized in Aspect-based sentiment Analysis and Item Recommendation.

She has worked in several analytical domains, ranging from Retail and Urban Analysis to Customer Intelligence. Now, she is trying to enhance the customers' relationship with the bank through Natural Language Processing and Text Analytics. María focuses on understanding business challenges and developing the best analytical solution for each problem.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google