Before someone could deal with a query, triage teams first classified the requests so that they can be appropriately routed for further processing. On their way to fully automating the triage process we were asked to develop a solution for solving the first piece of the puzzle, that is, classifying the incoming queries. Given the richness of natural language and the myriad ways customers can express their intent, neither simple rules-based solutions nor off-the-shelf cloud solutions were able to address the problem. A custom solution was required that could be deployed on-premise without ongoing monthly fees and vendor lock-in.

As this was a greenfield project we were given the freedom to prototype the solution with any open source technology. The solution comprised of a Dockerised API with two endpoints. One for predicting the labels for new queries and one for retraining the model with new data. We have built the solution with the Python/Keras stack and shipped to the client in a Docker image. This enabled us to quickly develop different tentative neural network models for the problem on our own GPU servers without worrying about compatibility issues with the client's infrastructure. Using GPU servers let us iterate quickly but it was not required for the client to have GPU servers already. The final models run on CPU servers as this was one of the first deep learning use cases for the client. Retraining took a couple of hours which was acceptable in this context as retraining had to happen only infrequently (not more frequent than monthly). The prediction pipeline, on the other hand, runs in a few milliseconds.

The deployed solution now allows the client to automatically route emails in seconds be they one-liners or one page long texts.

After finding an optimal architecture we worked closely with the business on identifying ways they can improve their training data. The dataset provided was highly imbalanced with hundreds of categories where some categories had thousands of examples and some only a few dozen. We could show which categories are washed together by the model and thus which would benefit most from better labeling. One of the issues we discovered was the discrepancy between human operators in how they label cases. As it turned out there was a high turnover in operators and the new ones were often less accurate than their more senior colleagues. This gave a further challenge in training and evaluating the model. For in many cases the model gave better labels than the ones provided. This necessitated a careful manual evaluation of the mismatches and refinement of the training data.

The deployed solution now allows the client to automatically route emails in seconds be they one-liners or one page long texts. The initial model was relatively large so at the final stage, we pruned the model size with gradually decreasing the number of parameters to find the optimal size with a smaller memory footprint, faster processing, but the same performance.

Telecommunications companies often outsource customer support for cost reduction purposes. While complete automation is rarely feasible (as models typically reach 95%(+-4%) accuracy) leaving a few operators for the problematic cases and algorithms operating 24/7 for the majority of messages can provide good value given a sufficiently large volume of messages.

If you are interested in working with us, please complete this form, or send an email to hello@neuralmachines.co.uk.