MEDIA
HELP
OTHER PRODUCTS
Training an intelligent classifier for your business data
Author
Johannes Giere
AI Expert @ fme AG
April 25, 2019
Artificial intelligence and similar keywords have been in the news for several years now. There is no doubt that intelligent algorithms can bring a huge advantage when improving workflows or digitalizing existing processes. There is a huge demand from businesses to apply artificial intelligence in favour to be more agile and to replace error-prone tasks. Unfortunately, there are very few products on the market, which give the businesses an easy way to create intelligent classifiers and at the same time grant flexibility of configurations.
Growth of classified content within an ECM system
Do not generalize too much
An intelligent classifier is used to predict categorical attributes automatically. It is a combination of data transformation functions and a classification algorithm. A data transformation function can either be a very basic or more complex function. For example, mapping every upper case letter to its lower case equivalent is a very basic function, but gives a very important opportunity to create a unified text corpus. On the other hand, sophisticated transformation algorithms as TF-IDF or doc2vec are available as well. Although these methods are algorithms, we still call them functions in this context.
On top of that, every classification algorithm has its own unique parameters with different effects.
As you can see, there are many functions and algorithms to be considered when employing artificial intelligence. It is understandable that nobody wants to deal with such a large number of parameters and therefore prefers an easy, pre-trained classifier. The disadvantage of this, however, is that when a classification scenario is generalized too much, the classifier accuracy gets worse. Every business data is unique and consequently every use case for artificial intelligence is unique.
It would be burned money if a classifier was not carefully prepared. This can happen quickly if a product is used that does not offer full flexibility for configuration.
The trade-off
I regularly encounter another phenomenon in projects, where the demanded classifier needs to be very fast, very accurate and should not take much time to configure. In addition, the finished classifier should be easy to understand. As you can probably see, this is a trade-off between classifier accuracy, performance and simplicity. This situation can often not be fulfilled, because they are mutually exclusive.
We consider the example of an already configured classifier that is very accurate, easy to understand, but slow. When the user configured the classifier, it was in his interest to create an easy understandable classifier and to not invest too much time. Because he experimented a little, he knew what parameters to set to make its predictions accurate. On the other side, the classifier is slow during training and prediction.
This example shows, that simple classifiers have its drawback for fast predictions.
If the user was not interested in an understandable classifier, but more in a rather quickly created classifier, we would see the same performance results. Either it takes time to adjust a classifier to the business use case or criteria needs to be cut back. If a realistic objective is made, it is possible to reduce the discrepancy between the criteria.
In the end, I hope that I could explain the difficulties in creating a classifier for a business use case. Projects with artificial intelligence are not easy to master, but good preparation can prevent most mistakes.
