Our Methodology

First, we obtain data from the client in whatever format they provide. We use automatic routines to transform this raw data into a standardized format to make processing easier. Then, we start a typical data science processing flow, starting from data cleaning (e.g. removing HTML tags, performing noise removal based on regular expressions as well as keyword replacing, character cleaning and character filtering). From there, we perform sentence and word tokenizations.

Next, we begin the exploratory process where we analyze the data, draw charts, histograms, extract task-related features and calculate various statistics. Then, we proceed to feature engineering. We have models for Named Entity Recognition, POS tagging, dependency parsing, TF-IDF and word embeddings. As a result of the process described above, the pre-processed dataset is ready to be fed into ML models. Here, the first iteration ends. From this point, we analyze the results, and, if they are not adequate, tweak the previous stages to improve overall performance. This is the next iteration.

This process is generally sequential, as one stage comes after the other. However, in practice, each stage performs its task with multiple sets of parameters and the result of every run is passed to the next stage. In this way, the process is more like a graph or a tree—not a sequence—and is somewhat similar to neural nets insofar as they relate to how computing nodes are organized. Also, this method stores all intermediate results. So, if only the second stage from the end of the pipeline is changed, then only the last two stages will be re-computed.

Once the steps in this methodology are completed and a certain desired level of accuracy is reached, the end result is a custom NLP framework based on the original input data.