Foursquare

Using machine learning-based model training, we produced a model that could predict anomalies regarding the performance of Foursquare’s data infrastructure and systems. Manual curation is often tedious and the model we developed was intended to make anomaly detection simple and automated. In this process, Foursquare supplied our engineering team with raw training data and metrics and we ‘trained’ the machine learning model to identify data anomalies that could be indicative of potential risks.

“The work Digamma did for us shows a lot of promise. To date, we have been able to produce a model verified to predict anomalies in server and system performance data for the purposes of detecting performance issues worth paying attention to. This is a significant achievement. Communicating with the team at Digamma was easy, their methodology was on-point and deliverables were clearly specified and met on time. The team at Digamma is knowledgeable and experienced in applying machine learning technologies and I would definitely recommend them to businesses looking to incorporate ML technologies into their systems.”

– Robert Joseph, VP Infrastructure Engineering, Foursquare

The machine learning components of the project included:

• Real-time anomaly/event detection in the behavior of a large server cluster
• Multivariate time series analysis, prediction and modelling
• Time series data mining and knowledge extraction

Machine learning techniques used:

• Prediction: seasonal ARIMA (SARIMA), modified Kalman filters, fuzzy neural networks, and LSTM networks
• Modelling: singular spectrum analysis (SSA), Fourier/wavelet analysis, digital filtering, STL decomposition, linear system identification, and Markov models
• Knowledge Extraction: rules extraction and sequence mining (WINEPI algorithm)