Log Anomaly Detection With Artificial Intelligence

Can artificial intelligence pick suspicious log entries? My students Tuomo Kuure and Joni Hakala are training a model for that.

They use ELK to collect logs from multiple computers to an ElasticSearch NoSQL database. These log entries are then analysed with their own model using Python Tensorflow in a Docker container with GPU support.

Blog and Git

Ailogs.design.blog tells the journey from zero to AI. As this is an ongoing project, you might want to subscribe their RSS feed.

Source code in GitHub. The whole stack runs in a Docker container with GPU support, so it should be both easy to install and fast to train with your graphics card. Free software, GPL 3.0.

This project is part of Haaga-Helia course Monialaprojekti infra. I'm teaching the course with Harto Holmström.

Video Demo in Finnish

Tuomo and Joni give a simple video demo for a model guessing priority of a log event. This is just a tech demo, the priority is the regular log priority (error, panic...) already known for these log events. This is an interim milestone, so full log anomaly detection is not yet demonstrated. In Finnish.

GPU Parallel Computing with Docker

Training a model is a parallel job, and graphics card is good (fast) at parallel. With many new Python libraries with specific dependencies of their own, Docker helps keeping the development environment constant between machines and over time.

Joni and Tuomo are using a simple Dockerfile:

FROM tensorflow/tensorflow:latest-gpu-py3
RUN pip install pandas
RUN pip install scikit-learn
RUN pip install matplotlib