Central Log Monitoring Solution

Project Background

Česká spořitelna, the Czech arm of the Erste Group, was looking for a custom solution for the central monitoring of log data from their operational banking systems. The central log collects and stores a massive amount of data – over a specific period of time – to analyse trends or record events from various banking systems plus network and IT environments.

Improvements to their data collection and storing processes had to be made to comply with the new cyber security regulations. But the bank also wanted their in-house security specialists to be able to analyse events as soon as possible after they are logged.

Challenge

The bank’s central monitoring system was based on batch processing of system logs using a standard relational database. It wasn’t compliant with new legislation as it didn’t collect complete data from every system. Plus, the existing system couldn’t meet the new requirements to retain data for a minimum period of time.

Aside from the compliance issues, this system’s operational mode didn’t allow the use of analytical queries that would include log data from the current date of generation – meaning rapid analysis was impossible.

Business Needs

The solution needed to meet the following specifications:

• Fulfil requirements of new cyber security legislation
• Fast and stable data processing 
• Able to analyse data including freshly logged events 
• Scalable solution for future data volume increases

Solution & Results

We developed a system that can process streams of logging data from all monitored banking systems in real time. Working with the bank’s security, IT, and operations departments, we tailored the solution to meet each of their specific needs.

Logging data is streamed from newly created central storage through Apache Kafka into a new, dedicated system. There, the data is being processed using Apache Spark and Spark Streaming for monitoring purposes. The whole solution can handle tens of thousands of log entries per second.

Thanks to the implementation of Apache Spark, using lambda architecture, we ensured a uniform handling of stream and batch processing of data. The same code is shared for both processing methods. There is no need to maintain two separate codebases, which makes development, deployment, and servicing much simpler and faster. The processed data is stored in Hive tables within a Hadoop cluster. 

Rapid analysis of security data

The unified data overview provides access for specialists in the security department for their analyses. The data is shown in the same format as previously, so no analytical process changes had to be implemented. However, it is now possible to access data within minutes of it being generated.