SBB Cargo – Data Lake

The silo problem 

In modern businesses, data is ubiquitous and comes from a wide array of different sources like:

– Company website(s)

– Company app(s)

– IoT devices

– CRMs

– Social Media

– Inventories & logistics tools (like SAP)

– Data warehouses

– Internal reports

– Sales channels

– You name it

The main problem of such a scenario is that each of those data are saved into so called «vertical siloes» and those siloes aren’t designed to allow cross-siloes data sharing.

This root problem, generates many other issues:

– trying to analyze those data with aggregation tools and transversally across siloes requires a lot of error-prone and tedious handwork, involving data exporting, data extraction, spreadsheet «mambo-jambos»: in few years, companies end up with a set of handcrafted tools.

– Such handcrafted tools are quite fragile and heavily dependent on who implemented it originally, making them a nightmare to be maintained.

– When it comes to data, it’s not clear what the «single source of truth is».

– It’s even more unclear where data reside, especially in mid-large organizations.

– Raw (unmodified) data are often missing.

The Data Lake

A way to solve the vertical siloes problem is to build a data lake. It’s a place where data, coming from different sources, can be stored and accessed in a consistent way. Data are stored in theirs natural/raw format, making possible future data manipulation. Sometime transformed data are stored in data lake also; they are necessary for tasks such as reporting, visualization, advanced analytics, and machine learning.

What is a data pipeline?

To fill a data lake, workers must be deployed to collect data from sources.  Each worker is responsible for a specific data source and is connected to services that enable further data analysis in near real time.

This combination of functions is commonly referred as “data pipelines”. Similar to water pumps and pipes, data is extracted from sources auf flows through services until it ends up in data lakes or specific storage solutions.

Building a data lake and hydrating data pipelines can be achieved using:

a) On-prem services
b) Cloud based self-managed services
c) Cloud based fully managed services
d) Cloud based serverless infrastructure

The level of commitment required to operate the previous architectures significally decreases from a) to d).

On a) all the ops work is on IT team, including hardware management.
On d) the only things that the IT team must focus on are:

– application logic development (Worker logic);

– using the right tools for the job and use their API, following proper security best practices.

Since data lakes involve a lot of moving part and different technologies, going fully serverless is a choice worth to be considered to reduce operational complexity.

The services used in this implementation are:

– AWS Lambda as computing infrastructure to run data sourcing worker and transformation worker

– AWS Kinesis stream / Firehose as data stream service provider

– Amazon S3 as data lake storage system

– AWS KMS as encryption service

– AWS System Manager Parameter Store as repository for secrets

– AWS IAM for policy / roles definition

Let’s go briefly on each component to analyze the logic behind the architectural choices.

Data Sourcing Worker => AWS Lambda:

This worker needs to connect to the silo/ service and to extract data from it. According to the type of data and its usage, this worker could run few times per day or continuously. In a serverless environment on AWS, the choices are:

Lambda (scheduled execution)

A container deployed on ECS using Fargate as computing infrastructure (continuous execution)

We’ve opted for Lambda because customer desired to have a data pipeline architecture able to deal with different data frequency; for nearly continuous data flows, a short invocation rate coupled with small datasets per execution provides a good solution to use a simple Lambda function instead of a more complicated containerized worker, keeping also architectural consistency with discrete data flows.

Transformation Worker => AWS Lambda:

This worker transforms data according to specific criteria, which depend on the actual data source. Lambda is the recommended solution to be used together with AWS Kinesis Firehose. An input event is generated from Kinesis with raw sourced data, Lambda process and transform data according to the logic implemented by code and the transformed data is sent back to Kinesis Firehose for that can be persisted.

Stream Service => Kinesis Stream / Firehose:

Streams are particularly useful in data ingestion because they buffer input data for a configurable period and enable a pluggable architecture to consume, analyze and process data, in batch and/or near real-time. Kinesis Firehose is a serverless managed data stream service, whose purpose is to dump data continuously to a specific target type. In this specific case, the target is a S3 bucket. Like Kinesis Stream, Firehose can generate events attaching data flowing in it. Those events can be used to execute code through Lambda or to trigger complex tasks using AWS Step-Functions or other AWS services.

Data lake => Amazon S3:

Infinite storage capacity, configurable lifecycle policies and custom metadata are perfect to build a data lake. Amazon S3 offers that, together with a powerful API and a seamless integration with other AWS services, like Firehose. The data lake may consist in a single bucket or in multiple ones (raw data, transformed data, aggregated data…)

Security 1 => KMS:

Data in transit and at rest are encrypted using KMS with scope limited CMKs (Customer Master Keys)

Security 2 => Lambda specific role & policies:

Lambda functions run using scoped-down permissions, expressed through IAM policy documents attached to a specific role which is assigned to the lambda.

Security 3 => Secrets:

To store secrets and service wide parameters, Parameter Store is used. Secrets are saved as secure-strings and encrypted with KMS using a dedicated CMK

AWS is all about choices, so there are services to build each of the architectures mentioned before. Customers can go from fully on-prem (renting bare metal machines) to fully serverless event-driven infrastructure. And even in a serverless scenario there are several options to choose from.