toogle sidebar button

Data pipeline

Analogy to water pipelines

Fetching data from lakes, rivers and ponds could take long distances and time. It was manual process but in time the demand was bigger and the water supply has been automated with the new technologies. Basics Data pipeline is a mechanism to transfer data from point A to point B through some intermediate points C,D and E where data processing takes place. Data pipeline receives data from the Data Producers and the result of the processing is used by the Data Consumers.

Responsibilities

Ingestion Data Governance Master Data Management Lineage

Segmentation

Bronze / Silver / Gold Data format Security

Usage

Data Pipelines are used in the following fields: Business Analytics Reporting Data Science Machine Learning

Types of the Data Pipelines

ETL, ELT, CDA Batch, Realtime

Architectures

Kappa, Delta, Lambda Storage Raw

Silver Gold

Tools

SAP BODS Kafka