What Is A Data Pipeline – How Does It Work?
I write about fintech, data, and everything around it
A data pipeline is a series of data processing steps. Each step delivers an output that is the input to the next step, and this continues until the pipeline is complete.
Data pipeline consists of three key elements – source, processing steps, and destination. As organizations look to build applications using microservices architecture, they are moving data between applications, making the efficiency of the data pipeline a critical consideration in their planning and development.
Data generated in one source system or application may feed multiple data pipelines, and those pipelines may have numerous other pipelines or applications that are dependent on their outputs.
Let’s look at an example.
You write an opinion piece on LinkedIn with a bunch of trending tags. Assuming that you are a famous individual, we can look at the following engagement activities:
- Hundreds of people would like the piece
- Hundreds of people would comment on the piece – positive, negative, and neutral sentiments on your opinion
- Multiple people can be tagged as a part of the comments who would be invited to contribute their opinions on your piece
- Hundreds of people would share your piece with additional tags on it
- Hundreds of people would refer to your article and add their views on top of it
While the source of the data is the same, the different metrics feed into different data pipelines. Your opinion piece is visible under your profile, under profiles of people who engaged with your content, and the innumerable tags used to define the content.
Common steps in data pipelines include data transformation, augmentation, enrichment, filtering, segmenting, aggregating, and algorithms running against the data that provide insights to the business.
Let us look at another big data example.
Netflix is a master when it comes to giving you personal recommendations. This is one reason we keep going back to Netflix for all our entertainment content needs.
Netflix is a data-driven company, and all its decisions are based on insights derived from data analysis. The charter of the data pipeline is to collect, aggregate, process, and move data at a cloud-scale. Here are some statistics about Netflix’s data pipeline:
- 500 billion events, 1.3 PB per day
- 8 million events and 24 GB per second during peak hours
- Several hundred event streams are flowing through the data pipeline – Video viewing activities, UI activities, error logs, performance events, troubleshooting, and diagnostic events.
Netflix does real-time analytics (sub-minute latency) with the data they capture and follows stream processing. The volumes that we are talking about here are massive, and the growth has been explosive.
We are talking about 150 clusters of elastic search adoption, totaling 3500 instances hosting 1.3 PB of data.
How does the data pipeline work?
To know how a data pipeline works, think of a pipe where something is ingested at the source, and carried to the destination. How the data is processed in the pipe depends on the business use case and the destination itself.
Data Source: Relational database or data from applications. This can be done using a push mechanism, an API call, a webhook, or an engine that pulls data at regular intervals or in real-time.
Data Destination: Destination can be an on-premises or cloud-based data warehouse, or it may be analytics or a BI application.
Data Transformation: Transformation refers to operations that change data – standardization, sorting, deduplication, validation, and verification. The idea is to make it possible to analyze and make sense of the data.
Data Processing: Processing has three models.
Model #1: Batch processing, in which source data is collected periodically and sent to the destination systems.
Model #2: Stream processing, in which data is sourced, manipulated and loaded as soon as it is created
Model #3: Lambda architecture, which combines both batch and stream processing into one architecture. This is popular in big data environments, and it encourages storing data in raw format to run new data pipelines continually.
Data Workflow: Workflow involves sequencing and dependency management, and the dependencies can be technical or business-oriented. Technical dependencies would mean validation and verification before moving it to the destination. Business dependency involves cross-verification of data from different sources for maintaining accuracy.
Data Monitoring: Monitoring is used to ensure data integrity. Potential failure scenarios include network congestion, offline source or destination, and it needs to have alerting mechanisms to inform the administrators.
ZIO, the data pipeline platform
ZIO can handle all data sources and can do data processing based on the technical and business dependencies and dump it in the destination. This would allow businesses to generate actionable insights.
So, whether you are an SME or enterprise company, data tracking is the key to the success of your business. Schedule a 30-minute call and learn about Zuci’s Data Engineering Services to craft a single source of truth system for real-time data analytics, business reporting, optimization, and analysis.
Related Posts
