Real-time data engineering and the unified modern data platform

Please log in to watch this conference skillscast.

Historically, data engineering has been relegated to batch processes running in the data warehouse or data lake - if the data didn't exist in the warehouse, it effectively didn't exist. In fact, operational systems - OLTP databases, messaging and streaming systems, caches, microservices, and so on - are not just primary sources of data, but destinations for processed data as well. In-app analytics, recommendation systems, online ML/AI infrastructure, automation, and a host of domain-specific services are just as much a part of a company's data platform as the data warehouse and BI tools. The common thread that ties these systems together is streaming and real-time data processing that handles both event-driven services and real-time data integration. While streaming systems have historically been the land of low level code and complex distributed systems, modern real-time data platforms and their associated patterns are making it just as easy to build data pipelines in SQL, on the stream, to power both operational and analytical applications. In this session we'll discuss the unified operational/analytical data platform architecture, the most common patterns, and how data engineering works in real-time.