Programming Study日本語

Disclosure: We earn a commission from Udemy when you buy a course through links on this page.

Real-Time Data Engineering: Python, dbt & DuckDB

Real-Time Data Engineering: Python, dbt & DuckDB

Udemy · English

About this course

Language
English
Category
Provider
Udemy
Data updated

Build a complete real-time data engineering pipeline using Python, DuckDB, dbt, Streamlit, and live API data. Data engineering is about much more than writing SQL or moving data from one place to another. Modern data engineers need to understand how data is collected, stored, transformed, tested, automated, and delivered to users. In this hands-on course, you will build a practical end-to-end data engineering project from scratch. We will use a live cryptocurrency market API as our data source, allowing you to work with continuously changing real-world data rather than static CSV files. You will start by setting up your Python development environment and installing the tools required for the project. Next, you will build a Python ingestion process that connects to a REST API, retrieves live market data, processes the JSON response with Pandas, and incrementally loads each new snapshot into DuckDB.You will then implement a modern Bronze, Silver, and Gold architecture. The Bronze Layer will preserve historical API data. Using dbt, you will transform this raw data into a clean and standardized Silver Layer before creating an analytics-ready Gold Layer containing the latest market information. You will also learn how to define dbt sources, create model dependencies, use SQL window functions, and implement automated data quality tests. But we won't stop at transformation. You will create a Python orchestration script that brings the different stages together, automatically fetching new API data, loading it into DuckDB, running dbt transformations, validating the results, and repeating the pipeline on a schedule. Finally, you will build an interactive Streamlit monitoring dashboard connected directly to the Gold Layer and use Plotly to visualize the latest market data. By the end of the course, you will understand how the diffe…

Courses in the same category