
Databricks Data Engineering: The Real World Simulator
Udemy ยท English
About this course
- Language
- English
- Category
- Provider
- Udemy
- Data updated
Helios Trading Corporation needs a Data Engineer. That is you. Helios ships parts across six depots in the solar system. Ten source feeds land every day. Nobody has built the pipeline yet. This is not a tutorial you follow. It is a job you do. Simulate, then ExecuteEvery build starts as a conversation. A short AI Role Play with a colleague, and goals you are measured against. Then you build what you talked through. Most labs are work tickets: context, task, and acceptance criteria that run as real checks in the notebook. What you will build1. Design Meet Helios, walk the ten source feeds with the person who owns them, and agree the Medallion lakehouse design that every later section implements.2. Databricks Jobs The classic build. Auto Loader into Bronze, Structured Streaming and MERGE into Silver, SCD Type 2 for prices and customer tiers, a Gold star schema with point in time pricing, then a five task orchestrated Job with retries, alerts and a schedule.3. Lakeflow Spark Declarative Pipelines The same platform, rebuilt declaratively. Streaming tables, expectations as quality gates, AUTO CDC. Then an inherited pipeline fails on a night run and you diagnose it from the event log.4. Genie and AI/BI Dashboards A semantic layer of Metric Views over Gold, so one definition of revenue feeds both a Genie Space and a dashboard. Then you make the case for self serve to the Commercial Director.5. Lakebase and a Databricks App Sync curated Gold into Lakebase and deploy a live Depot Operations Console. Source feeds to running app. Real data, from a purpose built generatorThe data comes from a custom synthetic generator written for this course, not a downloaded sample set. Ten interlocking feeds in three formats, around 293,000 order lines across fiveโฆ











