Programming Study日本語

Disclosure: We earn a commission from Udemy when you buy a course through links on this page.

Data Engineering Fundamentals with Prefect Workflow

Data Engineering Fundamentals with Prefect Workflow

Udemy · English

About this course

Language
English
Category
Provider
Udemy
Data updated

Data engineering is the process of designing and building systems that let people collect and analyze raw data from multiple sources and formats. These systems empower people to find practical applications of the data, which businesses can use to thrive. Companies of all sizes have huge amounts of disparate data to comb through to answer critical business questions. Data engineering is designed to support the process, making it possible for consumers of data, such as analysts, data scientists and executives, to reliably, quickly and securely inspect all of the data available. About a decade back, the data analysis was merely on the structured data available on the a Relational data base or in ERP system and any decision was made based on analysis of the historic data and tools like ETL (extract, Tranform & load) was used for datawarehousing system. However in this dynamic ever changing world, non relational data base information need to used for quick analysis. So apart from transactions in database, the other source of web information from CSV, webhooks, http & MQTT need to taken care as appropriate. Further more, the process of ETL as evolved into Data pipelines. A data pipeline is a method in which raw data is ingested from various data sources and then ported to data store, like a data lake or data warehouse, for analysis. In data pipe line task dependency can be build with different task. These task can be also based on some events happening like Order booked or Issues raise which can trigger a task. For this concepts of Webhooks are used. Prefect is one such newly evolved data pipeline or workflow tool, in which one can build not only static task dependency, but these task dependency can be built based on some event happeningas well. This course uses the cloud version Prefect worflow tool which can be invoked from a cloud based virtual machine. Knowledge of Python & shell scripting is essential. This course cov…

Courses in the same category