Data Engineer · London
Karsh
Vashi.
I build data pipelines and the systems around them. This is where I write up my own projects, with the architecture and the reasoning, and explain the ideas underneath them.
Data flows from the MovieLens 20M CSV files into five raw tables in Snowflake. dbt builds five staging views over them, then six dimension and fact models, two of them incremental, then two mart tables. A snapshot, snap_movies, reads the raw movies table directly and keeps SCD Type 2 history. Airflow runs the project monthly through Astronomer Cosmos, one task per model.
- MovieLens 20M feeds Snowflake RAW
- Snowflake RAW feeds staging
- Snowflake RAW feeds snap_movies
- staging feeds dims + facts
- dims + facts feeds marts
Selected builds
movie-ratings-analytics
A layered dbt project on Snowflake over the MovieLens 20M dataset, with SCD Type 2 snapshots, incremental models, source freshness, a custom generic test, per-pull-request CI and Airflow orchestration at per-model granularity.