What is dbt? Founded by fishtown analytics, data build tool (dbt) is one of the novel tools talked about in Data Engineering. The success of dbt derives from the simplicity to create an orchestration data pipeline by using standard SQL and YAML configurations. Fishtown analytics achieves this by abstracting any pipeline DAG complexities…

Here is a 3 step guide to significantly improving your deployment process by taking out the manual tagging and pushing in GitHub. Be aware that semantic release can be used in any cloud provider environment (GCP, AWS or Azure). Step 1: Understand Semantic Release Semantic release is an open-source npm framework that automates the release…

Here is a list of 5 things that you should know about to be an effective data engineer… 1. Become an Architect Nerd I guarantee that most of you will or have worked with a Data Architect before. Even if you work at a small company with 0 architects, often the data engineer has to…

Introduction If you have worked in the data engineering space for some time, you will have heard of Apache Airflow and cloud computing. These concepts have been around for some time, but how powerful is Apache Airflow with Cloud Composer? If you have ever faced issues with running and monitoring data…

For code to be considered complete and reliable, more often than not people will ask “How do you know the function works as expected?” or “Have you tried to break your own code?”. Either question, it’s clear that unit testing is salient to being confident that your program will execute…

If you are anything like me, when a significant event occurs, I become curious about the data sitting behind key measures. Have you ever looked at a visualisation shown by the Government during a daily Coronavirus briefing and thought about the underlying data? How about whether you want to create…