Building a Pipeline for your Medallion Architecture on your Microsoft Fabric Lakehouse
It was on a Friday morning…
I was sitting on my chair with my laptop with a serious face, studying the Microsoft Learn modules on the topic, ‘Organize a Fabric lakehouse using medallion architecture design’, while sipping orange juice.
Actually, that’s a lie.
No doubt, I was studying that topic but it was with a carefree mind and I was lying on my bed with no sort of tasty liquid wetting my mouth.
Sorry for the delay. I just wanted to be a little bit dramatic.
Okay. Back to business
Like, I said I was going through the Microsoft Learn module on Medallion Architecture. I also practised with the exercise. This exercise guided me on how to build the architecture by running Notebooks for each layer.
A quick description of Medallion Architecture…
See Medallion Architecture as a structured format of your usual Extract-Transform-Load (ETL). This architecture consists of three layers: Bronze, Silver, and Gold where each layer has more refined data than the previous layer. The Bronze layer usually handles the extraction of data from different sources. The Silver layer handles the heavy transformation tasks. While the Gold layer involves tasks that get the data more ready for consumption eg. data modeling, joins, aggregations, etc.
With an understanding of this architecture, there was one problem I had with the Microsoft Learn exercise. It was not suitable for a production environment. It didn’t make sense that I had to run the notebook for each layer manually.
How can this be automated?🤔
It took a while but I got the answer; A data pipeline.
So yes, in this article, I’ll take you through the process of building a data pipeline that orchestrates the process of implementing a medallion architecture in your Microsoft Fabric lakehouse.
I’m assuming that you should have gone through the module and built the notebooks as guided in the module exercise.
Now that you have the notebooks, let’s dive into my step-by-step guide.
- Create a new pipeline and give it your desired name.
2. A new window will open after your new pipeline is successfully created. Click the ‘Add pipeline Activity’ option.
3. Then click the ‘Notebook’ option in the pop-up menu as shown below
4. You’ll have a screen like this. Notice that ‘General’ and ‘Settings’ are at the bottom right. Drag the white bar up with your cursor.
5. Rename your activity and add the description if you wish.
6. Choose your workspace and notebook. Notice that we started with the Silver layer and not the bronze layer because the Microsoft Learn module goes through the bronze layer as simply the uploading of your files to your lakehouse.
If you have a notebook that extracts data from different sources. That should be your bronze layer notebook and you should start with it instead.
7. Add an ‘If condition’ since we want the next notebook to run after the previous notebook has completed successfully
8. Rename your activity and add the description if you wish.
9. When you click the ‘Activities’ tab, you’ll see an input box beside the word, ‘Expression’. That’s where we are going to add dynamic content that will run the gold layer notebook if the silver notebook runs successfully. And send you an email if the silver notebook fails to run.
As soon as you click the ‘Expression’ input box, you’ll see a text that states ‘add dynamic content’. Click it and this pop-up in the right part of the email will show.
Paste this into the box:
@equals(activity(‘<name of your previous activity>’).output.status, ‘Succeeded’)
10. Add your gold notebook to the True condition
11. Make the necessary settings
12. For the false condition, we’ll be setting up an email notification. So choose the ‘Office 365 Outlook’ option.
13. Adjust your settings and don’t forget to add your email address
14. Since this is a pipeline, you might want to schedule it to run at a specific period. In this case, I chose 8 am daily.
Now let’s test our pipeline to see if it worked as expected.
Tada!
It ran successfully. Now what happens if something is wrong with my Silver layer notebook and it fails?
The Silver layer notebook failed and I got an email.
Let’s quickly add another activity. We would also want to know if the gold layer notebook failed or was successful. So we’ll add another if condition that has an email notification for both the True and False conditions. The only difference will be the messages.
Here’s a screenshot of the new activity added and the run status
Now let’s see if I got an email on the status of my Gold layer notebook.
And yes, I got an email stating that my pipeline ran successfully.
I hope my guide was comprehensive enough to help you orchestrate a medallion architecture on your Fabric lakehouse. I’m looking forward to getting feedback on how helpful the guide was. So don’t forget to leave a comment when you try this out.
Catch ya later!
