Hey friends, if you've found this blog, like me, you probably want everything you do to be dynamic at runtime in some way or another. And of course metadata driven. Sadly, Microsoft Fabric does not currently want/like this. Maybe because its Software as a Service or for business users and everything must be static/hardcoded. No judgement. Its just not how I want to build a data platform.
Context
When building dynamic pipelines and other artifacts in Microsoft Fabric we are currently forced to reference everything using the underlying item IDs, rather than the more useful names (display names).
In the UI, setting a item value is fine when selecting items from the respective drop down lists. But they will of course be static and can't be changed at runtime. However, as soon as an expression is required (which in the real world, always is) those UI labels change to be the ID values. AKA the item GUIDs in the context of the workspace and wider solution.
Most 'Googling' on this will advise that you to simply get the GUID from the address bar of your browser. But of course this doesn't help for runtime executions!
Then the problem is further compounded when you want to deploy items across different environments (workspaces), because of course the item ID changes and is specific to the workspace you are in.
Maybe fine if could control this through deployment pipelines, but we can't. And still not helpful for metadata driven executions.
Problem - Part 1
Next, lets start exploring ways to get the item ID at runtime via the Fabric API. This is certainly possible for a given workspace and extending the GET request with the item type helps. Example URL below.
https://api.fabric.microsoft.com/v1/workspaces/{workspace ID}/items?type=DataPipeline
Full MS Docs on using the Fabric API here: https://learn.microsoft.com/en-us/rest/api/fabric/articles/
But, filtering the list of results by display name still isn't an option in the API call to help with the Value array returned.
Problem - Part 2
Assume we do this in a Fabric Pipeline with the Web Activity (Web v2) and the annoying connection manager, requiring SPN details, because Managed Identity isn't supported as an authentication method yet. Something like the below screen shot.
The result only in Fabric, doesn't list the ID values in the Value array returned. But it does if you hit it directly from PostMan or with something other than Fabric.
Why Microsoft?!
Solution (Hack)
Here's what I did. I'm not proud. But it works!
Hit the Fabric API using a Data Factory Pipeline with a Web Activity that supports Managed Identity, allows a dynamic URL and (for whatever reason) does return the item ID in the result set.
Filter the results to the display name.
Pass the item ID as a pipeline return value to whatever actually needs it.
Run the Data Factory Pipeline within Fabric Pipeline as a precursor to your other metadata driven processes for Notebooks etc etc.
This is what it looks like:
In Data Factory
In Fabric
Result
The ability at runtime to get an item ID and pass it to other activities as part of your orchestration.
Getting it to check itself! :-)
I fully accept that with some more coding this could be done in Python or with external Azure Functions. But that requires other effort and in Fabric Python running in a Notebook using Spark compute is overkill, for just hitting the Fabric API to fetch a single value.
Both pipeline JSON definitions are available in the following GitHub repo. Good luck.
Thanks for reading.
Comments