Menu

Post image 1
Post image 2
1 / 2
0

Why Your ADF Pipeline Shows Success But Your Data Is Wrong

DEV Community·Chandramouli Kondamudi·26 days ago
#24PkeoeJ
#azure#dataengineering#data#etl#pipeline#load
Reading 0:00
15s threshold

I have been working with ADF in production for 5 years. Most irritating errors are not in Pipeline monitoring dashboards we built , not in production review calls , but raised by user . Something is wrong but no trace in the pipeline on what tripped the system to get this bug . Here are 3 of those irritating issue which i wish i caught before any user raises concern on integrity of the data we are showing. The Decimal ID That Broke Your Fact Table Join ID Column from the Source is coming with decimals i.e 1453.0 , 1238.0 etc because source view changed the formatting , The Same ID Column in other Dimension Data Load is still integer , Now your Fact Table Cannot Join with Dimension Table Causing Sudden Failure of the Dashboard numbers , No ADF Pipeline Notification , but now you need to answer to Management on why numbers are wrong.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More