I Discovered That a Cancer Prediction AI Was Using the Hospital Name — Not the Tumor — To Make Its Diagnosis. So I Built a Tool to Catch It. Built for the Build with MeDo Hackathon · #BuiltWithMeDo The Moment That Changed Everything Last week I was studying how machine learning models are deployed in hospitals. I stumbled across a research paper that stopped me cold. A cancer prediction model — the kind that tells doctors whether a patient needs aggressive treatment — was tested across multiple hospitals. It worked brilliantly at Hospital A. Terrible at Hospital B. Same cancer. Same biology. Different results. Why? The model had learned to use hospital_site_id as its strongest predictor. Not tumor size. Not mitotic count. Not any clinical feature that a doctor would recognize. The AI had essentially memorized which hospital a patient came from and used that to predict cancer outcomes. This is called proxy bias , and it's one of the most dangerous failure modes in clinical AI.…