Menu

How I handle bulk WHOIS lookups at scale: lessons from running a domain API
📰
0

How I handle bulk WHOIS lookups at scale: lessons from running a domain API

DEV Community·WhoisJson·about 1 month ago
#dczvKlWF
#python#nodejs#devops#security#domain#const
Reading 0:00
15s threshold

I've been running a WHOIS API since 2016. For a long time, most users were doing one-off lookups, maybe a few hundred a day. Then at some point customers started showing up with different needs entirely. Threat intel pipelines, domain portfolio audits, registrant enrichment at scale. One customer hit 30 million requests in a single month. That's a different problem from building a WHOIS wrapper. And most of the articles I found when dealing with it were either too basic or skipped the parts that actually bite you in production. So here's what I've learned. Rate limits are the easy part The obvious problem is staying under your API rate limit. 429, back off, retry. Everyone knows this. But with WHOIS specifically there's a second layer: the upstream registries themselves. Hit certain ccTLDs too aggressively and you get silent failures, not clean errors. No 429, just empty responses or stale data that looks valid. I ran into this with .nu and .gr.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More