Menu

Post image 1
Post image 2
1 / 2
0

Designing Idempotent Bulk Import Pipelines (E.164, VIN, and the Rest)

DEV Community·Hammad Khan·20 days ago
#wD9lRpEw
#systemdesign#validation#node#user#rows#phone
Reading 0:00
15s threshold

Designing Idempotent Bulk Import Pipelines (E.164, VIN, and the Rest) Bulk imports are a special category of pain. You give the user a CSV uploader, they hand you a file, and your job is to take that file from "blob of text" to "thousands of clean rows in your database." Somewhere in the middle, every assumption you have about your data falls apart. I rebuilt the bulk import system for a dealership SaaS recently. The redesign covered phone number normalization, VIN format validation, file-format consistency, and the structural decision that makes the whole thing reliable: idempotency. If you've ever shipped a bulk importer, you've probably hit the same problems. Here's the shape of the design that finally worked. The four problems Any bulk import system has to solve four problems: Validation. Every row has to satisfy schema constraints before it goes into the database. Some constraints are obvious (required fields), some are domain-specific (a VIN must be 17 characters in a specific alphabet). Normalization.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More