Menu

Post image 1
Post image 2
Post image 3
1 / 3
0

# How I Built a PDF Processing Pipeline Designed to Scale Toward 10k Users — FastAPI, Inngest & Smart Async Design

DEV Community·Mohamed Elsayed Ali·26 days ago
#c3xvrM6H
Reading 0:00
15s threshold

I recently built a PDF processing pipeline inside my project Playground . The goal was simple: Users upload PDFs, the API responds immediately, and heavy processing happens in the background without blocking request latency. That sounds straightforward until traffic increases. A naive implementation works fine for a few users — but starts falling apart when many uploads arrive at once. In this article, I’ll walk through the architecture, the async design decisions, and a few lessons I learned while building it. Repository: Playground on GitHub The Problem A common first implementation looks something like this: receive upload parse PDF inside the request extract text store results return response It works — until multiple users upload files concurrently. The biggest issue is that PDF parsing is not purely async work . Libraries like pdfplumber perform CPU-heavy parsing. If that happens directly inside an async route, the event loop gets blocked.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More