Menu

Post image 1
Post image 2
1 / 2
0

How I Built a Hosted Web-Crawler-to-Knowledge-File Pipeline (and Shipped It as an MCP Server in 6 Weeks)

DEV Community·KazKN·30 days ago
#StwpH4XD
#ai#opensource#apify#actor#crawler#docs
Reading 0:00
15s threshold

I am a Switzerland-based indie developer. In the last six weeks I forked a 19,000-star open-source crawler, wrapped it for Apify's serverless infrastructure, added an MCP server mode, shipped 10 builds, and pushed the actor through Apify's automated daily test until the "Fix Actor health issues" warning cleared. The actor is kazkn/gpt-crawler-mcp . This post is the build log. I am writing it because three people I respect asked me independently in the same week how I would put a "knowledge crawler" into the hands of non-developers without making them install Playwright. Whoever you are, if you have ever thought "this would be a great Apify actor, but I do not want to fight the runtime" , this is for you. Key takeaway up front: the value is not the crawler. There are dozens of crawlers. The value is removing every reason a user would not click Run — pinned runtime, prefilled inputs, MCP standby, transparent pricing, screenshots that work. The build log below is mostly about that one job.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More