In this blog I will introduce the core ideas behind the darn package - designed to avoid degraded trust in RAG systems caused by lost context in chunks There are many documented chunking strategies for RAG systems readily available online. These can range from incredibly simple character (or token) limits, to rules-based splitting strategies, or even LLM backed semantic chunking methods. From my experience however, none of these methods provide the production-worthy ‘one-size-fits-all’ approach that they claim to: Limit-based strategies are not context aware enough to work in documents with anything more than short paragraphs of plain text involved, often leading to contextless half sentences or the bottom of tables being lost to orphaned chunks. ‘Simple’ rules-based strategies inevitably grow in complexity as more edge cases are found until the point that the code becomes completely unmanageable.…