Menu

Post image 1
Post image 2
1 / 2
0

Day 5 - Chunking continued - RAG

DEV Community·Indumathi R·18 days ago
#9tqpVPwi
#ai#nlp#rag#tutorial#chunking#window
Reading 0:00
15s threshold

Sliding window chunking To understand this method, we need to know about two parameters, window size and step size . Let's now see how with the help of these two parameters, sliding window chunking works. Consider the following : Sample text: Redis is an open-source, in-memory data store that is primarily used as a cache, database, and message broker. Unlike traditional databases that store data on disk, Redis keeps data in memory (RAM), which makes data access extremely fast. It is commonly used in applications where high performance and low latency are critical, such as caching frequently accessed data, managing user sessions, real-time analytics, task queues, and messaging systems. Window size =15 Step size =5 Window position is at the first character. It takes the first 15 characters and stores them in chunk1. Redis is an op . Now the window moves, how farther it is gonna move will be based on step size. Since we are considering it as 5, window moves 5 characters.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More