Menu

Post image 1
Post image 2
1 / 2
0

Day 4 - Chunking continued - RAG

DEV Community·Indumathi R·21 days ago
#JBVffy38
#ai#nlp#python#rag#sentence#chunking
Reading 0:00
15s threshold

Semantic Chunking Lets Consider two paragraphs A and B, focussing on strings in python. para A focus on typecasting and para B focus on accessing characters. These two paragraphs are not that related to each other but if i do overlapping, these two points will be closer to each other. We do not want to forcefully bring the two paragraphs together. To solve this problem, semantic chunking can be used. It will continue to add sentence to a chunk until the relevancy is present. i.e It will take first sentence, since there is nothing to compare it will add it to a chunk. Next it will the take the second sentence and compare it with the previous sentence, if the relevancy factor is > 0.75 , second sentence will be added to chunk. Next sentence will be taken and compared with the previous sentence. If the relevancy factor is < 0.75, it won't be added to chunk otherwise it will be added. Semantic chunking can be achieved by means of nltk package.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More