Chunk Settings

The 'Offset' property determines how many words Grooper moves forward in the text to start each new chunk. This controls the non-overlapping portion of each chunk and, together with 'Overlap', defines how much new content is included in each segment.

• A higher value reduces the number of chunks generated, improving performance but potentially reducing context.
• A lower value increases the number of chunks and the amount of overlap, which can improve search and embedding accuracy at the cost of additional processing.

Example:
If 'Offset' is set to 20 and 'Overlap' is 10, each chunk will start 20 words after the previous one, but will also include 10 words from before and after its main segment for context.

Adjust this value based on the typical length of meaningful content in your documents and the requirements of your search or embedding model.

The 'Overlap' property specifies how many words from the previous and next chunk are included at the beginning and end of each chunk, respectively. This overlap helps preserve context across chunk boundaries, which is especially important for natural language processing tasks.

• Increasing 'Overlap' ensures that important information near the edges of chunks is not lost, but may result in more redundant processing.
• Decreasing 'Overlap' can improve performance but may reduce the effectiveness of search and embedding operations if relevant context is split between chunks.

Example:
With 'Offset' set to 20 and 'Overlap' set to 10, each chunk will contain 10 words from the previous chunk, 20 new words, and 10 words from the next chunk, for a total of 40 words.

Choose an overlap value that matches the typical context window needed for your use case and model.

The 'Chunk Size' property displays the total number of words in each chunk, calculated as Offset + Overlap * 2. This value reflects the full size of each segment that will be processed for search or embedding.

• Use this property to verify the effective chunk size based on your current 'Offset' and 'Overlap' settings.
• Larger chunk sizes provide more context but may exceed the input limits of some models.
• Smaller chunk sizes may improve performance but risk losing important context.

Adjust 'Offset' and 'Overlap' to achieve a chunk size that fits your content and the requirements of your downstream processing.