Word Match

The 'Word Pattern' property defines the rule for what constitutes a "word" during extraction. This pattern is applied to the document text to locate sequences of characters that should be treated as words for further processing and phrase assembly.

How It Works

• The extractor scans the text and matches all substrings that fit the regular expression provided.
• By default, the pattern '\p{L}+' matches any sequence of Unicode letters, supporting words in any language.
• You can customize the pattern to restrict matches by language, character set, or length, or to tolerate common OCR errors.

Impact

• The choice of pattern directly affects which words are extracted and how phrases are assembled.
• A broad pattern may capture unwanted fragments, while a restrictive pattern may miss valid words.
• For OCR documents, patterns can be adapted to match common misreadings (e.g., digits in place of letters).

Examples

Usage Scenarios

• Extracting words for classification or feature generation.
• Identifying person names, codes, or key terms in structured or unstructured text.
• Improving extraction accuracy for OCR-based documents by accounting for common errors.

> Use diagnostics to review which words were matched and to tune the pattern for your data.

The 'Prefix Pattern' property allows you to specify a regular expression that must be present immediately before each match found by the Text Match extractor. This enables context-sensitive extraction, ensuring that only values preceded by a specific pattern, label, or structural element are returned.

Purpose

Use this property to restrict matches to those that occur after a particular label, whitespace, line break, or other context. This is especially useful for extracting labeled values, enforcing boundaries, or avoiding false positives in complex documents.

Configuration Guidance

• Enter a regular expression that describes the required prefix. The pattern is applied directly before each match.
• Common examples include whitespace, document labels, line or page boundaries, or exclusion of certain characters.
• Use anchors (such as ^ for start of document, \f for page break, \n for line break) to target structural positions.
• Combine multiple options using the | (OR) operator for flexibility.

Examples

Impact

• Only matches preceded by the specified pattern are included in the output.
• Improves accuracy by filtering out values that do not occur in the desired context.
• Can be used in combination with 'Suffix Pattern' for even more precise extraction.

Usage Scenarios

• Extracting values labeled with a specific phrase (e.g., Invoice No:).
• Restricting matches to the start of a page or line.
• Avoiding matches that are embedded within words or unwanted regions.

> Use diagnostics to review which prefix patterns were applied and to troubleshoot extraction boundaries.

The 'Suffix Pattern' property allows you to specify a regular expression that must be present immediately after each match found by the Text Match extractor. This enables context-sensitive extraction, ensuring that only values followed by a specific pattern, label, or structural element are returned.

Purpose

Use this property to restrict matches to those that occur before a particular label, whitespace, line break, or other context. This is especially useful for extracting values with trailing units, enforcing boundaries, or avoiding false positives in complex documents.

Configuration Guidance

• Enter a regular expression that describes the required suffix. The pattern is applied directly after each match.
• Common examples include whitespace, document labels, line or page boundaries, or exclusion of certain characters.
• Use anchors (such as $ for end of document, \f for page break, \r for line break) to target structural positions.
• Combine multiple options using the | (OR) operator for flexibility.

Examples

Impact

• Only matches followed by the specified pattern are included in the output.
• Improves accuracy by filtering out values that do not occur in the desired context.
• Can be used in combination with 'Prefix Pattern' for even more precise extraction.

Usage Scenarios

• Extracting values with trailing units or labels (e.g., acres).
• Restricting matches to the end of a page or line.
• Avoiding matches that are embedded within words or unwanted regions.

> Use diagnostics to review which suffix patterns were applied and to troubleshoot extraction boundaries.

Overview

The Environment Options class controls how merge variables are resolved and how culture information is applied during extraction. Merge variables are referenced in regular expressions using the syntax @VariableName. These variables can represent lists of values, culture-specific data, or reusable regex fragments.

Built-In Merge Variables

Several built-in merge variables are available (see Variable Providers), exposing culture-specific lists such as @DayNames, @MonthNames, and @CurrencySymbols. For example, the @DayNamesAbbreviated variable expands to a list of day abbreviations appropriate for the current culture:

• For English: Sun|Mon|Tue|Wed|Thu|Fri|Sat
• For French: dim|lun|mar|mer|jeu|ven|sam

These variables adapt automatically to the culture of the document or to a specified override.

Custom Merge Variables

You can define custom merge variables by referencing Lexicons that contain the desired values or key-value pairs. Custom variables are useful for:

• Merging in lists that are difficult to maintain directly in a regex.
• Creating libraries of reusable regex snippets shared by multiple extractors.
• Improving the readability and maintainability of complex patterns.
• Enforcing consistency across multiple extractors.

To define custom variables, use the 'Value Lists' and 'Snippet Libraries' properties to reference appropriate Lexicons. Each lexicon entry becomes a merge variable, which can be injected into your regular expressions using the @VariableName syntax.

Culture Settings

The culture used for variable expansion can be controlled via the 'Culture Override' and 'Culture Scope' properties. By default, variables are generated using the culture of the input document, but you can force a specific culture or restrict processing to certain languages using these options.

This is especially useful when extracting data from mixed-language documents, or when you need to standardize extraction behavior across different locales.

Usage Guidance

• Use built-in variables for common culture-specific lists.
• Reference Lexicons to define custom lists or regex snippets as merge variables.
• Adjust culture settings to match your extraction requirements, especially for multilingual or internationalized content.
• For more details on configuring merge variables and culture options, see the documentation for Lexicons, Variable Providers, and the relevant properties of this class.

The 'Case Sensitive' property controls whether the regular expression pattern, prefix, and suffix matching performed by the Text Match extractor will distinguish between uppercase and lowercase letters.

Purpose

Enable this property when the capitalization of text is meaningful for your extraction scenario, such as distinguishing between proper names, acronyms, or case-specific labels.

Configuration Guidance

• Set to true to require exact case matches (e.g., "Invoice" ≠ "invoice").
• Set to false to allow matches regardless of case (e.g., "Invoice", "INVOICE", and "invoice" are all equivalent).
• Use case-sensitive matching for scenarios where capitalization conveys meaning, such as extracting section headings, entity names, or codes that are case-dependent.
• For most business data, case-insensitive matching (false) is recommended to maximize extraction accuracy.

Impact

• When enabled, only text that matches the exact case of the pattern will be extracted.
• When disabled, matches will be found regardless of case, increasing the number of potential hits.

Examples

> Use diagnostics to verify which matches were found and to troubleshoot case-related extraction issues.

The Text Preprocessor enables advanced manipulation of control characters in a document's text, allowing regular expressions to match or ignore structural elements such as line breaks, paragraph boundaries, page breaks, tabs, and spaces.

Overview

Text preprocessing is performed immediately before extraction, transforming the document's text to improve the accuracy and flexibility of pattern matching. This is especially useful when data values span multiple lines, are separated by large whitespace gaps, or are affected by inconsistent formatting.

Key Features

• Paragraph Marking:
  Detects paragraph boundaries and converts line breaks within paragraphs to spaces, while preserving paragraph-ending breaks. This allows extractors to match values that span multiple lines within a paragraph, without matching across paragraph boundaries. See Paragraph Marker.

• Tab Marking:
  Replaces large horizontal whitespace gaps with TAB characters, making it possible to distinguish between normal spaces and significant gaps in regular expressions. See Horizontal Tab Marker.

• Vertical Tab Marking:
  Converts certain line breaks to vertical tab characters based on vertical spacing, enabling recognition of vertical structure in tabular or multi-column layouts. See Vertical Tab Marker.

• Control Character Ignoring:
  Removes or replaces selected control characters (such as spaces, newlines, form feeds, and carriage returns) according to the 'Ignore Control Characters' setting. This can simplify extraction in documents with inconsistent or excessive whitespace.

Usage Guidance

• Configure the desired preprocessing options by enabling or disabling paragraph, tab, and vertical tab marking, and by selecting which control characters to ignore.
• Preprocessing is typically used in conjunction with regular expression-based extractors, but can benefit any extraction scenario where document structure affects pattern matching.
• For best results, adjust preprocessing settings to match the structure and formatting of your source documents.

Example Scenarios

• Extracting values that span multiple lines within a paragraph:
  Enable paragraph marking to convert internal line breaks to spaces, allowing regular expressions to match values split across lines.

• Distinguishing between normal spaces and large gaps:
  Enable tab marking to insert TAB characters at significant horizontal gaps, so extractors can target fields separated by large whitespace.

• Cleaning up unwanted whitespace or control characters:
  Use the 'Ignore Control Characters' option to remove or replace problematic characters that interfere with extraction.

For more details, see the documentation for Paragraph Marker, Horizontal Tab Marker, and Vertical Tab Marker.

Examples

1. Sample Document

Consider the following sample document.

┌─────────────────────────────────────────────────────────────┐
│                        SAMPLE FORM                          │
├─────────────────────────────────────────────────────────────┤
│ Name:           John Doe                   ID: 12345        │
│ Date of Birth:  01/01/1980                 Status: Active   │
├─────────────────────────────────────────────────────────────┤
│ This is the first paragraph. It explains the purpose of     │
│ the form and the meaning of each field.                     │
│                                                             │
│ Please complete all fields and verify all personal          │
│ information before submitting. Thank you!                   │ 
└─────────────────────────────────────────────────────────────┘

2. Default Control Characters

With no preprocessing options enabled, the document data will look like this. Whitespace gaps, no matter how large, are represented by a single space character. A \r\n pair marks each location where the original document wrapped to the next line.

SAMPLE FORM\r\n
Name: John Doe ID: 12345\r\n
Date of Birth: 01/01/1980 Status: Active\r\n
This is the first paragraph. It explains the purpose of\r\n
the form and the meaning of each field.\r\n
Please complete all fields and verify all personal\r\n
information before submitting. Thank you!\r\n

3. Preprocessed Version

Preprocessing the document with paragraph marking and tab marking will place a tab character '\t' at each large whitespace gap, and replace newline pairs '\r\n' occuring inside a paragraph with a space.

SAMPLE FORM\r\n
Name: John Doe\tID: 12345\r\n
Date of Birth: 01/01/1980\tStatus: Active\r\n
This is the first paragraph. the form and the meaning of each field.\r\n
Please complete all fields and verify all personal information before submitting. Thank you!\r\n

The 'Word Lookup' property allows you to apply a vocabulary, exclusion list, or fuzzy matching logic to each word identified by the extractor. This is useful for enforcing allowed values, correcting OCR or spelling errors, and mapping words to standardized forms.

How It Works

• After words are extracted using the 'Word Pattern', each word is checked against the lookup.
• Words not found in the vocabulary can be discarded, corrected using fuzzy matching, or translated to a replacement value.
• Exclusions can be used to filter out unwanted words (e.g., "N/A", "Unknown").
• Options such as case normalization and punctuation cleaning are available.

Impact

• Improves extraction accuracy by filtering, correcting, or standardizing word values.
• Reduces noise in classification and data extraction by removing invalid or unwanted words.
• Enables mapping of synonyms, abbreviations, or alternate spellings to a single normalized value.

Examples

Usage Scenarios

• Enforcing a controlled vocabulary for codes, names, or terms.
• Automatically correcting common OCR or typographical errors.
• Mapping abbreviations or synonyms to standardized values.
• Filtering out irrelevant or blacklisted words.

> Use diagnostics to review which words were validated, corrected, or excluded by the lookup.

The 'Chunk Size' property enables chunked processing for large documents, allowing the Text Match extractor to break the document into smaller segments and process each chunk separately.

Purpose

Use this property to optimize extraction performance and memory usage when working with documents containing many pages (hundreds or thousands).

Configuration Guidance

• Enter the number of pages per chunk (e.g., 1000).
• If set to a value greater than zero, the extractor will process the document in chunks of the specified size.
• If left at 0, chunked processing is disabled and the entire document is processed as a single unit.
• Choose a chunk size that balances performance and resource usage for your environment.

Impact

• Reduces memory consumption and improves responsiveness for very large documents.
• Extraction results from each chunk are combined into the final result set.
• May affect the ability to match patterns that span chunk boundaries; adjust chunk size as needed.

Usage Scenarios

• Processing scanned books, contracts, or reports with thousands of pages.
• Avoiding out-of-memory errors or slow extraction times on large documents.
• Tuning chunk size to match available system resources and desired throughput.

> Use diagnostics to review chunk boundaries, extraction times, and the number of results produced per chunk.

Can be one of the following types:

The 'Phrase Size' property determines how many adjacent words are grouped together to form a phrase, also known as an N-gram. This enables the extractor to capture not only single words, but also meaningful multi-word combinations that provide richer context for classification and data extraction.

How It Works

• When set to 1, the extractor outputs single words (unigrams).
• When set to 2, it outputs all possible two-word phrases (bigrams).
• Higher values (up to 5) produce longer phrases, such as trigrams (3), four-grams (4), and five-grams (5).
• All contiguous combinations of the specified size are produced, subject to join rules and validation.

Impact

• Larger phrase sizes capture more context, which can improve classification accuracy and enable extraction of multi-word entities (e.g., "John Doe", "remittance advice").
• The number of output phrases increases with phrase size, especially in longer documents.
• Phrase extraction is valuable for identifying key terms, names, or features that span multiple words.

Examples

For the text 'the quick brown fox':

• Phrase Size 1: 'the', 'quick', 'brown', 'fox'
• Phrase Size 2: 'the quick', 'quick brown', 'brown fox'
• Phrase Size 3: 'the quick brown', 'quick brown fox'

Usage Scenarios

• Extracting person names, organization names, or other multi-word entities.
• Generating N-gram features for machine learning-based classification.
• Identifying key phrases for document categorization or search.

> Use diagnostics to review the number and type of phrases produced, and adjust phrase size for your scenario.

The 'Join Pattern' property controls how words are assembled into phrases (N-grams) by specifying what characters or whitespace are allowed between them. Only words separated by content matching this pattern are considered joinable.

How It Works

• By default, words are joinable if separated by a single space character.
• If a pattern is specified, words are joinable only if the gap between them matches the pattern.
• This allows for flexible phrase boundaries, such as joining words separated by spaces, hyphens, slashes, or other delimiters.

Impact

• Controls which word combinations are considered valid phrases.
• Enables extraction of phrases that span across delimiters (e.g., "self-employed", "A/C Number").
• Prevents joining words across unwanted boundaries, improving extraction accuracy.

Examples

For the text self-employed / contractor, a join pattern of [ /-] would allow extraction of:

• self-employed
• employed / contractor

Usage Scenarios

• Extracting multi-word entities that may be separated by punctuation or special characters.
• Controlling phrase boundaries to avoid joining across page breaks, line breaks, or unwanted delimiters.
• Improving the precision of N-gram extraction for classification or data normalization.

> Use diagnostics to review which word pairs were joinable and to tune the pattern for your documents.

The 'Term Options' property allows you to specify individual lookup, filtering, and normalization settings for each term (word) in a phrase. This enables fine-grained control over how each word in a multi-word phrase is validated, corrected, or standardized.

How It Works

• For each term in a phrase, you can assign a Group Options object that defines vocabulary, exclusions, fuzzy matching, and other lookup options.
• Each Group Options applies only to its corresponding term (e.g., first name, last name).
• This is especially useful when different terms require different vocabularies or validation logic.

Impact

• Enables extraction of complex entities where each part must meet distinct criteria (e.g., "John Doe" where "John" must be a valid first name and "Doe" a valid last name).
• Improves accuracy by allowing per-term correction, exclusion, or translation.
• Supports advanced scenarios such as multilingual extraction or entity normalization.

Examples

Usage Scenarios

• Validating each part of a multi-word entity against separate vocabularies.
• Correcting OCR errors for specific terms in a phrase.
• Mapping terms to standardized forms or codes.

> Use diagnostics to review which terms were validated or normalized for each phrase.

The 'Phrase Lookup' property allows you to apply vocabulary, exclusion, or fuzzy matching logic to the full phrase after it has been assembled from individual terms. This is useful for enforcing allowed multi-word values, correcting phrase-level errors, or mapping phrases to standardized forms.

How It Works

• After a phrase is assembled, it is checked against the lookup.
• Phrases not found in the vocabulary can be discarded, corrected using fuzzy matching, or translated.
• Exclusions can be used to filter out unwanted phrases.
• Options such as case normalization and punctuation cleaning are available.

Impact

• Ensures that only valid, normalized phrases are included in the output.
• Enables mapping of multi-word entities to codes, categories, or standardized values.
• Reduces noise and improves accuracy for downstream classification or extraction.

Examples

Usage Scenarios

• Validating extracted phrases against a list of allowed entities.
• Correcting or normalizing multi-word values for export or reporting.
• Mapping phrases to categories or codes for classification.

> Use diagnostics to review which phrases were validated, corrected, or excluded by the lookup.

The 'Minimum Term Hits' property controls how many individual terms in a phrase (N-gram) must pass their respective lookups or validations for the entire phrase to be included in the output. This is useful for scenarios where partial matches are acceptable, or where some terms may be optional or prone to errors.

How It Works

• Each term in a phrase is validated against its configured lookup or vocabulary.
• If the number of successful term validations meets or exceeds the value of 'Minimum Term Hits', the phrase is considered valid and included in the results.
• If fewer terms pass validation, the phrase is excluded.

Impact

• Allows for flexible extraction when some terms may be missing, misspelled, or not present in the vocabulary.
• Can be used to relax strict matching requirements, increasing recall in noisy or error-prone documents.
• Setting a higher value enforces stricter validation, improving precision but potentially reducing the number of results.

Examples

For a trigram phrase such as John Steven Doe with 'Minimum Term Hits' set to 2:

• If "John" and "Doe" are found in their respective vocabularies, but "Steven" is not, the phrase will still be considered valid.
• If only one term passes validation, the phrase will be excluded.

Usage Scenarios

• Extracting names or entities where some parts may be optional or variable.
• Handling OCR errors or incomplete data by allowing partial matches.
• Tuning extraction strictness for different document types or quality levels.

> Use diagnostics to review how many phrases were included or excluded based on term hits, and adjust this property to balance recall and precision for your scenario.

The 'Output Format' property allows you to customize how extracted phrases are presented by specifying a format string with placeholders for each term in the N-gram. This is useful for reordering terms, adding punctuation, or producing output in a specific style required for downstream processing or export.

How It Works

• The format string can include placeholders such as {Term1}, {Term2}, etc., which correspond to the first, second, and subsequent terms in the phrase.
• When a phrase is extracted, each placeholder is replaced with the value of the corresponding term.
• Any additional text, punctuation, or formatting in the string is preserved in the output.

Impact

• Enables flexible output formatting for extracted values, supporting custom requirements for display, export, or integration.
• Allows for reordering, combining, or transforming terms as needed.
• Can be used to match specific data formats, such as "Last, First Middle" for names.

Examples

For a trigram match on John Steven Doe with the format string: {Term3}, {Term1} {Term2} The output value will be: Doe, John Steven

For a bigram match on Invoice Number with the format string: {Term2}: {Term1} The output value will be: Number: Invoice

Usage Scenarios

• Formatting names, addresses, or codes for export or reporting.
• Reordering terms to match business or system requirements.
• Adding punctuation, labels, or other text to extracted values.

> Use diagnostics to review formatted output and ensure the format string produces the desired results.

The Result Filter allows you to configure a set of criteria that each Data Instance must meet to be included in the final result set. Results that do not match the specified conditions are excluded, enabling precise control over which values are retained for downstream processing or export.

Configuration and Usage

• Configure filter properties to target results by page, confidence, length, character composition, geometric region, font size, alignment, and more.
• Multiple filter criteria can be combined; a result must satisfy all enabled filters to be included.
• Use the 'Allowed Pages' property to restrict results to specific pages.
• Set 'Minimum Confidence' to exclude low-confidence results.
• Use 'Length Range' and 'Character Classes' to filter by value length and character makeup.
• The 'Exclusions' property allows you to specify a Value Extractor whose results will be used to exclude overlapping matches.
• Geometric filters (such as 'Region', 'Width Range', 'Height Range', and 'Font Size Range') allow you to target results by their position and size on the page.
• Emphasis filters (such as 'Character Casing', 'Case Similarity', 'Minimum Font Scale', and 'Require Underline') help isolate results with specific visual or typographic emphasis.

Typical Scenarios

• Restricting to a Region:
  Only include results that fall within a specific area of the page, such as a header or footer.
• Filtering by Confidence:
  Exclude results below a certain confidence threshold to improve data quality.
• Targeting Emphasized Text:
  Isolate headings, titles, or other elements with distinctive font size, casing, or underlining.
• Excluding Unwanted Results:
  Use the 'Exclusions' property to remove results that overlap with known false positives or irrelevant content.

Related Types

• Data Instance: The objects being filtered.
• Value Extractor: Used for exclusion filtering.
• Character Class Filter, Logical Rectangle, Logical Range, Horizontal Alignment Settings: Used for specific filter criteria.

For more information, see the documentation for Data Instance, Value Extractor, and Result Filter properties.

The Result Set Options class provides a flexible set of controls for shaping the output of data extraction and classification activities in Grooper. It allows you to define how individual results are adjusted, how the overall result set is filtered or ordered, and how output values are normalized for downstream use.

Overview

Use this class to:

• Normalize extracted values (e.g., unify variations, apply case or word transformations).
• Adjust or override confidence scores for results.
• Remove unwanted content from output values.
• Sort, filter, or deduplicate the result set.
• Enforce minimum/maximum hit counts or truncate the number of results.

These options are commonly configured on Data Fields, Data Types, or other extraction elements to ensure that the output meets business requirements and is ready for validation, export, or further processing.

Key Scenarios

• Value Normalization:
  Use 'Value Override', 'Case Normalization', or 'Word Normalization' to standardize output values, making them consistent for classification or export. For example, you can map multiple document title variations to a single normalized feature, or apply stemming to treat inflected word forms as equivalent.

• Confidence Adjustment:
  The 'Confidence Override' and 'Confidence Multiplier' properties allow you to boost or reduce the confidence of results, influencing their precedence in classification or extraction. You can also factor in character-level OCR confidence for more granular control.

• Result Set Shaping:
  Use 'Hit Count Range' to require a specific number of results, 'Distinct' to remove duplicates, 'Sort Order' to control result ordering, and 'Truncate At' to limit the number of returned results.

• Content Removal:
  The 'Subtractor' property lets you specify a Value Extractor to remove unwanted content (such as extraneous text or formatting) from each output value.

Processing Flow

When applied, the options in this class are processed in a defined sequence:

1. Individual Value Adjustments:
   Each result is normalized, cleaned, or transformed according to the configured options.
2. Result Set Operations:
   The set of results is filtered, sorted, deduplicated, or truncated as specified.
3. Type Enforcement:
   If a specific value type is required, results are converted and validated accordingly.

This ensures that the final output is both clean and conforms to the requirements of downstream consumers.

Usage Guidance

• Configure only the options relevant to your scenario; defaults are designed to minimize unnecessary processing.
• For best results, test your configuration with representative data to ensure that normalization, filtering, and confidence adjustments behave as expected.
• For more details on individual options, see the documentation for each property.