Text Match

The 'Prefix Pattern' property allows you to specify a regular expression that must be present immediately before each match found by the Text Match extractor. This enables context-sensitive extraction, ensuring that only values preceded by a specific pattern, label, or structural element are returned.

Purpose

Use this property to restrict matches to those that occur after a particular label, whitespace, line break, or other context. This is especially useful for extracting labeled values, enforcing boundaries, or avoiding false positives in complex documents.

Configuration Guidance

• Enter a regular expression that describes the required prefix. The pattern is applied directly before each match.
• Common examples include whitespace, document labels, line or page boundaries, or exclusion of certain characters.
• Use anchors (such as ^ for start of document, \f for page break, \n for line break) to target structural positions.
• Combine multiple options using the | (OR) operator for flexibility.

Examples

Impact

• Only matches preceded by the specified pattern are included in the output.
• Improves accuracy by filtering out values that do not occur in the desired context.
• Can be used in combination with 'Suffix Pattern' for even more precise extraction.

Usage Scenarios

• Extracting values labeled with a specific phrase (e.g., Invoice No:).
• Restricting matches to the start of a page or line.
• Avoiding matches that are embedded within words or unwanted regions.

> Use diagnostics to review which prefix patterns were applied and to troubleshoot extraction boundaries.

The 'Suffix Pattern' property allows you to specify a regular expression that must be present immediately after each match found by the Text Match extractor. This enables context-sensitive extraction, ensuring that only values followed by a specific pattern, label, or structural element are returned.

Purpose

Use this property to restrict matches to those that occur before a particular label, whitespace, line break, or other context. This is especially useful for extracting values with trailing units, enforcing boundaries, or avoiding false positives in complex documents.

Configuration Guidance

• Enter a regular expression that describes the required suffix. The pattern is applied directly after each match.
• Common examples include whitespace, document labels, line or page boundaries, or exclusion of certain characters.
• Use anchors (such as $ for end of document, \f for page break, \r for line break) to target structural positions.
• Combine multiple options using the | (OR) operator for flexibility.

Examples

Impact

• Only matches followed by the specified pattern are included in the output.
• Improves accuracy by filtering out values that do not occur in the desired context.
• Can be used in combination with 'Prefix Pattern' for even more precise extraction.

Usage Scenarios

• Extracting values with trailing units or labels (e.g., acres).
• Restricting matches to the end of a page or line.
• Avoiding matches that are embedded within words or unwanted regions.

> Use diagnostics to review which suffix patterns were applied and to troubleshoot extraction boundaries.

Overview

The Environment Options class controls how merge variables are resolved and how culture information is applied during extraction. Merge variables are referenced in regular expressions using the syntax @VariableName. These variables can represent lists of values, culture-specific data, or reusable regex fragments.

Built-In Merge Variables

Several built-in merge variables are available (see Variable Providers), exposing culture-specific lists such as @DayNames, @MonthNames, and @CurrencySymbols. For example, the @DayNamesAbbreviated variable expands to a list of day abbreviations appropriate for the current culture:

• For English: Sun|Mon|Tue|Wed|Thu|Fri|Sat
• For French: dim|lun|mar|mer|jeu|ven|sam

These variables adapt automatically to the culture of the document or to a specified override.

Custom Merge Variables

You can define custom merge variables by referencing Lexicons that contain the desired values or key-value pairs. Custom variables are useful for:

• Merging in lists that are difficult to maintain directly in a regex.
• Creating libraries of reusable regex snippets shared by multiple extractors.
• Improving the readability and maintainability of complex patterns.
• Enforcing consistency across multiple extractors.

To define custom variables, use the 'Value Lists' and 'Snippet Libraries' properties to reference appropriate Lexicons. Each lexicon entry becomes a merge variable, which can be injected into your regular expressions using the @VariableName syntax.

Culture Settings

The culture used for variable expansion can be controlled via the 'Culture Override' and 'Culture Scope' properties. By default, variables are generated using the culture of the input document, but you can force a specific culture or restrict processing to certain languages using these options.

This is especially useful when extracting data from mixed-language documents, or when you need to standardize extraction behavior across different locales.

Usage Guidance

• Use built-in variables for common culture-specific lists.
• Reference Lexicons to define custom lists or regex snippets as merge variables.
• Adjust culture settings to match your extraction requirements, especially for multilingual or internationalized content.
• For more details on configuring merge variables and culture options, see the documentation for Lexicons, Variable Providers, and the relevant properties of this class.

The 'Case Sensitive' property controls whether the regular expression pattern, prefix, and suffix matching performed by the Text Match extractor will distinguish between uppercase and lowercase letters.

Purpose

Enable this property when the capitalization of text is meaningful for your extraction scenario, such as distinguishing between proper names, acronyms, or case-specific labels.

Configuration Guidance

• Set to true to require exact case matches (e.g., "Invoice" ≠ "invoice").
• Set to false to allow matches regardless of case (e.g., "Invoice", "INVOICE", and "invoice" are all equivalent).
• Use case-sensitive matching for scenarios where capitalization conveys meaning, such as extracting section headings, entity names, or codes that are case-dependent.
• For most business data, case-insensitive matching (false) is recommended to maximize extraction accuracy.

Impact

• When enabled, only text that matches the exact case of the pattern will be extracted.
• When disabled, matches will be found regardless of case, increasing the number of potential hits.

Examples

> Use diagnostics to verify which matches were found and to troubleshoot case-related extraction issues.

The Text Preprocessor enables advanced manipulation of control characters in a document's text, allowing regular expressions to match or ignore structural elements such as line breaks, paragraph boundaries, page breaks, tabs, and spaces.

Overview

Text preprocessing is performed immediately before extraction, transforming the document's text to improve the accuracy and flexibility of pattern matching. This is especially useful when data values span multiple lines, are separated by large whitespace gaps, or are affected by inconsistent formatting.

Key Features

• Paragraph Marking:
  Detects paragraph boundaries and converts line breaks within paragraphs to spaces, while preserving paragraph-ending breaks. This allows extractors to match values that span multiple lines within a paragraph, without matching across paragraph boundaries. See Paragraph Marker.

• Tab Marking:
  Replaces large horizontal whitespace gaps with TAB characters, making it possible to distinguish between normal spaces and significant gaps in regular expressions. See Horizontal Tab Marker.

• Vertical Tab Marking:
  Converts certain line breaks to vertical tab characters based on vertical spacing, enabling recognition of vertical structure in tabular or multi-column layouts. See Vertical Tab Marker.

• Control Character Ignoring:
  Removes or replaces selected control characters (such as spaces, newlines, form feeds, and carriage returns) according to the 'Ignore Control Characters' setting. This can simplify extraction in documents with inconsistent or excessive whitespace.

Usage Guidance

• Configure the desired preprocessing options by enabling or disabling paragraph, tab, and vertical tab marking, and by selecting which control characters to ignore.
• Preprocessing is typically used in conjunction with regular expression-based extractors, but can benefit any extraction scenario where document structure affects pattern matching.
• For best results, adjust preprocessing settings to match the structure and formatting of your source documents.

Example Scenarios

• Extracting values that span multiple lines within a paragraph:
  Enable paragraph marking to convert internal line breaks to spaces, allowing regular expressions to match values split across lines.

• Distinguishing between normal spaces and large gaps:
  Enable tab marking to insert TAB characters at significant horizontal gaps, so extractors can target fields separated by large whitespace.

• Cleaning up unwanted whitespace or control characters:
  Use the 'Ignore Control Characters' option to remove or replace problematic characters that interfere with extraction.

For more details, see the documentation for Paragraph Marker, Horizontal Tab Marker, and Vertical Tab Marker.

Examples

1. Sample Document

Consider the following sample document.

┌─────────────────────────────────────────────────────────────┐
│                        SAMPLE FORM                          │
├─────────────────────────────────────────────────────────────┤
│ Name:           John Doe                   ID: 12345        │
│ Date of Birth:  01/01/1980                 Status: Active   │
├─────────────────────────────────────────────────────────────┤
│ This is the first paragraph. It explains the purpose of     │
│ the form and the meaning of each field.                     │
│                                                             │
│ Please complete all fields and verify all personal          │
│ information before submitting. Thank you!                   │ 
└─────────────────────────────────────────────────────────────┘

2. Default Control Characters

With no preprocessing options enabled, the document data will look like this. Whitespace gaps, no matter how large, are represented by a single space character. A \r\n pair marks each location where the original document wrapped to the next line.

SAMPLE FORM\r\n
Name: John Doe ID: 12345\r\n
Date of Birth: 01/01/1980 Status: Active\r\n
This is the first paragraph. It explains the purpose of\r\n
the form and the meaning of each field.\r\n
Please complete all fields and verify all personal\r\n
information before submitting. Thank you!\r\n

3. Preprocessed Version

Preprocessing the document with paragraph marking and tab marking will place a tab character '\t' at each large whitespace gap, and replace newline pairs '\r\n' occuring inside a paragraph with a space.

SAMPLE FORM\r\n
Name: John Doe\tID: 12345\r\n
Date of Birth: 01/01/1980\tStatus: Active\r\n
This is the first paragraph. the form and the meaning of each field.\r\n
Please complete all fields and verify all personal information before submitting. Thank you!\r\n

The 'Chunk Size' property enables chunked processing for large documents, allowing the Text Match extractor to break the document into smaller segments and process each chunk separately.

Purpose

Use this property to optimize extraction performance and memory usage when working with documents containing many pages (hundreds or thousands).

Configuration Guidance

• Enter the number of pages per chunk (e.g., 1000).
• If set to a value greater than zero, the extractor will process the document in chunks of the specified size.
• If left at 0, chunked processing is disabled and the entire document is processed as a single unit.
• Choose a chunk size that balances performance and resource usage for your environment.

Impact

• Reduces memory consumption and improves responsiveness for very large documents.
• Extraction results from each chunk are combined into the final result set.
• May affect the ability to match patterns that span chunk boundaries; adjust chunk size as needed.

Usage Scenarios

• Processing scanned books, contracts, or reports with thousands of pages.
• Avoiding out-of-memory errors or slow extraction times on large documents.
• Tuning chunk size to match available system resources and desired throughput.

> Use diagnostics to review chunk boundaries, extraction times, and the number of results produced per chunk.

The Result Filter allows you to configure a set of criteria that each Data Instance must meet to be included in the final result set. Results that do not match the specified conditions are excluded, enabling precise control over which values are retained for downstream processing or export.

Configuration and Usage

• Configure filter properties to target results by page, confidence, length, character composition, geometric region, font size, alignment, and more.
• Multiple filter criteria can be combined; a result must satisfy all enabled filters to be included.
• Use the 'Allowed Pages' property to restrict results to specific pages.
• Set 'Minimum Confidence' to exclude low-confidence results.
• Use 'Length Range' and 'Character Classes' to filter by value length and character makeup.
• The 'Exclusions' property allows you to specify a Value Extractor whose results will be used to exclude overlapping matches.
• Geometric filters (such as 'Region', 'Width Range', 'Height Range', and 'Font Size Range') allow you to target results by their position and size on the page.
• Emphasis filters (such as 'Character Casing', 'Case Similarity', 'Minimum Font Scale', and 'Require Underline') help isolate results with specific visual or typographic emphasis.

Typical Scenarios

• Restricting to a Region:
  Only include results that fall within a specific area of the page, such as a header or footer.
• Filtering by Confidence:
  Exclude results below a certain confidence threshold to improve data quality.
• Targeting Emphasized Text:
  Isolate headings, titles, or other elements with distinctive font size, casing, or underlining.
• Excluding Unwanted Results:
  Use the 'Exclusions' property to remove results that overlap with known false positives or irrelevant content.

Related Types

• Data Instance: The objects being filtered.
• Value Extractor: Used for exclusion filtering.
• Character Class Filter, Logical Rectangle, Logical Range, Horizontal Alignment Settings: Used for specific filter criteria.

For more information, see the documentation for Data Instance, Value Extractor, and Result Filter properties.

The Result Set Options class provides a flexible set of controls for shaping the output of data extraction and classification activities in Grooper. It allows you to define how individual results are adjusted, how the overall result set is filtered or ordered, and how output values are normalized for downstream use.

Overview

Use this class to:

• Normalize extracted values (e.g., unify variations, apply case or word transformations).
• Adjust or override confidence scores for results.
• Remove unwanted content from output values.
• Sort, filter, or deduplicate the result set.
• Enforce minimum/maximum hit counts or truncate the number of results.

These options are commonly configured on Data Fields, Data Types, or other extraction elements to ensure that the output meets business requirements and is ready for validation, export, or further processing.

Key Scenarios

• Value Normalization:
  Use 'Value Override', 'Case Normalization', or 'Word Normalization' to standardize output values, making them consistent for classification or export. For example, you can map multiple document title variations to a single normalized feature, or apply stemming to treat inflected word forms as equivalent.

• Confidence Adjustment:
  The 'Confidence Override' and 'Confidence Multiplier' properties allow you to boost or reduce the confidence of results, influencing their precedence in classification or extraction. You can also factor in character-level OCR confidence for more granular control.

• Result Set Shaping:
  Use 'Hit Count Range' to require a specific number of results, 'Distinct' to remove duplicates, 'Sort Order' to control result ordering, and 'Truncate At' to limit the number of returned results.

• Content Removal:
  The 'Subtractor' property lets you specify a Value Extractor to remove unwanted content (such as extraneous text or formatting) from each output value.

Processing Flow

When applied, the options in this class are processed in a defined sequence:

1. Individual Value Adjustments:
   Each result is normalized, cleaned, or transformed according to the configured options.
2. Result Set Operations:
   The set of results is filtered, sorted, deduplicated, or truncated as specified.
3. Type Enforcement:
   If a specific value type is required, results are converted and validated accordingly.

This ensures that the final output is both clean and conforms to the requirements of downstream consumers.

Usage Guidance

• Configure only the options relevant to your scenario; defaults are designed to minimize unnecessary processing.
• For best results, test your configuration with representative data to ensure that normalization, filtering, and confidence adjustments behave as expected.
• For more details on individual options, see the documentation for each property.