List Match

The 'Local Entries' property allows you to define a custom list of search terms directly within the extractor. Each entry should be placed on a separate line. These terms are used to identify and extract matching text segments from the document. Local entries are ideal for field-specific lists or when you need to quickly configure a set of terms without referencing external lexicons.

• Use this property for small, targeted lists of terms, such as field labels, entity names, or classification features.
• Entries are matched exactly as written, unless regular expression syntax is enabled in the vocabulary.
• For advanced scenarios, you can combine local entries with external lexicons or include values from the parent field.

Example: billed amount amount charged total charges

Regularly review and update local entries to ensure all relevant term variants are included. For large or shared lists, consider using an external lexicon for easier management and reuse.

The 'Vocabulary' property defines the set of terms, phrases, or key-value pairs that the extractor will use to identify matches in the document. You can configure vocabulary entries locally, reference external lexicons, or merge values from the parent field. Each entry is treated as a distinct search term, and when a match is found, the value is extracted.

• For direct matching, add each term on a separate line. For translation, use the key=value format to map input values to normalized outputs.
• Vocabulary can be configured as a lookup lexicon to support translation and normalization of matched values.
• Use the 'Include Field List Values' property to merge values from the parent field's value list, supporting dynamic or context-sensitive lookups.
• The 'Type' property controls the entry format: Vocabulary (simple list), Lookup (key-value pairs), or Frequency.

Example: IBM International Business Machines=IBM IBM Corporation=IBM Dell Dell Corporation=Dell

For best results, ensure all expected term variants and translations are included. Regularly review and update the vocabulary to maintain extraction accuracy and consistency.

The 'Prefix Pattern' property allows you to specify a regular expression that must be present immediately before each match found by the Text Match extractor. This enables context-sensitive extraction, ensuring that only values preceded by a specific pattern, label, or structural element are returned.

Purpose

Use this property to restrict matches to those that occur after a particular label, whitespace, line break, or other context. This is especially useful for extracting labeled values, enforcing boundaries, or avoiding false positives in complex documents.

Configuration Guidance

• Enter a regular expression that describes the required prefix. The pattern is applied directly before each match.
• Common examples include whitespace, document labels, line or page boundaries, or exclusion of certain characters.
• Use anchors (such as ^ for start of document, \f for page break, \n for line break) to target structural positions.
• Combine multiple options using the | (OR) operator for flexibility.

Examples

Impact

• Only matches preceded by the specified pattern are included in the output.
• Improves accuracy by filtering out values that do not occur in the desired context.
• Can be used in combination with 'Suffix Pattern' for even more precise extraction.

Usage Scenarios

• Extracting values labeled with a specific phrase (e.g., Invoice No:).
• Restricting matches to the start of a page or line.
• Avoiding matches that are embedded within words or unwanted regions.

> Use diagnostics to review which prefix patterns were applied and to troubleshoot extraction boundaries.

The 'Suffix Pattern' property allows you to specify a regular expression that must be present immediately after each match found by the Text Match extractor. This enables context-sensitive extraction, ensuring that only values followed by a specific pattern, label, or structural element are returned.

Purpose

Use this property to restrict matches to those that occur before a particular label, whitespace, line break, or other context. This is especially useful for extracting values with trailing units, enforcing boundaries, or avoiding false positives in complex documents.

Configuration Guidance

• Enter a regular expression that describes the required suffix. The pattern is applied directly after each match.
• Common examples include whitespace, document labels, line or page boundaries, or exclusion of certain characters.
• Use anchors (such as $ for end of document, \f for page break, \r for line break) to target structural positions.
• Combine multiple options using the | (OR) operator for flexibility.

Examples

Impact

• Only matches followed by the specified pattern are included in the output.
• Improves accuracy by filtering out values that do not occur in the desired context.
• Can be used in combination with 'Prefix Pattern' for even more precise extraction.

Usage Scenarios

• Extracting values with trailing units or labels (e.g., acres).
• Restricting matches to the end of a page or line.
• Avoiding matches that are embedded within words or unwanted regions.

> Use diagnostics to review which suffix patterns were applied and to troubleshoot extraction boundaries.

Overview

The Environment Options class controls how merge variables are resolved and how culture information is applied during extraction. Merge variables are referenced in regular expressions using the syntax @VariableName. These variables can represent lists of values, culture-specific data, or reusable regex fragments.

Built-In Merge Variables

Several built-in merge variables are available (see Variable Providers), exposing culture-specific lists such as @DayNames, @MonthNames, and @CurrencySymbols. For example, the @DayNamesAbbreviated variable expands to a list of day abbreviations appropriate for the current culture:

• For English: Sun|Mon|Tue|Wed|Thu|Fri|Sat
• For French: dim|lun|mar|mer|jeu|ven|sam

These variables adapt automatically to the culture of the document or to a specified override.

Custom Merge Variables

You can define custom merge variables by referencing Lexicons that contain the desired values or key-value pairs. Custom variables are useful for:

• Merging in lists that are difficult to maintain directly in a regex.
• Creating libraries of reusable regex snippets shared by multiple extractors.
• Improving the readability and maintainability of complex patterns.
• Enforcing consistency across multiple extractors.

To define custom variables, use the 'Value Lists' and 'Snippet Libraries' properties to reference appropriate Lexicons. Each lexicon entry becomes a merge variable, which can be injected into your regular expressions using the @VariableName syntax.

Culture Settings

The culture used for variable expansion can be controlled via the 'Culture Override' and 'Culture Scope' properties. By default, variables are generated using the culture of the input document, but you can force a specific culture or restrict processing to certain languages using these options.

This is especially useful when extracting data from mixed-language documents, or when you need to standardize extraction behavior across different locales.

Usage Guidance

• Use built-in variables for common culture-specific lists.
• Reference Lexicons to define custom lists or regex snippets as merge variables.
• Adjust culture settings to match your extraction requirements, especially for multilingual or internationalized content.
• For more details on configuring merge variables and culture options, see the documentation for Lexicons, Variable Providers, and the relevant properties of this class.

The 'Case Sensitive' property controls whether the regular expression pattern, prefix, and suffix matching performed by the Text Match extractor will distinguish between uppercase and lowercase letters.

Purpose

Enable this property when the capitalization of text is meaningful for your extraction scenario, such as distinguishing between proper names, acronyms, or case-specific labels.

Configuration Guidance

• Set to true to require exact case matches (e.g., "Invoice" ≠ "invoice").
• Set to false to allow matches regardless of case (e.g., "Invoice", "INVOICE", and "invoice" are all equivalent).
• Use case-sensitive matching for scenarios where capitalization conveys meaning, such as extracting section headings, entity names, or codes that are case-dependent.
• For most business data, case-insensitive matching (false) is recommended to maximize extraction accuracy.

Impact

• When enabled, only text that matches the exact case of the pattern will be extracted.
• When disabled, matches will be found regardless of case, increasing the number of potential hits.

Examples

> Use diagnostics to verify which matches were found and to troubleshoot case-related extraction issues.

The Text Preprocessor enables advanced manipulation of control characters in a document's text, allowing regular expressions to match or ignore structural elements such as line breaks, paragraph boundaries, page breaks, tabs, and spaces.

Overview

Text preprocessing is performed immediately before extraction, transforming the document's text to improve the accuracy and flexibility of pattern matching. This is especially useful when data values span multiple lines, are separated by large whitespace gaps, or are affected by inconsistent formatting.

Key Features

• Paragraph Marking:
  Detects paragraph boundaries and converts line breaks within paragraphs to spaces, while preserving paragraph-ending breaks. This allows extractors to match values that span multiple lines within a paragraph, without matching across paragraph boundaries. See Paragraph Marker.

• Tab Marking:
  Replaces large horizontal whitespace gaps with TAB characters, making it possible to distinguish between normal spaces and significant gaps in regular expressions. See Horizontal Tab Marker.

• Vertical Tab Marking:
  Converts certain line breaks to vertical tab characters based on vertical spacing, enabling recognition of vertical structure in tabular or multi-column layouts. See Vertical Tab Marker.

• Control Character Ignoring:
  Removes or replaces selected control characters (such as spaces, newlines, form feeds, and carriage returns) according to the 'Ignore Control Characters' setting. This can simplify extraction in documents with inconsistent or excessive whitespace.

Usage Guidance

• Configure the desired preprocessing options by enabling or disabling paragraph, tab, and vertical tab marking, and by selecting which control characters to ignore.
• Preprocessing is typically used in conjunction with regular expression-based extractors, but can benefit any extraction scenario where document structure affects pattern matching.
• For best results, adjust preprocessing settings to match the structure and formatting of your source documents.

Example Scenarios

• Extracting values that span multiple lines within a paragraph:
  Enable paragraph marking to convert internal line breaks to spaces, allowing regular expressions to match values split across lines.

• Distinguishing between normal spaces and large gaps:
  Enable tab marking to insert TAB characters at significant horizontal gaps, so extractors can target fields separated by large whitespace.

• Cleaning up unwanted whitespace or control characters:
  Use the 'Ignore Control Characters' option to remove or replace problematic characters that interfere with extraction.

For more details, see the documentation for Paragraph Marker, Horizontal Tab Marker, and Vertical Tab Marker.

Examples

1. Sample Document

Consider the following sample document.

┌─────────────────────────────────────────────────────────────┐
│                        SAMPLE FORM                          │
├─────────────────────────────────────────────────────────────┤
│ Name:           John Doe                   ID: 12345        │
│ Date of Birth:  01/01/1980                 Status: Active   │
├─────────────────────────────────────────────────────────────┤
│ This is the first paragraph. It explains the purpose of     │
│ the form and the meaning of each field.                     │
│                                                             │
│ Please complete all fields and verify all personal          │
│ information before submitting. Thank you!                   │ 
└─────────────────────────────────────────────────────────────┘

2. Default Control Characters

With no preprocessing options enabled, the document data will look like this. Whitespace gaps, no matter how large, are represented by a single space character. A \r\n pair marks each location where the original document wrapped to the next line.

SAMPLE FORM\r\n
Name: John Doe ID: 12345\r\n
Date of Birth: 01/01/1980 Status: Active\r\n
This is the first paragraph. It explains the purpose of\r\n
the form and the meaning of each field.\r\n
Please complete all fields and verify all personal\r\n
information before submitting. Thank you!\r\n

3. Preprocessed Version

Preprocessing the document with paragraph marking and tab marking will place a tab character '\t' at each large whitespace gap, and replace newline pairs '\r\n' occuring inside a paragraph with a space.

SAMPLE FORM\r\n
Name: John Doe\tID: 12345\r\n
Date of Birth: 01/01/1980\tStatus: Active\r\n
This is the first paragraph. the form and the meaning of each field.\r\n
Please complete all fields and verify all personal information before submitting. Thank you!\r\n

Can be one of the following types:

Unlike a normal regular expression, which finds values exactly matching the pattern, a fuzzy regular expression (FRX) finds values which match the pattern to a specific degree of similarity, and automatically repairs the output value whenever possible.

When using FRX mode, there are a few limitations on regular expression syntax and some performance implications which need to be considered. These are outlined below.

Regular Expression Syntax

Fuzzy regular expressions support most of the syntax and features of standard regular expressions, with a handful of exceptions noted below. The following regular expression features are NOT supported in FuzzyRegEx mode:

• Quantifiers : + and *. Also, the 'few times as possible' construct (i.e. \w*?) is not supported.
• Character Escapes : \a \b \e \nnn \cX \cx \unnnn.
• Character Classes : \p{name} abd \P{name}
• Grouping Constructs : Only basic named and unnamed group constructs are supported
• Other: Anchors other than ^ and $, Backreference Constructs, and Alternation Constructs are unsupported.

FRX also supports an option which is unavailable in normal regular expressions. (?r) will turn on required mode, and (?-r) will turn it off. At the start of an FRX, required mode always defaults to off. Once turned on, required mode will stay on until it is turned off. This mechanism can be used, for example, to require the start of a new line. The syntax to accomplish this would be be (?r)\n(?-r).

Performance Considerations

The processing time for an FRX is considerably longer than a normal regular expression, particularly for complex regular expressions. The execution time is proportional to the perplexity of the regular expression - which measures the number of possible permutations in the pattern. For example:

• A{1,2}B{1,2} has a perplexity of 2 * 2 = 4 (i.e. it could match AB, AAB, ABB, or AABB).
• A{1,2}B{1,2}C{1,2} has a perplexity of 2 * 2 * 2 = 8.
• A{1,5}B{1,5}C{1,5} has a perplexity of 5 * 5 * 5 = 125.
• [0-9]{4} (miles|kilometers) has a perplexity of 1 * 2 = 2.
• [0-9]{1,5} (miles|kilometers) has a perplexity of 5 * 2 = 10.

There is a point at which perplexity gets so high that fuzzy matching is computationally impractical. As such, FRX is not suitable for every extraction task, and should be used with caution.

Can be one of the following types:

The Constrained Wrap Options class enables extraction of values that span line breaks inside a defined region, such as a table cell or boxed area. This is useful for scenarios where data (like numbers, dates, or labels) may be split across lines due to formatting or limited space.

For example, enabling this option allows a pattern like \d+ acres to match "340 acres" in the following document, even though the value wraps across two lines:

Table headers also frequently wrap text inside a box, as shown below:

How It Works

When enabled, this option combines the text content from a region (such as a table cell) into a single string, replacing line breaks with spaces. Extraction patterns are then applied to this combined text, allowing matches that span multiple lines.

You can further constrain which regions are considered by specifying minimum and maximum values for width, height, character count, and line count using the properties below.

Usage Guidance

• Use Constrained Wrap Options when extracting data from documents where values may be split across lines within a bounded area.
• Adjust the 'Width Range', 'Height Range', 'Character Count', and 'Line Count' properties to target only regions of interest and avoid false positives.
• This option is especially useful for extracting data from table headers, boxed fields, or any layout where text wrapping is common.

Can be one of the following types:

The Vertical Wrap Detection class enables extraction of search terms or values that are split across multiple lines in a vertical arrangement. This is especially useful for multi-word labels or values that may be wrapped due to document formatting, such as table headers or stacked field names.

For example, this option allows the extractor to find the search term "Purchase Order Number" in any of the following layouts:

• The extractor groups vertically adjacent text segments that meet the alignment and spacing criteria defined by the properties below.
• The combined text is compared to the set of search terms or values, allowing matches that span multiple lines.
• You can control which segments are grouped by adjusting the maximum line spacing, alignment, and whether horizontal or vertical rules are allowed between lines.

Usage Guidance

• Use Vertical Wrap Detection when extracting multi-line labels, table headers, or any field that may be split vertically.
• Adjust the 'Maximum Line Spacing', 'Alignment', and 'Alignment Tolerance' properties to fine-tune which lines are grouped.
• Enable or disable 'Allow Horizontal Rule' and 'Allow Vertical Rule' to control how lines separated by graphical elements are handled.

The 'Chunk Size' property enables chunked processing for large documents, allowing the Text Match extractor to break the document into smaller segments and process each chunk separately.

Purpose

Use this property to optimize extraction performance and memory usage when working with documents containing many pages (hundreds or thousands).

Configuration Guidance

• Enter the number of pages per chunk (e.g., 1000).
• If set to a value greater than zero, the extractor will process the document in chunks of the specified size.
• If left at 0, chunked processing is disabled and the entire document is processed as a single unit.
• Choose a chunk size that balances performance and resource usage for your environment.

Impact

• Reduces memory consumption and improves responsiveness for very large documents.
• Extraction results from each chunk are combined into the final result set.
• May affect the ability to match patterns that span chunk boundaries; adjust chunk size as needed.

Usage Scenarios

• Processing scanned books, contracts, or reports with thousands of pages.
• Avoiding out-of-memory errors or slow extraction times on large documents.
• Tuning chunk size to match available system resources and desired throughput.

> Use diagnostics to review chunk boundaries, extraction times, and the number of results produced per chunk.

The 'Use List Case' property determines how the character casing of extracted values is handled. When set to false, output values will use the casing found in the document. When set to true, output values will use the casing specified in the vocabulary entry.

• Use false to preserve the original casing from the document, which may be important for downstream processes or when case conveys meaning.
• Use true to normalize output values to a consistent format, as defined in the vocabulary. This is useful for standardizing data for export, reporting, or classification.
• This property is especially helpful when matching terms that may appear in various cases (e.g., "IBM", "ibm", "Ibm").

Example: If the document contains "INTERNATIONAL BUSINESS MACHINES" and the vocabulary entry is "International Business Machines", enabling 'Use List Case' will output "International Business Machines" regardless of the document's casing.

Choose the setting that best fits your normalization and data quality requirements.

The 'Translate' property allows you to normalize or replace matched values with standardized outputs, as defined in the vocabulary. To use translation, configure the vocabulary as a lookup lexicon and enter key-value pairs using the = symbol as a delimiter.

• When enabled, matched values are replaced with their corresponding replacement value from the vocabulary.
• If an entry does not specify a replacement value, the matched value is output as-is.
• Use translation to map multiple variants of a term to a single normalized value, improving consistency for downstream processing, reporting, or classification.

Example: International Business Machines=IBM IBM Corporation=IBM IRS=Internal Revenue Service

If the extractor matches "IBM Corporation" in the document, the output value will be "IBM".

For best results, ensure all key-value pairs are correctly defined in the vocabulary. Review extraction results to verify that translation is working as intended and adjust entries as needed.

The Result Filter allows you to configure a set of criteria that each Data Instance must meet to be included in the final result set. Results that do not match the specified conditions are excluded, enabling precise control over which values are retained for downstream processing or export.

Configuration and Usage

• Configure filter properties to target results by page, confidence, length, character composition, geometric region, font size, alignment, and more.
• Multiple filter criteria can be combined; a result must satisfy all enabled filters to be included.
• Use the 'Allowed Pages' property to restrict results to specific pages.
• Set 'Minimum Confidence' to exclude low-confidence results.
• Use 'Length Range' and 'Character Classes' to filter by value length and character makeup.
• The 'Exclusions' property allows you to specify a Value Extractor whose results will be used to exclude overlapping matches.
• Geometric filters (such as 'Region', 'Width Range', 'Height Range', and 'Font Size Range') allow you to target results by their position and size on the page.
• Emphasis filters (such as 'Character Casing', 'Case Similarity', 'Minimum Font Scale', and 'Require Underline') help isolate results with specific visual or typographic emphasis.

Typical Scenarios

• Restricting to a Region:
  Only include results that fall within a specific area of the page, such as a header or footer.
• Filtering by Confidence:
  Exclude results below a certain confidence threshold to improve data quality.
• Targeting Emphasized Text:
  Isolate headings, titles, or other elements with distinctive font size, casing, or underlining.
• Excluding Unwanted Results:
  Use the 'Exclusions' property to remove results that overlap with known false positives or irrelevant content.

Related Types

• Data Instance: The objects being filtered.
• Value Extractor: Used for exclusion filtering.
• Character Class Filter, Logical Rectangle, Logical Range, Horizontal Alignment Settings: Used for specific filter criteria.

For more information, see the documentation for Data Instance, Value Extractor, and Result Filter properties.

The Result Set Options class provides a flexible set of controls for shaping the output of data extraction and classification activities in Grooper. It allows you to define how individual results are adjusted, how the overall result set is filtered or ordered, and how output values are normalized for downstream use.

Overview

Use this class to:

• Normalize extracted values (e.g., unify variations, apply case or word transformations).
• Adjust or override confidence scores for results.
• Remove unwanted content from output values.
• Sort, filter, or deduplicate the result set.
• Enforce minimum/maximum hit counts or truncate the number of results.

These options are commonly configured on Data Fields, Data Types, or other extraction elements to ensure that the output meets business requirements and is ready for validation, export, or further processing.

Key Scenarios

• Value Normalization:
  Use 'Value Override', 'Case Normalization', or 'Word Normalization' to standardize output values, making them consistent for classification or export. For example, you can map multiple document title variations to a single normalized feature, or apply stemming to treat inflected word forms as equivalent.

• Confidence Adjustment:
  The 'Confidence Override' and 'Confidence Multiplier' properties allow you to boost or reduce the confidence of results, influencing their precedence in classification or extraction. You can also factor in character-level OCR confidence for more granular control.

• Result Set Shaping:
  Use 'Hit Count Range' to require a specific number of results, 'Distinct' to remove duplicates, 'Sort Order' to control result ordering, and 'Truncate At' to limit the number of returned results.

• Content Removal:
  The 'Subtractor' property lets you specify a Value Extractor to remove unwanted content (such as extraneous text or formatting) from each output value.

Processing Flow

When applied, the options in this class are processed in a defined sequence:

1. Individual Value Adjustments:
   Each result is normalized, cleaned, or transformed according to the configured options.
2. Result Set Operations:
   The set of results is filtered, sorted, deduplicated, or truncated as specified.
3. Type Enforcement:
   If a specific value type is required, results are converted and validated accordingly.

This ensures that the final output is both clean and conforms to the requirements of downstream consumers.

Usage Guidance

• Configure only the options relevant to your scenario; defaults are designed to minimize unnecessary processing.
• For best results, test your configuration with representative data to ensure that normalization, filtering, and confidence adjustments behave as expected.
• For more details on individual options, see the documentation for each property.