Lexicon

Can be one of the following values:

The LexiconType enumeration determines the structure and interpretation of entries in a Lexicon or LexicalDictionary. Each type defines a different format for the entries and is used for different extraction, lookup, or normalization scenarios.

• Lookup: Key-value pairs for mapping or translation.
• Vocabulary: A simple list of values, one per line.
• Frequency: Key-count pairs for frequency analysis or weighted lookups.

The selected type affects how entries are parsed, stored, and used at runtime.

Can be one of the following types:

Overview

The 'Language' property determines which language the entries in this Lexicon represent. This is especially important when working with multi-language document sets or when building composite Lexicons that include language-specific variants.

Usage

• Set the 'Language' property to the appropriate language when the Lexicon is intended to store entries for a specific language (e.g., English, Spanish, French).
• Leave the 'Language' property blank for a parent Lexicon that includes other Lexicons for different languages. This parent Lexicon acts as a multi-language dictionary.
• When a Lexicon is included in another Lexicon (using the 'Included Lexicons' property), the 'Language' property helps Grooper select the correct set of entries at runtime based on the language of the document being processed.

Multi-Language Scenarios

When processing documents in multiple languages, you can create a parent Lexicon with no language specified and include child Lexicons, each with a different 'Language' value. At runtime, Grooper will automatically select the correct entries based on the detected language of the document.

For example, to extract an "invoice date" label in various languages:

• Create separate Lexicons for each language (e.g., one for Spanish, one for French, etc.), each with the 'Language' property set accordingly.
• Create a parent Lexicon with no language specified and include the language-specific Lexicons using the 'Included Lexicons' property.
• Assign the parent Lexicon to the relevant extraction or validation property (such as 'List Values' on a Data Field).

At runtime, Grooper will use the document's language to select the appropriate entries from the included Lexicons.

> Note:
> - A multi-language Lexicon should define no local entries and have no language specified. > - If the 'Language' property is not set, the Lexicon is considered language-neutral.

Overview

The 'Case Sensitive' property determines if the entries in this Lexicon will be matched using case-sensitive logic. When enabled, the exact casing of each character in the entry must match the input text for a successful match.

Usage

• Set 'Case Sensitive' to true if your data requires strict matching of uppercase and lowercase letters (for example, distinguishing between "ABC" and "abc").
• Set to false to allow case-insensitive matching, so "Invoice", "INVOICE", and "invoice" are all considered equivalent.

Practical Guidance

• For most business data (such as names, codes, or labels), case-insensitive matching (false) is recommended to improve extraction accuracy and user experience.
• Use case-sensitive matching (true) only when the distinction between uppercase and lowercase is meaningful for your use case.
• This property affects all lookups, validations, and extractions that use this Lexicon, including those performed by Data Fields, Data Tables, and custom extractors.

> Note:
> Changing this property will clear the internal cache and may affect extraction results. Review your configuration and test after making changes.

Overview

The 'Regular Expression Syntax' property determines if the entries in this Lexicon are treated as regular expressions or as plain text.

• When set to false, special characters (such as ., -, +, etc.) are automatically escaped, so entries are matched as literal text.
• When set to true, entries are passed through as-is and evaluated using .NET regular expression rules, allowing for advanced pattern matching.

Usage

• Enable this property (true) if your Lexicon contains regular expression patterns and you want to leverage pattern-based matching.
• Leave this property disabled (false) for standard word or phrase lists, where entries should be matched exactly as written.

Practical Guidance

• Use regular expressions for advanced scenarios, such as matching variable formats, optional text, or complex patterns.
• Be cautious: invalid or overly broad regular expressions can impact extraction performance or accuracy.
• When in doubt, keep this property disabled to ensure simple, literal matching.

Overview

The 'Included Lexicons' property allows you to build composite Lexicons by referencing other Lexicons. All entries from the included Lexicons are merged into this Lexicon at runtime.

Usage

• Use this property to create reusable dictionaries by combining multiple Lexicons into a single resource.
• For multi-language scenarios, include language-specific Lexicons in a parent Lexicon with no language specified. Grooper will select the appropriate entries based on the document's language.
• Entries from included Lexicons are combined with any local entries and database-sourced entries in this Lexicon.

Best Practices

• Avoid circular references (e.g., Lexicon A includes Lexicon B, and Lexicon B includes Lexicon A).
• Ensure that all included Lexicons use the same 'Lexicon Type' as the parent Lexicon.
• Use descriptive names and documentation for included Lexicons to clarify their purpose.
• Regularly review included Lexicons to ensure they remain relevant and accurate.

> Note:
> Changes to any included Lexicon will automatically update the composite Lexicon at runtime.

Overview

The Abbreviations property allows you to define a set of search-and-replace rules that are automatically applied to lexicon entries to generate alternate forms and permutations. This is useful for handling common abbreviations, alternate spellings, or variations in entity names, improving extraction accuracy and recall.

Format

• Each abbreviation is specified as a key-value pair in the form:
  value=replacement
• Enter one abbreviation per line.
• For visual clarity, spaces may be represented using the underscore character (_).
• All search and replace operations are case-sensitive.
• If a replacement value ends with .?, permutations will be generated with and without the ending period.

How It Works

• At runtime, for each abbreviation rule, the system scans all lexicon entries for occurrences of the value (the text to find).
• For each match, a new entry is generated by replacing value with replacement.
• If .? is used at the end of the replacement, both forms (with and without the period) are generated.
• All generated permutations are added to the lexicon, increasing the number of ways a value can be matched.

Example

Suppose your lexicon contains the following entries:

Acme Energy Corporation
Smith Exploration Company
ABC Exploration & Production Company, Inc.

To account for various permutations of these entity names, you could use the following abbreviation list:

Corporation=Corp
Company=Co
&=and

At runtime, the lexicon will contain the following entries:

Acme Energy Corporation
Acme Energy Corp
Smith Exploration Company
Smith Exploration Co
ABC Exploration & Production Company, Inc.
ABC Exploration and Production Company, Inc.
ABC Exploration & Production Co, Inc.
ABC Exploration and Production Co, Inc.

Additional Notes

• For visual clarity, spaces may be represented using the underscore character (_) in the abbreviation list.
• All search and replace operations are case-sensitive. For example, "Company" and "company" are treated as different values.
• Abbreviation replacement values ending in .? will generate permutations with and without the ending period.

Best Practices

• Use abbreviations to account for common variations, such as "Corporation" vs. "Corp", or "&" vs. "and".
• Avoid overly broad or ambiguous abbreviations that could generate unintended permutations.
• Review the expanded list of entries to ensure accuracy and avoid duplicates or conflicts.
• Abbreviations are applied globally to all entries in the lexicon and are not context-aware.
• Only simple string replacements are supported; regular expressions are not used.

Overview

The 'Connection' property allows this Lexicon to retrieve its entries dynamically from an external database. When a Data Connection is specified, the SQL query defined in the 'Query' property will be executed, and the results will be used to populate the Lexicon.

Usage

• Select a Data Connection that defines how to connect to your target database (such as SQL Server, Oracle, or another supported provider).
• Configure the 'Query' property with a valid SQL statement to retrieve the desired entries.
• Each row returned by the query will be added as an entry in the Lexicon. If the query returns two columns, the first is used as the key and the second as the value.

Practical Guidance

• Use this property to keep your Lexicon synchronized with external systems, such as ERP, CRM, or master data repositories.
• Ensure the Data Connection is properly configured and tested before relying on database-driven lexicons in production.
• If the 'Connection' property is not set, the Lexicon will use only local entries, included Lexicons, and abbreviations.

Example

1. Create a Data Connection to your database.
2. Set the 'Connection' property to reference this Data Connection.
3. Enter a SQL query in the 'Query' property, such as:

SELECT product_code, product_description FROM Products WHERE active = 1

4. The Lexicon will be populated with all active product codes and their descriptions.

Overview

The Query property defines the SQL statement that will be executed against the configured Data Connection. The results of this query are used to populate the entries in this Lexicon.

• The query should return either one or two columns:
  • If one column is returned, it is used as the key for each entry.
  • If two columns are returned, the first is used as the key and the second as the value.

Usage

• Write a SQL SELECT statement that retrieves the desired data from your database.
• Use column aliases or expressions as needed to format the results.
• The query is executed each time the Lexicon is loaded or refreshed.

Practical Guidance

• Ensure the query is efficient and returns only the necessary data.
• Test your query in your database environment before using it in Grooper.
• If the query is invalid or the Data Connection is not set, the Lexicon will not be populated from the database.

Intellisense and Code Editor Features

The query editor for this property provides advanced intellisense features to assist with SQL editing:

• SQL Syntax Highlighting: The editor highlights SQL keywords, functions, and identifiers for improved readability.
• Code Completion: As you type, the editor suggests:
  • Table names from the connected database.
  • Column names for tables referenced in the current query context.
  • SQL keywords (such as SELECT, FROM, WHERE, ORDER BY, etc.).
  • SQL functions (such as COUNT, SUM, MIN, MAX, etc.).
  • Aliases defined in your query.
• Function Parameter Hints: When typing SQL functions, parameter information and overloads are displayed where available.
• Keyboard Shortcuts:
  • Press Ctrl + Shift + Space to manually trigger intellisense suggestions.
  • Use Up/Down arrows to navigate the suggestion list.
  • Press Enter to accept a suggestion.
  • Press Escape to close the suggestion list.

Examples

1. Single-column list (simple lookup):

SELECT po_number FROM PurchaseOrder WHERE po_status='open'

Populates the Lexicon with a list of open purchase order numbers.

2. Key-value pairs:

SELECT po_number, vendor_id FROM PurchaseOrder WHERE po_status='open'

Populates the Lexicon with key-value pairs mapping open PO numbers to vendor IDs.

3. Concatenated values:

SELECT student_id, first_name + ' ' + last_name FROM Students WHERE active=1

Populates the Lexicon with student IDs as keys and full names as values for active students.

4. Using column aliases:

SELECT product_code AS Code, product_name AS Name FROM Products WHERE active=1

Populates the Lexicon with product codes and names for active products.

5. Filtering and ordering:

SELECT department_id, department_name FROM Departments WHERE active=1 ORDER BY department_name

Populates the Lexicon with active departments, sorted by name.

> Note:
> The query is executed as-is against the selected Data Connection. Ensure that your query is valid for the target database type.