Lexical

---

Can be one of the following types:

Value	Description
Reference	Delegates extraction to another configured extractor, enabling reuse and centralization of extraction logic.
AI Column Extractor	Extracts structured content from documents with two-column layouts.
Ask AI	Executes a completion using a large language model (LLM) and returns one hit for each choice in the response.
Detect Signature	Detects a signature within a specified region of a document page by measuring the percentage of the area that is filled.
Entity Recognition	Identifies and categorizes entities such as people, organizations, locations, and quantities in unstructured text.
Field Match	Matches the value stored in a previously-extracted field or table column.
Find Barcode	Searches for barcode values in document Layout Data previously detected during image processing.
Highlight Zone	Defines a region of a document to be visually highlighted, without extracting any data values.
Key Phrase Extraction	Identifies key concepts and topics in text using Azure AI Language key phrase extraction.
Label Match	Matches a list of one or more label values, using matching options defined by a Labeling Behavior.
Labeled OMR	Reads a group of one or more checkboxes located nearby text labels.
Labeled Value	Extracts a field presented as a label-value pair within a document, associating labels and values based on their spatial relationship.
List Match	Extracts values from document text that match any entry in a list of search terms.
Ordered OMR	Reads one or more checkboxes with a consistent order of appearance inside a rectangular region.
Pattern Match	Extracts values from document text that match a specified regular expression pattern.
Pii Entity Recognition	Identifies, categorizes, and redacts sensitive information (PII) in unstructured text using Azure AI Language Services.
Query HTML	Extracts values from an HTML document using a CSS or XPath selector.
Query XML	Extracts values from XML documents using XPATH queries, enabling structured data extraction from XML content in Grooper.
Read Barcode	Extracts barcode values from document images using configurable barcode recognition.
Read Metadata	Reads a metadata value from a document by accessing a property on an attachment or content link.
Read Zone	Extracts text content from a specified rectangular region (zone) of a document.
Select Page	Selects and outputs the full content of one or more pages from a document, based on page number and/or content criteria.
Word Match	Extracts individual words and multi-word phrases (N-grams) from document text for use in classification, data extraction, and normalization.
Zonal OMR	Reads one or more checkboxes using manually-configured zones.

The 'Text Feature Extractor' defines how Grooper analyzes the textual content of documents to extract features for classification. Each value matched by this extractor is considered a feature, and the set of features is used to compare documents to trained Document Types.

How It Works

• During training, the extractor is applied to sample documents to build a feature profile for each Document Type.
• During classification, the extractor is applied to the document being classified, and the resulting features are compared to the trained profiles.
• Features can be individual words, n-grams (sequences of words), or custom tokens representing patterns, data types, or semantic elements.

Feature Types

• Unigrams (Single Words):
  The most common feature type. Each word is treated as a feature.
  • Tip: Use stop word filtering to remove common words (e.g., "and", "the") that do not help distinguish document types.
  • Tip: Enable stemming to reduce words to their root form (e.g., "insured", "insuring" → "insur").
• n-Grams (Bigrams, Trigrams, etc.):
  Sequences of adjacent words. Useful for capturing phrases or context (e.g., "oil well").
  • Note: n-gram extraction increases processing time and should be used only when needed.
• Custom Tokens:
  Extractors can match patterns such as VIN numbers, dates, or other entities and return a token (e.g., "VIN_Number") as a feature.

Configuration Guidance

• Assign a Value Extractor that matches the most distinctive and relevant features for your document set.
• Use built-in extractors for common patterns, or create custom extractors for specialized needs.
• Combine multiple extraction strategies (e.g., words + tokens) for best results.
• For best accuracy, filter out noise (stop words) and normalize features (stemming, case normalization).

Practical Examples

• Simple Word Extraction:
  Use a pattern match extractor to find all words, excluding stop words.
• Phrase Extraction:
  Configure the extractor to return bigrams or trigrams for documents where phrases are more distinctive than single words.
• Entity Extraction:
  Use a regular expression to match VIN numbers, invoice numbers, or other identifiers, and return a token for each match.

Best Practices

• For small, structured document sets, focus on unique words or phrases.
• For large or unstructured sets, use a combination of word, phrase, and entity extraction.
• Regularly review and refine the extractor as your document set evolves.

For more information, see the documentation for Value Extractor, Document Type, and Content Model.

---

The 'Image Feature Extractor' allows Grooper to analyze visual content (such as graphics, logos, or layout patterns) in addition to text.

• Assign an IP Profile to enable image-based feature extraction.
• Image features are combined with text features to improve classification accuracy, especially for visually distinctive documents.
• If not set, only text features are used.

For more information, see the documentation for IP Profile and Content Model.

---

Can be one of the following types:

Value	Description
Reference	Delegates extraction to another configured extractor, enabling reuse and centralization of extraction logic.
AI Column Extractor	Extracts structured content from documents with two-column layouts.
Ask AI	Executes a completion using a large language model (LLM) and returns one hit for each choice in the response.
Detect Signature	Detects a signature within a specified region of a document page by measuring the percentage of the area that is filled.
Entity Recognition	Identifies and categorizes entities such as people, organizations, locations, and quantities in unstructured text.
Field Match	Matches the value stored in a previously-extracted field or table column.
Find Barcode	Searches for barcode values in document Layout Data previously detected during image processing.
Highlight Zone	Defines a region of a document to be visually highlighted, without extracting any data values.
Key Phrase Extraction	Identifies key concepts and topics in text using Azure AI Language key phrase extraction.
Label Match	Matches a list of one or more label values, using matching options defined by a Labeling Behavior.
Labeled OMR	Reads a group of one or more checkboxes located nearby text labels.
Labeled Value	Extracts a field presented as a label-value pair within a document, associating labels and values based on their spatial relationship.
List Match	Extracts values from document text that match any entry in a list of search terms.
Ordered OMR	Reads one or more checkboxes with a consistent order of appearance inside a rectangular region.
Pattern Match	Extracts values from document text that match a specified regular expression pattern.
Pii Entity Recognition	Identifies, categorizes, and redacts sensitive information (PII) in unstructured text using Azure AI Language Services.
Query HTML	Extracts values from an HTML document using a CSS or XPath selector.
Query XML	Extracts values from XML documents using XPATH queries, enabling structured data extraction from XML content in Grooper.
Read Barcode	Extracts barcode values from document images using configurable barcode recognition.
Read Metadata	Reads a metadata value from a document by accessing a property on an attachment or content link.
Read Zone	Extracts text content from a specified rectangular region (zone) of a document.
Select Page	Selects and outputs the full content of one or more pages from a document, based on page number and/or content criteria.
Word Match	Extracts individual words and multi-word phrases (N-grams) from document text for use in classification, data extraction, and normalization.
Zonal OMR	Reads one or more checkboxes using manually-configured zones.

The provided Extractor must define and output a group named 'PageNo', and optionally may define and output a group named 'PageCount'. For example, if the document set contains page numbering like 'Page 1 of 4', the following pattern would generate the required group names: Page (?<PageNo>\d+) of (?<PageCount>\d+).

---

Can be one of the following types:

Value	Description
Field Match	Matches the value stored in a previously-extracted field or table column.
Detect Signature	Detects a signature within a specified region of a document page by measuring the percentage of the area that is filled.
Labeled Value	Extracts a field presented as a label-value pair within a document, associating labels and values based on their spatial relationship.
Label Match	Matches a list of one or more label values, using matching options defined by a Labeling Behavior.
Read Metadata	Reads a metadata value from a document by accessing a property on an attachment or content link.
Select Page	Selects and outputs the full content of one or more pages from a document, based on page number and/or content criteria.
Find Barcode	Searches for barcode values in document Layout Data previously detected during image processing.
Highlight Zone	Defines a region of a document to be visually highlighted, without extracting any data values.
Labeled OMR	Reads a group of one or more checkboxes located nearby text labels.
Ordered OMR	Reads one or more checkboxes with a consistent order of appearance inside a rectangular region.
Read Barcode	Extracts barcode values from document images using configurable barcode recognition.
Read Zone	Extracts text content from a specified rectangular region (zone) of a document.
Reference	Delegates extraction to another configured extractor, enabling reuse and centralization of extraction logic.
Zonal OMR	Reads one or more checkboxes using manually-configured zones.
List Match	Extracts values from document text that match any entry in a list of search terms.
Pattern Match	Extracts values from document text that match a specified regular expression pattern.
Word Match	Extracts individual words and multi-word phrases (N-grams) from document text for use in classification, data extraction, and normalization.
Query XML	Extracts values from XML documents using XPATH queries, enabling structured data extraction from XML content in Grooper.
Query HTML	Extracts values from an HTML document using a CSS or XPath selector.
Ask AI	Executes a completion using a large language model (LLM) and returns one hit for each choice in the response.
AI Column Extractor	Extracts structured content from documents with two-column layouts.
Entity Recognition	Identifies and categorizes entities such as people, organizations, locations, and quantities in unstructured text.
Key Phrase Extraction	Identifies key concepts and topics in text using Azure AI Language key phrase extraction.
Pii Entity Recognition	Identifies, categorizes, and redacts sensitive information (PII) in unstructured text using Azure AI Language Services.

The bullet extractor is used during ESP Auto Separation when analyzing Unstructured Document Types. In some cases, it is possible to determine that two pages are part of the same document based on the numbering in bulleted lists. For example, consider a page containing the bullet numbers shown below:

1. Definitions. Paragraph text...
2. Parties. Paragraph text...
3. Effective Date. Paragraph text...

If the following page contains these bullet numbers, this is evidence that both pages are part of the same document.

4. Severability. Paragraph text...
5. Term. Paragraph text...
6. Confidentiality. Paragraph text...

The bullet extractor should return only individual letters or numeric digits. Surrounding punctuation should excluded from the output value.

---

The 'Use Class Frequency' option determines whether the class frequency (CF) of features is incorporated into the feature weighting scheme during training-based classification.

What is Class Frequency (CF)?

• Class Frequency measures how often a particular feature appears across all training samples for a given Document Type.
• When enabled, the classifier uses the TF-IDF-CF weighting scheme, which extends the traditional TF-IDF (Term Frequency-Inverse Document Frequency) model by also considering how representative a feature is for a specific class.
• This can help distinguish features that are not only frequent in a document, but also characteristic of a particular class, improving classification accuracy in some scenarios.

How It Works

• TF-IDF alone weighs features based on their frequency in a document (TF) and their rarity across all documents (IDF).
• TF-IDF-CF further adjusts the weight by considering how concentrated a feature is within a specific class versus being spread across multiple classes.
• Features that are common in one class but rare in others receive a higher weight, making them more influential in classification decisions.

When to Enable

• Enable 'Use Class Frequency' for document sets where certain features are highly indicative of specific Document Types, and you want to emphasize these distinctions.
• Particularly useful for large, diverse, or unbalanced training sets, or when some features are common across many types but only truly meaningful for a subset.
• For small or highly uniform document sets, the impact may be minimal.

Practical Guidance

• Recommended: Use in combination with well-curated training samples to maximize the benefit of class-specific features.
• Not Recommended: If your features are already highly unique per class, or if you have very few training samples per class, enabling CF may have limited effect.
• Adjust in conjunction with other weighting options (such as 'Frequency Scaling' and 'Use Confidence') to fine-tune classification performance.

References

• For more details on the TF-IDF-CF approach, see:
  "An improvement of TF-IDF weighting in text categorization" (Mingyong Liu & Jiangang Yang, 2012).

---

The 'Use Confidence' option controls whether the confidence score of each extracted feature is incorporated into the term frequency calculation during training-based classification.

What is Feature Confidence?

• Many extraction processes, such as OCR or pattern matching, assign a confidence score to each extracted value, indicating the system's certainty in the result.
• Confidence scores typically range from 0.0 (no confidence) to 1.0 (high confidence).
• When enabled, these scores are used to weight the contribution of each feature, so features with higher confidence have a greater impact on classification.

How It Works

• If 'Use Confidence' is enabled, the frequency of each feature is multiplied by its confidence score when building the feature profile for a document.
• This means that features extracted with low confidence (e.g., uncertain OCR results) will have less influence on the classification outcome.
• If disabled, all features are treated equally, regardless of their confidence, and only their occurrence count is considered.

Assigning Multipliers with ResultSetOptions

• You can further control feature weighting by assigning a multiplier to the confidence values for specific extractors using ResultSetOptions.
• This allows you to boost the influence of features extracted by certain Value Extractors that are known to be highly distinctive or important for classification.
• For example, you might configure a multiplier for an extractor that identifies the centered title of a lease contract, ensuring that this key feature is given extra weight in the classification process.
• By combining confidence scores with custom multipliers, you can fine-tune the model to prioritize features that are most indicative of a particular Document Type.

When to Enable

• Enable 'Use Confidence' when your feature extraction process produces meaningful confidence scores, such as when using OCR or probabilistic extractors.
• Use in conjunction with ResultSetOptions multipliers to emphasize critical features or extraction strategies.
• For extractors that do not provide confidence scores (or always return 1.0), enabling this option has no effect.

Practical Guidance

• Use in combination with high-quality extractors to maximize the benefit of confidence weighting.
• Review the distribution of confidence scores in your extraction results to ensure they are meaningful and not uniformly high or low.
• Assign multipliers in ResultSetOptions to boost the impact of features that are especially relevant for your classification goals.
• Adjust in conjunction with other weighting options, such as 'Use Class Frequency' and 'Frequency Scaling', to fine-tune classification performance.

Example

• If a word is extracted twice, once with confidence 0.95 and once with 0.60, its total frequency will be 1.55 (0.95 + 0.60) when 'Use Confidence' is enabled.
• If a multiplier of 2.0 is assigned to the extractor using ResultSetOptions, the weighted frequency becomes 3.10 ((0.95 + 0.60) * 2.0).
• If disabled, the frequency would simply be 2, regardless of confidence or multiplier.

---

Can be one of the following values:

Name	Value	Description
Normal	0	Normalizes term frequency to the size of the document. Each feature's count is divided by the total number of features in the document. This mode provides matching that is independent of document size, so short and long documents are compared on equal footing. Note: In some cases, this can produce false positives, as a very short document may match a long document with high similarity if the same features are present.	►
Logarithmic	1	Scales term frequency logarithmically. Applies a logarithmic function to feature counts, reducing the impact of very frequent features. Useful for legacy models or when you want to minimize the influence of repeated features. Provided primarily for backward compatibility.	►
Augmented	2	Normalizes term frequency to the most-frequent feature in the document, with optional dampering. Each feature's count is divided by the count of the most frequent feature, then scaled by the 'Frequency Scaling' property. This mode allows document size and feature repetition to play a greater role in classification. The dampering factor ('Frequency Scaling') controls how much frequency matters: higher values increase the effect, lower values reduce it. Useful when you want to tune the sensitivity to repeated features or document length.	►

The 'TfModes' enumeration controls the method used to compute term frequency (TF) for features during document classification. The choice of mode affects how feature occurrences are weighted, which in turn influences similarity calculations between documents and trained Document Types.

Overview

• Term Frequency (TF): Measures how often a feature (such as a word or token) appears in a document, relative to the document's size or other features.
• The selected mode determines how TF is normalized or scaled, impacting sensitivity to document length, feature repetition, and feature prominence.

Available Modes

• Normal:
  Normalizes feature counts by the total number of features in the document, making TF independent of document size.
• Logarithmic:
  Applies a logarithmic scale to feature counts, reducing the impact of very frequent features. Provided for backward compatibility.
• Augmented:
  Normalizes feature counts by the most frequent feature in the document, allowing document size to play a greater role in classification. Includes a dampering factor controlled by the 'Frequency Scaling' property.

Practical Guidance

• Use 'Normal' for most scenarios where document length should not affect classification.
• Use 'Logarithmic' for legacy models or when you want to reduce the influence of highly repetitive features.
• Use 'Augmented' when you want longer documents or repeated features to have more influence, or when tuning with the 'Frequency Scaling' property.

---

The 'Frequency Scaling' property controls how much the frequency of a feature (term) influences its weight during classification, specifically when 'Term Frequency Mode' is set to 'Augmented'.

Purpose

• In training-based classification, features that appear more frequently in a document are typically considered more important.
• However, if frequency is weighted too heavily, common but less distinctive features may dominate the classification, reducing accuracy.
• 'Frequency Scaling' provides a way to fine-tune the impact of repeated features, balancing between sensitivity to frequency and the need to avoid overemphasizing common terms.

How It Works

• When 'Term Frequency Mode' is set to 'Augmented', the raw frequency of each feature is adjusted using the 'Frequency Scaling' value.
• The scaling value is a decimal between 0.0 and 1.0:
  • Higher values (closer to 1.0) make feature frequency more significant, so repeated features have a greater effect on classification.
  • Lower values (closer to 0.0) reduce the impact of frequency, making each feature count more equally regardless of repetition.
• The adjusted frequency is then used in the TF-IDF (or TF-IDF-CF) weighting calculation for each feature.

Practical Guidance

• Increase 'Frequency Scaling' if you want the classifier to be more sensitive to features that appear multiple times in a document.
• Decrease the value if you want to minimize the influence of repeated features, which can help prevent common or boilerplate terms from skewing results.
• Typical values range from 0.5 to 1.0, but the optimal setting depends on your document set and classification goals.
• This property only affects weighting when 'Term Frequency Mode' is set to 'Augmented'. For other modes, it is ignored.

Example

• If 'Frequency Scaling' is set to 0.75, a feature that appears 4 times in a document will have its frequency adjusted less aggressively than if the scaling were set to 1.0.
• Setting the value to 0.0 effectively disables frequency scaling, treating all features as if they appear only once.

---

Can be one of the following values:

Name	Value	Description
Normal	0	Standard IDF calculation. Calculates IDF as log(TotalClasses / ClassCount), where 'TotalClasses' is the number of Document Types and 'ClassCount' is the number of types containing the feature. Features that appear in only one Document Type receive the highest weight. Use this mode when you want rare features to have maximum influence on classification.	►
Smooth	1	Smoothed IDF calculation. Calculates IDF as log(1 + TotalClasses / ClassCount), adding a smoothing factor to prevent division by zero and reduce extreme weights. Useful when the number of Document Types is small or when you want to avoid overemphasizing rare features. Provides more stable and balanced feature weighting in edge cases.	►

The 'IdfModes' enumeration controls the method used to compute inverse document frequency (IDF) for features during document classification. IDF measures how unique or common a feature is across all Document Types, and is a key component in weighting features for similarity calculations.

Overview

• Inverse Document Frequency (IDF):
  Reduces the weight of features that are common across many Document Types, and increases the weight of features that are rare or distinctive.
• The selected mode determines whether standard or smoothed IDF is used, which can affect the handling of rare or ubiquitous features.

Available Modes

• Normal:
  Uses the standard IDF calculation, which may assign very high weights to features that appear in only one Document Type.
• Smooth:
  Adds smoothing to the IDF calculation, preventing extreme weights for rare features and improving stability when the number of Document Types is small.

Practical Guidance

• Use 'Normal' for most scenarios where the number of Document Types is moderate to large and rare features should be highly weighted.
• Use 'Smooth' when you want to avoid extreme weighting for features that appear in only one or very few Document Types, or when working with small sets of types.

For more information, see the documentation for Lexical, Document Type, and 'Document Frequency Mode'.

---

• Enter a time span (e.g., "15m" for 15 minutes).
• The model is reloaded after this period to ensure up-to-date results.

---

• Enter a time span (e.g., "1m" for 1 minute).
• If the model is unused for this period, it is refreshed on next use.