Pick

The Data Generator object configures how Grooper uses large language models (LLMs) to generate structured data from unstructured document content. It is embedded as a property within various Grooper features that leverage generative AI, such as advanced data extraction, document classification, and content transformation.

Configuration Guidance

• Choose a model that matches your accuracy, speed, and cost requirements.
• Use structured output mode whenever possible for the most reliable results.
• Adjust chat parameters to fine-tune the LLM's behavior for your specific use case.
• Lowering the 'Temperature' can improve consistency and reduce variability in responses, especially for structured data extraction.
• Increase 'Max Tries' if you encounter intermittent failures or unparseable responses.

How It Works

When Grooper performs an extraction using a Data Generator:

1. The extraction schema (as a JSON object) and system/user messages are sent to the LLM.
2. The LLM is prompted to return data in the required JSON format.
3. If 'Use Structured Output' is enabled and supported by the model, the LLM is given explicit instructions to return strictly valid JSON matching the schema.
4. Grooper parses the LLM's response and maps it to the extraction schema.
5. If the response is invalid, incomplete, or a transient error occurs, Grooper retries the request up to 'Max Tries'.

This approach enables robust, schema-driven extraction from a wide variety of document types, even when traditional rule-based methods are insufficient.

Diagnostics and Logging

When extraction is performed, Data Generator logs diagnostic artifacts to aid in troubleshooting and auditing:

• Chat Log.jsonl: A line-delimited JSON log of all messages exchanged with the LLM during extraction.
• Response Data.json: The raw JSON data returned by the LLM for each extraction attempt.
• LLM Completion Operation: Timing and performance metrics for each LLM request, recorded in the diagnostic log.

These artifacts are available in the extraction diagnostics and can be used to review LLM interactions, analyze failures, and optimize configuration.

Provide clear, specific instructions that explain the criteria the AI should use to select a document. These instructions are sent to the AI along with information about each candidate document. Be explicit about what makes a document preferable.

For example, to select the highest quality scan, you might write: "Select the document with the clearest text, highest resolution, and fewest visual artifacts or distortions." To choose the most complete form, you might write: "Select the document that has the most fields filled out and contains the required signature."

The more specific and measurable your criteria, the more consistent and accurate the AI's selections will be.

Can be one of the following types:

Specifies a Quoting Method that controls what information about each document is included in the AI's analysis. The quoting method determines which document properties, metadata, or content excerpts are provided to help the AI make an informed selection decision.

If not specified, the activity uses a standard format that includes basic document information. For more control over what the AI sees when evaluating documents, configure a custom Quoting Method that extracts and formats the specific properties or content relevant to your selection criteria.

Can be one of the following types:

Specifies a Quoting Method that controls what information about each document is included in the AI's analysis. The quoting method determines which document properties, metadata, or content excerpts are provided to help the AI make an informed selection decision.

If not specified, the activity uses a standard format that includes basic document information. For more control over what the AI sees when evaluating documents, configure a custom Quoting Method that extracts and formats the specific properties or content relevant to your selection criteria.

Specifies which Content Type the candidate documents must have to be included in the selection process. Only child Batch Folders that match this content type will be evaluated by the AI. This property is required.

For example, if you set this to an "Invoice" content type, only folders classified as invoices will be considered. Documents with other content types in the same parent folder will be ignored.

When enabled, the selected document will be flagged in Grooper with the AI's explanation of why it was chosen. This provides a visible audit trail of the selection decision directly on the document.

Flagging is useful for review workflows where users need to understand why a particular document was selected over others, or for troubleshooting selection logic during configuration. The flag text contains the reason provided by the AI model.

When disabled, the document is still selected but no flag is applied. The selection reason is still recorded in diagnostic logs if diagnostics are enabled.

Specifies a Content Type to apply to the selected document after it has been chosen by the AI. This allows you to automatically reclassify or promote the selected document to a different content type as part of the selection process.

For example, if you're selecting the best quality scan from multiple "Scanned Document" renditions, you might assign a "Final Invoice" content type to the winner. This reclassification can trigger different extraction rules, validation logic, or downstream routing for the selected document.

If not specified, the selected document retains its original content type. When a content type is assigned, the 'Keep Existing Data' property controls whether existing extracted data is preserved or discarded.

Controls whether index data that has already been extracted from the document is retained when 'Type To Assign' applies a new Content Type. This property only appears when 'Type To Assign' is configured.

When enabled, existing data field values are preserved if the new content type has data fields in common with the existing content type. This is useful when the new content type is a subtype or closely related to the original, allowing you to keep relevant extracted information without re-extraction.

When disabled, all existing data is cleared and the document starts fresh with the new content type's data model. This ensures a clean state for extraction and is appropriate when the new content type has a completely different structure or data requirements.

Specifies one or more Data Rules to automatically run on the selected document. The rules execute after the document has been selected and any content type assignment has been applied. This allows you to immediately extract, transform, or validate data as part of the selection workflow.

Rules are executed in the order they appear in the list. Each rule operates on the document's index data using either the newly assigned content type (if 'Type To Assign' is configured) or the original 'Content Type'.

For example, after selecting the highest quality invoice scan, you might run extraction rules to capture invoice number, date, and amount fields. This eliminates the need for a separate extraction step and ensures the selected document is immediately ready for downstream processing.

If no rules are specified, the selected document is left as-is with no additional data processing.

A combination of the following flags:

The Error Disposition property determines the system's response to errors that occur while executing the activity.

• Flag: The affected Batch Folder or Batch Page will be flagged for review.
• Log: The error will be written to the Grooper event log.
• Stop: Processing will halt, the Batch will be set to an error state, and all pending tasks will be deleted.

You can combine multiple options using the IssueDisposition flags.
Adjust this property to match your organization's error handling and escalation policies.

If the number of consecutive errors reaches this value, a critical stop will occur and the Activity Processing service will halt further processing.

• Use this property to prevent runaway failures and to ensure that persistent errors are escalated for review.
• Set to a higher value to allow more retries before stopping, or to a lower value for stricter error control.

Can be one of the following values:

The Concurrency Mode property determines whether the activity can be executed in parallel across multiple Batch Objects, or if it must run sequentially.

• This setting affects which Processing Queue the activity is submitted to and how tasks are distributed.
• Choose the appropriate mode based on the activity's thread safety and resource requirements.