AI Schema Extractor

The Data Generator object configures how Grooper uses large language models (LLMs) to generate structured data from unstructured document content. It is embedded as a property within various Grooper features that leverage generative AI, such as advanced data extraction, document classification, and content transformation.

Configuration Guidance

• Choose a model that matches your accuracy, speed, and cost requirements.
• Use structured output mode whenever possible for the most reliable results.
• Adjust chat parameters to fine-tune the LLM's behavior for your specific use case.
• Lowering the 'Temperature' can improve consistency and reduce variability in responses, especially for structured data extraction.
• Increase 'Max Tries' if you encounter intermittent failures or unparseable responses.

How It Works

When Grooper performs an extraction using a Data Generator:

1. The extraction schema (as a JSON object) and system/user messages are sent to the LLM.
2. The LLM is prompted to return data in the required JSON format.
3. If 'Use Structured Output' is enabled and supported by the model, the LLM is given explicit instructions to return strictly valid JSON matching the schema.
4. Grooper parses the LLM's response and maps it to the extraction schema.
5. If the response is invalid, incomplete, or a transient error occurs, Grooper retries the request up to 'Max Tries'.

This approach enables robust, schema-driven extraction from a wide variety of document types, even when traditional rule-based methods are insufficient.

Diagnostics and Logging

When extraction is performed, Data Generator logs diagnostic artifacts to aid in troubleshooting and auditing:

• Chat Log.jsonl: A line-delimited JSON log of all messages exchanged with the LLM during extraction.
• Response Data.json: The raw JSON data returned by the LLM for each extraction attempt.
• LLM Completion Operation: Timing and performance metrics for each LLM request, recorded in the diagnostic log.

These artifacts are available in the extraction diagnostics and can be used to review LLM interactions, analyze failures, and optimize configuration.

Can be one of the following types:

The 'Document Quoting' property determines the quoting method used to provide document context to the LLM during both data extraction. Quoting methods control how text is formatted, delimited, and annotated in the prompt sent to the AI model.

Usage and Configuration

• Choose a quoting method that best preserves the structure and meaning of your document content.
• Custom quoting methods can be used for specialized document types or extraction scenarios.

Impact

• The quoting method can significantly affect extraction accuracy, especially for documents with complex or ambiguous layouts.
• Consistent quoting improves the LLM's ability to align extracted data with the original document.

Example

• Use a quoting method that adds line numbers for multi-line transactions, or a delimiter-based method for tabular data.

For more information, see the documentation for Quoting Method.

The 'Extraction Instructions' property allows you to supply additional, user-defined instructions to the LLM for extracting data from each detected transaction. Use this property to clarify extraction requirements, handle edge cases, or provide document-specific guidance.

Usage and Configuration

• Enter clear, concise instructions to improve extraction accuracy, especially for complex or non-standard documents.
• Instructions are appended to the LLM prompt for each extraction operation.
• If left blank, the LLM will rely on the default schema and context.

Example

Extract only the "Invoice Date" and "Total Amount" fields. Ignore any summary or footer lines.
Use the date format YYYY-MM-DD.

Best Practices

• Use this property to address ambiguous layouts, special formatting, or business rules not covered by the schema.
• Review extraction diagnostics to refine instructions as needed.

The 'JSON Schema' property allows you to specify a JSON Schema that defines the exact structure of the LLM's response. When provided, the LLM is instructed to return output that conforms to this schema, enabling highly reliable and consistent extraction of structured data.

How It Works

• The schema must be a valid JSON object with "type": "object" and a "title" property.
• All expected properties, their types, and any validation rules should be defined in the schema.
• When a schema is set, the LLM is prompted to return only JSON matching the schema, and the extractor will parse and validate the response accordingly.

Configuration Guidance

• Use the code editor to author or paste a JSON schema that matches your extraction requirements.
• Always include a "title" property (e.g., "title": "InvoiceLineItem").
• Define each property with its type (e.g., "type": "string", "type": "number").
• Optionally, add required, enum, default, or description fields for more precise control.

Example

Validation

• The schema is checked at runtime. If "type" is not "object" or "title" is missing, a validation error is raised.
• Invalid or incomplete schemas may result in extraction errors or unpredictable LLM output.

Usage Scenarios

• Extracting structured records such as parties, line items, or key-value pairs.
• Enforcing a consistent output format for downstream processing in tables or sections.
• Reducing post-processing by ensuring the LLM returns only the required fields.

Providing a schema is highly recommended for complex or multi-field extraction tasks.

The 'Selector' property specifies a JSONPath expression used to extract a specific portion of the JSON returned by the LLM when 'Parse JSON Response' is enabled. This allows you to target and extract only the relevant part of a potentially complex or nested JSON response.

Purpose and Usage

• Use the selector to isolate arrays, objects, or fields within the JSON that correspond to the data you want to extract.
• This is especially useful when the LLM returns additional metadata, summaries, or nested structures, and only a subset is needed for downstream processing.

How to Construct a Selector

• Prefix expressions with $ to indicate the root object.
• Use dot notation for nested access (e.g., $.data.items).
• Use wildcards (*) or filters to target arrays or conditionally included values.

Example

If the LLM responds with: { "summary": "Overview text...", "parties": [ { "name": "Alice" }, { "name": "Bob" } ] } Setting Selector to $.parties will isolate the array and generate one output instance per object inside it.

Configuration Guidance

• Leave blank to parse the entire JSON response.
• Use selectors to extract only the relevant array or object for Data Table or Data Section compatibility.
• Test your selector using sample JSON to ensure it matches the intended data.

Troubleshooting

• If the selector path yields no matches, no instances will be produced.
• Ensure the JSONPath expression is valid and matches the structure of the LLM's response.
• Use online JSONPath evaluators or Grooper's diagnostic tools to verify your selector.

The selector provides precise control over which part of the LLM's JSON output is treated as the extractable payload, enabling robust and flexible structured extraction workflows.

The 'Parse JSON Response' property controls whether the extractor expects and parses JSON output from the LLM.

When enabled, the extractor will:

• Instruct the LLM (via the prompt or schema) to return only JSON.
• Parse the returned JSON into a hierarchy of Data Instances, with named children for each property or array element.
• Support both single-object and array-of-object responses, enabling extraction of tables, lists, or multi-field records.

Primary Goal: Data Table and Section Compatibility

The primary goal when parsing JSON to produce child Data Instances is to output a data instance hierarchy that matches the data layout of a Data Table or Data Section. This ensures that the extracted structure can be seamlessly consumed by a Table Extract Method or a Section Extract Method, supporting downstream structured extraction workflows in Grooper.

• Each object or array in the JSON is mapped to a Data Instance or a collection of instances, mirroring the row/field structure expected by tables and sections.
• This compatibility allows for direct use of LLM-extracted data in Grooper's table and section processing features, reducing the need for manual mapping or transformation.

Configuration Guidance

• Enable this property when you want to extract structured data, such as tables, lists, or objects with multiple fields.
• Ensure your 'Instructions' property clearly requests JSON output, and provide examples or a schema if needed.
• Use the 'Selector' property to extract a specific portion of the JSON (e.g., a nested array or object).

Example Prompts

• Return a JSON array of all parties, each with "name" and "role" fields.
• Extract the invoice header as a JSON object with "date", "number", and "total".

Output Structure

• If the LLM returns a single JSON object, one Data Instance is created with children for each property.
• If the LLM returns a JSON array, one Data Instance is created for each object in the array.
• This structure is compatible with downstream Table Extract Method and Section Extract Method operations.

Usage Tips

• For best results, instruct the LLM to return only JSON, without extra text or formatting.
• If using a schema, the extractor will validate and enforce the expected structure.
• If the response is not valid JSON, extraction will fail or produce no results.

This property is essential for reliable structured extraction using LLMs in Grooper.

The 'Match All' property determines whether the extractor returns every matching instance of a value found in the source content, or only the single best match for each extracted value.

Purpose and Behavior

• When enabled (true), the extractor will search for and return all occurrences of each extracted value within the document context. This is especially useful when the LLM response may correspond to multiple locations in the document, such as repeated values, lists, or references that appear in several places.
• When disabled (false), only the single best-matching instance (typically the one with the highest confidence or best alignment) is returned for each extracted value. This is appropriate for unique values or when only one instance is needed.

Configuration Guidance

• Enable 'Match All' when extracting data that may appear multiple times, such as party names, repeated line items, or references in tables and lists.
• Disable 'Match All' for fields where only a single, most relevant value is required (e.g., invoice number, effective date).

Example Scenarios

• Enabled: Extracting all instances of a party name that may appear in both the header and signature block.
• Disabled: Extracting a unique contract number that should only appear once.

This property provides fine-grained control over how multiple matches are handled during extraction, supporting both single-value and multi-value use cases.