Fuzzy Lookup Options

Overview

The 'Minimum Similarity' property sets the minimum similarity score (as a percentage, e.g., 0.90 for 90%) required for a fuzzy match to be accepted. Similarity is calculated using edit distance (such as Levenshtein distance), reflecting how closely the input value matches an entry in the vocabulary.

• A higher value (e.g., 0.95) requires nearly exact matches, reducing the chance of incorrect substitutions.
• A lower value (e.g., 0.80) allows more flexible matching, which may be useful for correcting severe OCR or typographical errors, but can increase the risk of false positives.

How Similarity Is Calculated:
Similarity = (Length - EditCost) / Length, where EditCost is the total cost of substitutions, insertions, and deletions needed to transform the input into the candidate value. Custom weightings (see Fuzzy Match Weightings) can adjust these costs.

Usage Scenarios

• Use a high threshold (e.g., 0.95) for strict validation when only minor errors are expected.
• Use a lower threshold (e.g., 0.85) when data is noisy or OCR quality is poor.

Configuration Guidance

• Test with representative data to find the right balance between accuracy and flexibility.
• Adjust in conjunction with 'Weightings' and 'MinimumLength' for best results.

Overview

The 'Minimum Length' property specifies the minimum number of characters a value must have to be eligible for fuzzy matching. Values shorter than this threshold are not submitted for fuzzy matching, and must match the vocabulary exactly.

Usage Scenarios

• Prevents short values (such as single letters or digits) from being incorrectly matched to unrelated entries.
• Useful for avoiding false positives when the vocabulary contains many short or similar entries.

Configuration Guidance

• Set to 0 to allow all values to be fuzzy matched.
• Increase to 3 or more to restrict fuzzy matching to longer values, especially when working with codes or abbreviations.

Best Practices

• Use a higher minimum length if your vocabulary contains many short entries or if you observe over-matching on short values.

Overview

The 'Maximum Depth' property restricts fuzzy matching to the first N entries in the vocabulary or lexicon. This is primarily used to improve performance when working with very large vocabularies, or to prioritize more common or important entries.

• A value of 0 means no limit; all entries are considered.
• When set, only the top N entries (as ordered in the lexicon) are eligible for fuzzy matching.

Usage Scenarios

• Use with frequency-sorted lexicons to prioritize matching against the most common values.
• Improve lookup speed when the vocabulary contains tens or hundreds of thousands of entries.

Configuration Guidance

• Set to a value that balances performance and coverage (e.g., 10,000 for large English word lists).
• For small vocabularies, leave at 0.

Best Practices

• Use in combination with 'AlternateVocabulary' for fine-grained control over which entries are considered for fuzzy matching.

The Fuzzy Match Weightings class specifies custom weightings for fuzzy matching operations, allowing fine-tuned control over the cost of character-level edits such as swaps, insertions, and deletions. By configuring these weightings, users can optimize fuzzy matching accuracy for OCR, data extraction, and document processing scenarios where common character misrecognitions or typographical errors are expected.

By customizing weightings, you can instruct the fuzzy matching engine to treat certain character edits as more or less significant, improving both the likelihood of correct matches and the quality of automatic corrections. For example, you may reduce the cost of swapping visually similar characters (such as 'O' and '0', or 'B' and '8') to account for common OCR confusion, or make certain characters immutable to prevent them from being altered during matching.

How Weightings Work

Fuzzy matching operates by calculating the minimum "edit distance" between a document value and a pattern, where each edit operation (swap, insert, delete) has an associated cost. The total cost is then used to determine the match percentage. By default, all edit operations have a cost of 1.0, but Fuzzy Match Weightings allows you to override these defaults for specific characters or character pairs.

• Swap: The cost to substitute one character for another.
• Insert: The cost to insert a character present in the pattern but missing from the document.
• Delete: The cost to remove a character present in the document but not in the pattern.
• Immutable: Characters that cannot be edited (cost is infinite).

Syntax for Defining Weightings

Weightings are defined using a simple lexicon syntax, with each entry specifying an operation, the target character(s), and the associated cost. See Fuzzy Match Cost Map for more details on the format of entries. The following general forms are supported:

• Immutable=CharSet
  Marks all characters in CharSet as immutable. Example: Immutable=\r\n\t\f.

• Swap(CharacterPair)=Cost
  Sets the cost for swapping a specific character pair. Example: Swap(S5)=0.25 or S5=0.25.

• Swap([[FromSet]], [[ToSet]])=Cost
  Sets the cost for swapping any character in FromSet with any in ToSet. Example: Swap([;:'"=-],[\d])=2.

• Swap(CharSet)=Cost
  Sets the cost for swapping any character in CharSet for a different character. Example: Swap(0-9)=1.5.

• Delete(CharSet)=Cost
  Sets the cost for deleting any character in CharSet. Example: Delete(')=0.1.

• Insert(CharSet)=Cost
  Sets the cost for inserting any character in CharSet. Example: Insert(.,)=0.50.

• Swap=BaseCost, Delete=BaseCost, Insert=BaseCost
  Sets the default cost for all swap, delete, or insert operations. Default is 1.0 for each.

Practical Guidance

• Use lower costs for character pairs that are frequently confused (e.g., O0=0.25 for 'O' and '0').
• Make control or formatting characters immutable to prevent accidental edits (e.g., Immutable=\r\n\t\f).
• Increase costs for edits that should be discouraged or are unlikely in your data.
• If no weightings are defined, all edit operations default to a cost of 1.0.

Example Scenario

Suppose a document contains the value 245B instead of 2458 due to an OCR error. By default, this would match a pattern like \d{4} at 75%. If you add a weighting B8=0.5, the match percentage increases to 87.5%, and the fuzzy matching engine can automatically correct the value to 2458.

Sample Weightings Lexicon

Best Practices

• Review common OCR or data entry errors in your documents and adjust weightings accordingly.
• Test your configuration with real-world samples to ensure optimal extraction accuracy.

Overview

The 'Alternate Vocabulary' property allows you to specify a different Embedded Lexicon to be used exclusively for fuzzy matching, instead of the main vocabulary. This is useful for improving performance or accuracy by restricting fuzzy matching to a smaller, more relevant set of entries.

• By default, fuzzy matching uses the main vocabulary.
• If 'AlternateVocabulary' is set and not empty, only its entries are considered for fuzzy matching.

Usage Scenarios

• Limit fuzzy matching to a curated list of canonical or high-confidence values, while allowing exact matches against a broader set.
• Improve performance when the main vocabulary is very large, but only a subset should be considered for fuzzy correction.

Configuration Guidance

• Populate the alternate vocabulary with the most important or representative entries for your use case.
• Use in conjunction with 'MaximumDepth' for further control.

Best Practices

• Regularly review the alternate vocabulary to ensure it reflects current business needs and data quality requirements.
• Test with real-world data to confirm that desired corrections are still made.