Fuzzy Match Weightings

Defines custom character-level edit costs for fuzzy matching, enabling precise control over how swaps, insertions, and deletions are evaluated during data extraction.

Remarks

The Fuzzy Match Weightings class specifies custom weightings for fuzzy matching operations, allowing fine-tuned control over the cost of character-level edits such as swaps, insertions, and deletions. By configuring these weightings, users can optimize fuzzy matching accuracy for OCR, data extraction, and document processing scenarios where common character misrecognitions or typographical errors are expected.

By customizing weightings, you can instruct the fuzzy matching engine to treat certain character edits as more or less significant, improving both the likelihood of correct matches and the quality of automatic corrections. For example, you may reduce the cost of swapping visually similar characters (such as 'O' and '0', or 'B' and '8') to account for common OCR confusion, or make certain characters immutable to prevent them from being altered during matching.

How Weightings Work

Fuzzy matching operates by calculating the minimum "edit distance" between a document value and a pattern, where each edit operation (swap, insert, delete) has an associated cost. The total cost is then used to determine the match percentage. By default, all edit operations have a cost of 1.0, but Fuzzy Match Weightings allows you to override these defaults for specific characters or character pairs.

• Swap: The cost to substitute one character for another.
• Insert: The cost to insert a character present in the pattern but missing from the document.
• Delete: The cost to remove a character present in the document but not in the pattern.
• Immutable: Characters that cannot be edited (cost is infinite).

Syntax for Defining Weightings

Weightings are defined using a simple lexicon syntax, with each entry specifying an operation, the target character(s), and the associated cost. See Fuzzy Match Cost Map for more details on the format of entries. The following general forms are supported:

• Immutable=CharSet
  Marks all characters in CharSet as immutable. Example: Immutable=\r\n\t\f.

• Swap(CharacterPair)=Cost
  Sets the cost for swapping a specific character pair. Example: Swap(S5)=0.25 or S5=0.25.

• Swap([[FromSet]], [[ToSet]])=Cost
  Sets the cost for swapping any character in FromSet with any in ToSet. Example: Swap([;:'"=-],[\d])=2.

• Swap(CharSet)=Cost
  Sets the cost for swapping any character in CharSet for a different character. Example: Swap(0-9)=1.5.

• Delete(CharSet)=Cost
  Sets the cost for deleting any character in CharSet. Example: Delete(')=0.1.

• Insert(CharSet)=Cost
  Sets the cost for inserting any character in CharSet. Example: Insert(.,)=0.50.

• Swap=BaseCost, Delete=BaseCost, Insert=BaseCost
  Sets the default cost for all swap, delete, or insert operations. Default is 1.0 for each.

Practical Guidance

• Use lower costs for character pairs that are frequently confused (e.g., O0=0.25 for 'O' and '0').
• Make control or formatting characters immutable to prevent accidental edits (e.g., Immutable=\r\n\t\f).
• Increase costs for edits that should be discouraged or are unlikely in your data.
• If no weightings are defined, all edit operations default to a cost of 1.0.

Example Scenario

Suppose a document contains the value 245B instead of 2458 due to an OCR error. By default, this would match a pattern like \d{4} at 75%. If you add a weighting B8=0.5, the match percentage increases to 87.5%, and the fuzzy matching engine can automatically correct the value to 2458.

Sample Weightings Lexicon

Immutable=\t\r\n\f Delete(')=0.10 Insert(\d)=2 Swap(['".,;:-],[\d])=2 O0=0.25 o0=0.25 Q0=0.25 D0=0.25 C0=0.25 c0=0.5 I1=0.25 i1=0.25 ]1=0.25 l1=0.25 t1=0.25 L1=0.75 il=0.25 !1=0.25 ?2=0.75 ?7=0.8 Z2=0.5 B3=0.5 A4=0.5 S5=0.25 s5=0.25 G6=0.5 B8=0.25 g9=0.25

Best Practices

• Review common OCR or data entry errors in your documents and adjust weightings accordingly.
• Test your configuration with real-world samples to ensure optimal extraction accuracy.

Properties

Name	Type		Description
Include Field List Values	Boolean	►	Specifies that values from the parent field's value list should be included in this Embedded Lexicon. DefaultFalse Enable this property to merge values from the parent field's value list with local entries and included lexicons. Useful for feeding the value list configured on a Data Field into an extractor such as List Match. To use this property, the Embedded Lexicon must exist on a Data Field or a Node which is a descendant of a Data Field.
Local Entries	String	►	A list of local lexicon entries. EditorLocal Entries Editor ConverterPg Text Lines Converter Overview The 'Local Entries' property allows you to define local entries for this Embedded Lexicon, one per line. The format of each entry depends on the selected 'Type' (see the 'Type' property). Local entries are merged with any included Lexicons and, if enabled, values from the parent field. Entry Format by Lexicon Type Lookup: Each entry is a key-value pair separated by '='. Use for mapping codes to descriptions, abbreviations, or translations. Vocabulary: Each entry is a single value (word, phrase, or code), one per line. Use for simple lists of valid values, such as stopwords or allowed codes. Frequency: Each entry is a key-count pair separated by '='. Use for frequency analysis or weighted lookups. Practical Guidance Place one entry per line. For Lookup and Frequency types, use '=' to separate key and value/count. Blank lines are ignored. When editing, use the code editor for syntax highlighting and validation. Local entries are merged with included Lexicons and parent field values (if enabled), with duplicates removed at runtime. Review the effective list to ensure accuracy and avoid unintended duplicates or conflicts.
Included Lexicons	Lexicon[]	►	One or more external Lexicons to include in this Embedded Lexicon. EditorLexicons Editor ConverterCollection Converter Overview The 'Lexicons' property allows you to reference one or more external Lexicons. All entries from these Lexicons are merged with local entries and, if enabled, parent field values to form the complete value list at runtime. How It Works All entries from included Lexicons are merged with local entries and parent field values (if enabled). Duplicates are removed at runtime. The 'Runtime Type' property determines the effective type used for parsing and matching entries. If all included Lexicons share the same type, that type is used. If types are mixed, the type defaults to 'Lookup' for compatibility. Usage Scenarios Share value lists across multiple fields or components by referencing a common Lexicon. Build composite or multi-language dictionaries by including multiple Lexicons. Centralize management of large or frequently updated value lists. Practical Guidance Avoid circular references when including Lexicons. For consistent results, ensure all included Lexicons use the same Lexicon Type. Review the merged list to ensure there are no unintended duplicates or conflicts. Use the 'Included Lexicons' editor to browse and select Lexicons from the repository tree.

See Also

Lexicon

Used By

FRX OptionsLabeling BehaviorFuzzy Lookup Options