Lexicon - Normalize

Normalizes character data in the lexicon to the character set of the configured language.

Remarks

The Normalize command scans all entries in the selected Lexicon and attempts to convert any characters that are not valid for the configured language into their appropriate equivalents (homoglyphs). This is especially useful for cleaning up imported data, resolving issues with visually similar but distinct Unicode characters, and ensuring consistent matching during extraction and validation.

When to Use

• To correct entries that contain characters from the wrong script or language (e.g., Cyrillic vs. Latin homoglyphs).
• When imported or user-entered data may contain mixed or invalid characters for the target language.
• To improve extraction accuracy and prevent missed matches due to character set inconsistencies.

How It Works

• The command examines each entry in the Lexicon for characters not valid in the configured language.
• If an invalid character is found, the command attempts to convert it to a valid homoglyph.
• If conversion is successful and a matching entry exists, frequency data is merged; otherwise, the entry is considered invalid.
• Depending on the 'Save Invalid Entries' property, invalid entries are either deleted or saved to a new Lexicon for review.

> Note: This command is only enabled when the Lexicon has a language configured.

For more information about language normalization and homoglyph handling, see the documentation for Lexicon.

Properties

Name	Type		Description
Save Invalid Entries	Boolean	►	If true, entries which cannot be normalized will be saved to a new lexicon. Otherwise, invalid entries will be deleted. DefaultTrue Controls the handling of entries that cannot be normalized to the configured language's character set. Usage Guidance Set to true to preserve invalid entries in a new Lexicon (with an " (Invalid Entries)" suffix) for review or correction. Set to false to permanently delete invalid entries from the Lexicon. > Tip: Use this property to audit and correct problematic entries before finalizing your dictionary.

See Also

Lexicon