i* |UdZddlmZddlZddlZddlmZdRdZideddhd ed d hd ed hd ed hded hdeddhdeddhdedhdedhdedhdedhdedhdehdded d!hd"ed#hd$edhd%ed&hed'hed(hed)hed*hed+hed,hd-Z d.e d/<d0d1d0d2d3d4d5d6d7d8d9d8d: Z d;e d<< dSd=Z d>Z d?e d@<dAZd?e dB<dTdCZeZd.e dD<dUdEZe j%Dcic]\}}e|edF|Dc}}Zd.e dG<dTdHZeZd.e dI<dVdJZdWdKZedLdMhZdNe dO<dXdPZ dYdQZycc}}w)ZaAEncoding equivalences and legacy name remapping. This module defines: 1. **Directional supersets** for accuracy evaluation: detecting a superset encoding when the expected encoding is a subset is correct (e.g., detecting utf-8 when expected is ascii), but not the reverse. 2. **Bidirectional equivalents**: groups of encodings where detecting any member when another member was expected is considered correct. This includes UTF-16/UTF-32 endian variants (which encode the same text with different byte order) and ISO-2022-JP branch variants (which are compatible extensions of the same base encoding). 3. **Preferred superset mapping** for the ``should_rename_legacy`` API option: replaces detected ISO/subset encoding names with their Windows/CP superset equivalents that modern software actually uses. ) annotationsN) DetectionDictc tj|jS#t$r3|j j ddj ddcYSwxYw)zNormalize encoding name for comparison. :param name: The encoding name to normalize. :returns: The canonical codec name, or a lowered/stripped fallback. -_)codecslookupname LookupErrorlowerreplace)r s d/Users/bowang/.openclaw/workspace/ChatDev/.venv/lib/python3.12/site-packages/chardet/equivalences.pynormalize_encoding_namersQ >}}T"''' >zz|##C,44S"==>s!9AAasciizutf-8z windows-1252tis-620 iso-8859-11cp874gb2312gb18030gbkbig5 big5hkscscp950 shift_jiscp932shift_jis_2004zshift-jisx0213zeuc-jpz euc-jis-2004z euc-jisx0213euc-krcp949cp037cp1140z iso-2022-jp> iso2022-jp-2iso2022-jp-extiso2022-jp-2004z iso2022-jp-1r"r#z iso2022-jp-3r$ iso-8859-1 iso-8859-2z windows-1250z windows-1251z windows-1256z windows-1253z windows-1255z windows-1254z windows-1257) iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-13dict[str, frozenset[str]] SUPERSETSz Windows-1252CP949z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254CP874z Windows-1257) rrr%r&r'r(r)r*r+rr,rzdict[str, str]PREFERRED_SUPERSETc|jd}t|tr'tj|j ||d<|S)a7Replace the encoding name with its preferred Windows/CP superset. Modifies the ``"encoding"`` value in *result* in-place and returns *result* for fluent chaining. :param result: A detection result dict containing an ``"encoding"`` key. :returns: The same *result* dict, modified in-place. encoding)get isinstancestrr1r )resultencs rapply_legacy_renamer9bs> **Z C#s/33CIIKEz M))zutf-16z utf-16-lez utf-16-be)zutf-32z utf-32-lez utf-32-be)r"r$r#ztuple[tuple[str, ...], ...]BIDIRECTIONAL_GROUPS))skcs)ukrubgbe)msid)nodasvLANGUAGE_EQUIVALENCEScNi}tD]}t|}|D]}|||< |S)zBBuild a lookup: ISO code -> frozenset of all equivalent ISO codes.)rG frozenset)r7group group_setcodes r_build_language_equiv_indexrMs5(*F&e$ D$F4L' Mr:_LANGUAGE_EQUIVcL||k(rytj|}|duxr||vS)aCheck whether *detected* is an acceptable language for *expected*. Returns ``True`` when *expected* and *detected* are the same ISO 639-1 code, or belong to the same equivalence group in :data:`LANGUAGE_EQUIVALENCES`. :param expected: Expected ISO 639-1 language code. :param detected: Detected ISO 639-1 language code. :returns: ``True`` if the languages are equivalent. TN)rNr4)expecteddetectedrJs ris_language_equivalentrRs48    )E   2U!22r:c32K|]}t|ywNr).0ss r rXs/,5q"I_NORMALIZED_SUPERSETScni}tD])}td|D}|D]}||t|<+|S)z1Build the bidirectional equivalence lookup index.c32K|]}t|ywrTrU)rVns rrXz%_build_bidir_index..sEu!215urY)r;rIr)r7rJnormedr s r_build_bidir_indexr_sB(*F%EuEED4:F*40 1& Mr:_NORMALIZED_BIDIRc||duS|yt|}t|}||k(ry|tvr |t|vry|tvxr |t|vS)aCheck whether *detected* is an acceptable answer for *expected*. Acceptable means: 1. Exact match (after normalization), OR 2. Both belong to the same bidirectional byte-order group, OR 3. *detected* is a known superset of *expected*. :param expected: The expected encoding name, or ``None`` for binary files. :param detected: The detected encoding name, or ``None``. :returns: ``True`` if the detection is acceptable. NFT)rr`rZ)rPrQnorm_expnorm_dets r is_correctrds}4&x0H&x0H8$$5Fx5P)P )) 8 -h7 7r:c^tjd|}djd|DS)z4NFKD-normalize *text* and strip all combining marks.NFKDrc3LK|]}tj|r|ywrT) unicodedata combining)rVcs rrXz#_strip_combining..sCd+*?*?*B1ds$$)rh normalizejoin)textnfkds r_strip_combiningros)   .D 77CdC CCr:)¤€)rqrpzfrozenset[tuple[str, str]]_EQUIVALENT_SYMBOL_PAIRScR||k(ry||ftvryt|t|k(S)uReturn True if characters *a* and *b* are functionally equivalent. Equivalent means: - Same character, OR - Same base letter after stripping combining marks, OR - An explicitly listed symbol equivalence (e.g. ¤ ↔ €) T)rrro)abs r_chars_equivalentrvs4 Av 1v)) A "21"5 55r:c4||duS|yt|}t|}||k(ry |j|}|j|}||k(ryt |t |k7ryt dt ||dDS#ttf$rYywxYw)uCheck whether *detected* produces functionally identical text to *expected*. Returns ``True`` when: 1. *detected* is not ``None`` and both encoding names normalize to the same codec, OR 2. Decoding *data* with both encodings yields identical strings, OR 3. Every differing character pair is functionally equivalent: same base letter after stripping combining marks, or an explicitly listed symbol equivalence (e.g. ¤ ↔ €). Returns ``False`` if *detected* is ``None``, either encoding is unknown, or either encoding cannot decode *data*. :param data: The raw byte data that was detected. :param expected: The expected encoding name, or ``None`` for binary files. :param detected: The detected encoding name, or ``None``. :returns: ``True`` if decoding with *detected* yields functionally identical text to decoding with *expected*. NFTc3:K|]\}}t||ywrT)rv)rVrtrus rrXz*is_equivalent_detection..4sX3W41a A&3Ws)strict)rdecodeUnicodeDecodeErrorr lenallzip)datarPrQrbrctext_exptext_dets ris_equivalent_detectionrs.4&x0H&x0H8;;x(;;x(8 8}H % X3xRV3WX XX  ,s"BBB)r r6returnr6)r7rrr)rr-)rPr6rQr6rbool)rP str | NonerQrrr)rmr6rr6)rtr6rur6rr)rbytesrPrrQrrr)__doc__ __future__rr rhchardet.pipelinerrrIr.__annotations__r1r9r;rGrMrNrRitemsrZr_r`rdrorrrvr)subset supersetss00rrs!&# * >*( Y0 1( y-12(9gY'( i $ (  9i[ ! (  I{G, - (G%567(i!1 23( i()(I~./( i "( Yz "(9RS(I~/?@A( I012!($)^,-%(&)^,-'((^,-^,-^,-^,-^,-n-.3( $D       ! &N   $5162".I-J*J3,'__. 4/ F#Y/,5/&/ 400B/C,C FD8A84 6 .Y .Y%.Y1;.Y .Yq4s F8