iZUdZddlmZddlmZddlmZddlmZm Z m Z m Z ddl m Z mZmZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,ede dZ-edddZ.edddZ/dZ0e1hdZ2de3d<e1hdZ4de3d<e1hdZ5de3d <e1hd!Z6de3d"<e4e5e6d#Z7d$e3d%<e1hd&Z8de3d'<d8d(Z9d)Z:d*Z;d+Z d9d.Z? d:d/Z@ d;d0ZA d;d1ZBd2ZCdu@Pipeline orchestrator — runs all detection stages in sequence.) annotations)DEFAULT_MAX_BYTES) EncodingEra) BigramProfilehas_model_variantsinfer_languagescore_best_language)DETERMINISTIC_CONFIDENCE HIGH_BYTESDetectionResultPipelineContext) detect_ascii) is_binary) detect_bom)resolve_confusion_groups)detect_escape_encoding)detect_markup_charset)score_candidates)compute_lead_byte_diversitycompute_multibyte_byte_coveragecompute_structural_score) detect_utf8)detect_utf1632_patterns)filter_by_validity)REGISTRY EncodingInfoget_candidatesNencoding confidencelanguageutf-8g? windows-1252g333333?> iso-8859-1 iso-8859-15r#zfrozenset[str]_COMMON_LATIN_ENCODINGS>.zfrozenset[int]_ISO_8859_10_DISTINGUISHING>r'r(r*r+r,r-r/r0r1r3r4r5r6r7r8r:r;r<r=r>r?r@rArIrR_ISO_8859_14_DISTINGUISHING>rXrYrZr[_WINDOWS_1254_DISTINGUISHING)z iso-8859-10z iso-8859-14z windows-1254zdict[str, frozenset[int]]_DEMOTION_CANDIDATES> r'r(r+r8_KOI8_T_DISTINGUISHINGc^tj|ytfd|D S)auReturn True if encoding is a demotion candidate with no distinguishing bytes. Checks whether any non-ASCII byte in *data* falls in the set of byte values that decode differently under the given encoding vs iso-8859-1. If none do, the data is equally valid under both encodings and there is no byte-level evidence for preferring the candidate encoding. Fc32K|]}|dkDs |vywN).0bdistinguishings m/Users/bowang/.openclaw/workspace/ChatDev/.venv/lib/python3.12/site-packages/chardet/pipeline/orchestrator.py z!_should_demote..sA1D1&s  )r`getany)rdatarqs @rr_should_demoterws2*--h7NAAA AAg?gffffff?c4g}|D]}|jrt|||}||j|j<|tkr@|j 0t |t |jdtz |_|j tkrt||||j }||j|j<|tkr|j tk\rt|||}|tkr|j!| t#|S)aEliminate CJK multi-byte candidates that lack genuine multi-byte structure. Four checks are applied in order to each multi-byte candidate: 1. **Structural pair ratio** (valid_pairs / lead_bytes) must be >= ``_CJK_MIN_MB_RATIO``. Catches files with many orphan lead bytes. 2. **Minimum non-ASCII byte count**: the data must contain at least ``_CJK_MIN_NON_ASCII`` bytes > 0x7F. Tiny files with 1-5 high bytes can accidentally form perfect pairs and score 1.0 structurally. 3. **Byte coverage** (non-ASCII bytes in valid multi-byte sequences / total non-ASCII bytes) must be >= ``_CJK_MIN_BYTE_COVERAGE``. Latin text has many high bytes that are NOT consumed by multi-byte pairs; genuine CJK text has nearly all high bytes accounted for. 4. **Lead byte diversity**: the number of distinct lead byte values in valid pairs must be >= ``_CJK_MIN_LEAD_DIVERSITY``. Genuine CJK text draws from a wide repertoire of lead bytes; European false positives cluster in a narrow band (e.g. 0xC0-0xDF for accented Latin). Returns the filtered candidate list. Structural scores are cached in ``ctx.mb_scores`` for reuse in Stage 2b. N)non_ascii_count) is_multibyter mb_scoresname_CJK_MIN_MB_RATIOr}len translater _CJK_MIN_NON_ASCIIr mb_coverage_CJK_MIN_BYTE_COVERAGE_CJK_DIVERSITY_MIN_NON_ASCIIr_CJK_MIN_LEAD_DIVERSITYappendtuple)rvvalid_candidatesctxgatedencmb_score byte_coveragelead_diversitys rr_gate_cjk_candidatesrs:!#E   /c3?H&.CMM#(( #++""*&)$i#dnnT:6V2W&W#""%77;c30C0CM)6COOCHH %55""&BB!= 0.95) receive a confidence boost proportional to coverage. When nearly all non-ASCII bytes form valid multi-byte pairs, the structural evidence is strong and should increase the candidate's ranking relative to single-byte alternatives whose bigram models may score higher on small samples. Note: boosted confidence values may exceed 1.0 and are used only for relative ranking among candidates. ``run_pipeline`` clamps all confidence values to [0.0, 1.0] before returning to callers. c38K|]\}}|vs |ywNrn)ror_sc enc_lookups rrrsz/_score_structural_candidates..5s&*;YT3tz?Q 4*;s  c3:K|]}|jr|ywr)r~)roes rrrsz/_score_structural_candidates..8sJ#3a1>>#3sgffffff?rc|jSr)r xs rrz._score_structural_candidates..Isq||rxTkeyreverse) r~rrlistrrrrtrr r r!sort) rvstructural_scoresrrrvalid_mb single_byteresultsboostedrcoveragers @rr_score_structural_candidatesrs .&6H%5!&&!)%5HJ*;HJ#3JJK#D*CH*C{*CDEG&(G ;<::3??&&qzz373 t  NNZZ ||q8|<ZZ  NN1  LL+TL: N-Is DDctt|dkDr|djt|dj|rw|dj}|ddD]`}|jtvs|Dcgc]}|j|k7s||us|}}|Dcgc]}|j|k(s|}}|g||cS|Scc}wcc}w)aDemote niche Latin encodings when no distinguishing bytes are present. Some bigram models (e.g. iso-8859-10, iso-8859-14, windows-1254) can win on data that contains only bytes shared with common Western Latin encodings. When there is no byte-level evidence for the winning encoding, promote the first common Western Latin candidate to the top and push the demoted encoding to last. rrN)rrrwr&)rvrdemoted_encodingrrothersdemoted_entriess rr_demote_niche_latinrMs G q AJ   + 71:.. 5"1:..Azz44&&!!**8H*HQVWZAw/6"XgGW9W1g"X5F5_55  N #Ys%B0:B0?B0 B5B5c|r|djdk7r|Sd}t|D]\}}|jdk(s|}n||Std|Dr-||}t|Dcgc] \}}||k7s |}}}|g|S|Scc}}w)aPromote KOI8-T over KOI8-R when Tajik-specific bytes are present. KOI8-T and KOI8-R share the entire 0xC0-0xFF Cyrillic letter block, making statistical discrimination difficult. However, KOI8-T maps 12 bytes in 0x80-0xBF to Tajik-specific Cyrillic letters where KOI8-R has box-drawing characters. If any of these bytes appear, KOI8-T is the better match. rzkoi8-rNzkoi8-tc38K|]}|dkDs |tvywrl)ri)rorps rrrsz!_promote_koi8t..s A1D1& &s  )r enumerateru)rvr koi8t_idxir koi8t_resultrs rr_promote_koi8tris gaj))X5I'"1 :: !I # A AAy) )' 2E 21a9n! 2E&v&& NFs ( B6Bic|dk(r|S |j|djddS#ttf$rYywxYw)aPDecode data from encoding and re-encode as UTF-8 for language scoring. Returns None if the encoding is unknown. For UTF-8, returns data as-is. Uses ``errors="ignore"`` because the data already passed byte-validity filtering for the detected encoding; any residual invalid bytes are irrelevant for language scoring. r"ignore)errors surrogatepassN)decodeencode LookupError TypeError)rvrs rr_to_utf8rsY7 {{8H{5<< O=    #s #-??c2g}d}d}|D] }|j|jt|j}|?|r=t|jr(| t |}t ||j|\}}|R|rPtdrEt ||j}|r-||jdk7r t |}t |d|\}}|2|jt|j|j||j| |S)aFill in language for results missing it. Tier 1: single-language encodings via hardcoded map (instant). Tier 2: multi-language encodings via statistical bigram scoring (lazy). Tier 3: decode to UTF-8, score against UTF-8 language models (universal fallback). N)profiler"r) r!rrrrr rrr r ) rvrfilledr utf8_profileresultlang_ utf8_datas rr_fill_languagers%'F$(G)-L ?? "v'B!&//2D|);FOO)L?+D1G-dFOOWU4|);G)D$T6??; #+v'/I'4Y'? 1!7LGAt #!'#)#4#4!% f78 MrxcJt||}t||}t||S)zGApply confusion resolution, niche Latin demotion, and KOI8-T promotion.)rrr)rvrs rr_postprocess_resultsrs) 'tW5G!$0G $ ((rxct}|d|}|stgSt|}||gSt|}||gSt |}|?|j 3t j|j }|||jzr|gSt|}|t||rtgSt|} | | gSt|} | | gS||gSt|} t|| } | st gSt#|| |} | st gSg} | D]f}|j$s|j&j|j(}| t+|||}|dkDsJ| j-|j(|fh| r?| j/dd| d\}}|t0k\rt3|| | |}t5||St7t9|t;| }|st gSt5||S)zBCore pipeline logic. Returns list of results sorted by confidence.N) max_bytesrc |dS)Nrrnrs rrrz$_run_pipeline_core..0sQqTrxTrr)r _EMPTY_RESULTrrrrrrterarr_BINARY_RESULTrrrr_FALLBACK_RESULTrr~rrrrr _STRUCTURAL_CONFIDENCE_THRESHOLDrrrrr)rv encoding_erarr bom_resultutf1632_result escape_resultenc_info utf8_precheck markup_result ascii_result candidatesrrrscorer best_scorers rr_run_pipeline_corers9  C   D D!J| -T2N!+40M ]%;%;%G<< 6 67  |hll:!? " %M49!E *$/M  %L~  -J)$ ;  !!,D2BCH  !!24   MM%%chh/E}0sC@s{!((#((E):;  >4@)!, : 9 92')93G(g6 6#D%0@*ABCG  !! g ..rxc t|||}t|dt|}|s d}t||Dcgc]H}|jdkDr5t |j t|jd|jn|Jc}Scc}w)a,Run the full detection pipeline. :param data: The raw byte data to analyze. :param encoding_era: Filter candidates to a specific era of encodings. :param max_bytes: Maximum number of bytes to process. :returns: A list of :class:`DetectionResult` sorted by confidence descending. Nz/pipeline must always return at least one resultg?) rr_LANG_SCORE_MAX_BYTES RuntimeErrorr r rminr!)rvrrrmsgrs rr run_pipeliner@s!|Y?GT"8#897CG ?3  A <<#   C c$:AJJG     sA B)rstrrvbytesreturnbool)rvrrtuple[EncodingInfo, ...]rr rr) rvrrzlist[tuple[str, float]]rrrr rlist[DetectionResult])rvrrrrr)rvrrrrz bytes | None)rvrrrrintrr)I__doc__ __future__rchardet._utilsr chardet.enumsrchardet.modelsrrrr chardet.pipeliner r r r chardet.pipeline.asciirchardet.pipeline.binaryrchardet.pipeline.bomrchardet.pipeline.confusionrchardet.pipeline.escaperchardet.pipeline.markuprchardet.pipeline.statisticalrchardet.pipeline.structuralrrrchardet.pipeline.utf8rchardet.pipeline.utf1632rchardet.pipeline.validityrchardet.registryrrrrrrr frozensetr&__annotations__rUr\r_r`rirwrrrrrrrrrrrrrrrrnrxrrrsF",%  0-+?:99 .<8CC 6  TDQ # t $( +4+/8/1/^1n/8 "/^"T09(0n/.03/*3L* B$   "3 3.3 3 3l- -.-/-  -  -`  "8  "B$( (/((V) ) "))'i/ i/i/i/ i/^'  rx