cRG ddlmZddlmZddlmZmZddlmZm Z m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZGddZGd d eZGd d eZGd deZGddeZ GddeZ!GddeZ"GddeZ#GddeZ$eddee%dee%de&fdZ'ed d'd"e%d#e(d$e&de(fd%Z)d&S)() lru_cache) getLogger)ListOptional)COMMON_SAFE_ASCII_CHARACTERSTRACEUNICODE_SECONDARY_RANGE_KEYWORD)is_accentuatedis_asciiis_case_variableis_cjk is_emoticon is_hangul is_hiragana is_katakanais_latinis_punctuation is_separator is_symbolis_thaiis_unprintable remove_accent unicode_rangecVeZdZdZdedefdZdeddfdZd dZe de fdZ dS) MessDetectorPluginzy Base abstract class used for mess detection plugins. All detectors MUST extend and implement given methods. characterreturnct)z@ Determine if given character should be fed in. NotImplementedErrorselfrs 7/usr/lib/python3/dist-packages/charset_normalizer/md.pyeligiblezMessDetectorPlugin.eligible$ "!Nct)z The main routine to be executed upon character. Insert the logic in witch the text would be considered chaotic. r r"s r$feedzMessDetectorPlugin.feed*s "!r'ct)zB Permit to reset the plugin to the initial state. r r#s r$resetzMessDetectorPlugin.reset1r&r'ct)z Compute the chaos ratio based on what your feed() has seen. Must NOT be lower than 0.; No restriction gt 0. r r+s r$ratiozMessDetectorPlugin.ratio7s "!r'rN) __name__ __module__ __qualname____doc__strboolr%r)r,propertyfloatr.r'r$rrs "#"$"""" "c"d"""""""" "u"""X"""r'rcZeZdZd dZdedefdZdeddfdZd dZe de fdZ dS) TooManySymbolOrPunctuationPluginrNcLd|_d|_d|_d|_d|_dS)NrF)_punctuation_count _symbol_count_character_count_last_printable_char_frenzy_symbol_in_wordr+s r$__init__z)TooManySymbolOrPunctuationPlugin.__init__As0'("#%&37!,1###r'rc*|SN isprintabler"s r$r%z)TooManySymbolOrPunctuationPlugin.eligibleI$$&&&r'c(|xjdz c_||jkro|tvrft|r|xjdz c_nF|dur0t |r!t|dur|xjdz c_||_dS)NrF) r>r?rrr<isdigitrrr=r"s r$r)z%TooManySymbolOrPunctuationPlugin.feedLs " 2 2 2!===i(( (''1,'''!!##u,,i((- **e33""a'""$-!!!r'c0d|_d|_d|_dSNr)r<r>r=r+s r$r,z&TooManySymbolOrPunctuationPlugin.reset^s "# !r'c^|jdkrdS|j|jz|jz }|dkr|ndS)Nrg333333?)r>r<r=)r#ratio_of_punctuations r$r.z&TooManySymbolOrPunctuationPlugin.ratiocsK  A % %3  #d&8 8  !'"(_accentuated_countr+s r$rAz!TooManyAccentuatedPlugin.__init__ps%&'(r'rc*|SrC)isalphar"s r$r%z!TooManyAccentuatedPlugin.eligiblets  """r'ch|xjdz c_t|r|xjdz c_dSdSNr)r>r rTr"s r$r)zTooManyAccentuatedPlugin.feedwsJ " ) $ $ )  # #q ( # # # # ) )r'c"d|_d|_dSrKrSr+s r$r,zTooManyAccentuatedPlugin.reset}s !"#r'cd|jdks |jdkrdS|j|jz }|dkr|ndS)NrrMgffffff?rS)r#ratio_of_accentuations r$r.zTooManyAccentuatedPlugin.ratiosI  A % %)>)B)B3'+'>AV'V(=(E(E$$3Nr'r/rOr8r'r$rQrQos))))###$####)c)d)))) $$$$OuOOOXOOOr'rQcZeZdZd dZdedefdZdeddfdZd dZe de fdZ dS) UnprintablePluginrNc"d|_d|_dSrK)_unprintable_countr>r+s r$rAzUnprintablePlugin.__init__s'(%&r'rcdSNTr8r"s r$r%zUnprintablePlugin.eligibletr'cdt|r|xjdz c_|xjdz c_dSrX)rr`r>r"s r$r)zUnprintablePlugin.feeds@ ) $ $ )  # #q ( # # "r'cd|_dSrK)r`r+s r$r,zUnprintablePlugin.resets"#r'c@|jdkrdS|jdz|jz S)NrrMr[)r>r`r+s r$r.zUnprintablePlugin.ratios+  A % %3'!+t/DDDr'r/rOr8r'r$r^r^s''''#$#c#d#### $$$$EuEEEXEEEr'r^cZeZdZd dZdedefdZdeddfdZd dZe de fdZ dS) SuspiciousDuplicateAccentPluginrNc0d|_d|_d|_dSrK_successive_countr>_last_latin_characterr+s r$rAz(SuspiciousDuplicateAccentPlugin.__init__s &'%&48"""r'rcH|ot|SrC)rVrr"s r$r%z(SuspiciousDuplicateAccentPlugin.eligibles!  "":x ':'::r'cl|xjdz c_|jt|rt|jrr|r)|jr|xjdz c_t |t |jkr|xjdz c_||_dSrX)r>rlr isupperrkrr"s r$r)z$SuspiciousDuplicateAccentPlugin.feeds "  & 2y)) 3t9:: 3  "" ,t'A'I'I'K'K ,&&!+&&Y''=9S+T+TTT&&!+&&%."""r'c0d|_d|_d|_dSrKrjr+s r$r,z%SuspiciousDuplicateAccentPlugin.resets !" !%)"""r'c@|jdkrdS|jdz|jz S)NrrMrH)r>rkr+s r$r.z%SuspiciousDuplicateAccentPlugin.ratios+  A % %3&*d.CCCr'r/rOr8r'r$rhrhs9999 ;#;$;;;; /c /d / / / /**** DuDDDXDDDr'rhcZeZdZd dZdedefdZdeddfdZd dZe de fdZ dS) SuspiciousRangerNc0d|_d|_d|_dSrK)"_suspicious_successive_range_countr>_last_printable_seenr+s r$rAzSuspiciousRange.__init__s 78/%&37!!!r'rc*|SrCrDr"s r$r%zSuspiciousRange.eligiblerFr'cD|xjdz c_|st|s |tvr d|_dS|j ||_dSt |j}t |}t ||r|xjdz c_||_dSrX)r>isspacerrrvr is_suspiciously_successive_rangeru)r#runicode_range_aunicode_range_bs r$r)zSuspiciousRange.feeds "      i(( 888(,D % F  $ ,(1D % F)6t7P)Q)Q)6y)A)A +O_ M M 9  3 3q 8 3 3$-!!!r'c0d|_d|_d|_dSrK)r>rurvr+s r$r,zSuspiciousRange.resets !23/$(!!!r'cT|jdkrdS|jdz|jz }|dkrdS|S)NrrMrHg?)r>ru)r#ratio_of_suspicious_range_usages r$r.zSuspiciousRange.ratiosH  A % %3  3a 7  !2"' +S 0 03..r'r/rOr8r'r$rsrss8888 '#'$''''.c.d.....))))  /u / / /X / / /r'rscZeZdZd dZdedefdZdeddfdZd dZe de fdZ dS) SuperWeirdWordPluginrNcd|_d|_d|_d|_d|_d|_d|_d|_d|_dS)NrF) _word_count_bad_word_count_foreign_long_count_is_current_word_bad_foreign_long_watchr>_bad_character_count_buffer_buffer_accent_countr+s r$rAzSuperWeirdWordPlugin.__init__sO !$%() */!). %&)*! )*!!!r'rcdSrbr8r"s r$r%zSuperWeirdWordPlugin.eligible rcr'c|r|xj|z c_t|r|xjdz c_|jdur|t |dust|r\t |durKt|dur:t|dur)t|durt|durd|_dS|jsdS| st|st|r"|jr|xjdz c_t|j}|xj|z c_|dkre|j|z dkrd|_t|jdr6|jdr|xjdz c_d|_|dkr|jr|xjdz c_d|_|jr9|xjdz c_|xjt|jz c_d|_d|_d|_d |_dS|d vr>|dur*t/|rd|_|xj|z c_dSdSdSdS) NrFTg(\?rr>_-<=>|~)rVrr rrrrrrrrryrrrlenr>rrorrrrIr)r#r buffer_lengths r$r)zSuperWeirdWordPlugin.feed s       LLI %LLi(( /))Q.))(E11i((E11^I5N5N19%%..i((E11 **e33 **e33I&&%//+/( F|  F     " )#<#<" &@LY@W@W" &l" &    !  !$T\!2!2M  ! !] 2 ! !!!,}->>)),1)',D $DL()D % % % @ @ @!!##u,,)$$-)-D % LLI %LLLL A @,,,,r'cvd|_d|_d|_d|_d|_d|_d|_d|_dS)NrFr)rrrrrr>rrr+s r$r,zSuperWeirdWordPlugin.resetBsG $)!#(   !$%!#$   r'cP|jdkr |jdkrdS|j|jz S)N rrM)rrrr>r+s r$r.zSuperWeirdWordPlugin.ratioLs3  r ! !d&>!&C&C3(4+@@@r'r/rOr8r'r$rrs + + + +#$4&c4&d4&4&4&4&l%%%%AuAAAXAAAr'rc^eZdZdZd dZdedefdZdeddfdZd dZ e de fd Z dS) CjkInvalidStopPluginu GB(Chinese) based encoding often render the stop incorrectly when the content does not fit and can be easily detected. Searching for the overuse of '丅' and '丄'. rNc"d|_d|_dSrK_wrong_stop_count_cjk_character_countr+s r$rAzCjkInvalidStopPlugin.__init__Zs&')*!!!r'rcdSrbr8r"s r$r%zCjkInvalidStopPlugin.eligible^rcr'ct|dvr|xjdz c_dSt|r|xjdz c_dSdS)N>丄丅r)rrrr"s r$r)zCjkInvalidStopPlugin.feedasZ  & &  " "a ' " " F )   +  % % * % % % % + +r'c"d|_d|_dSrKrr+s r$r,zCjkInvalidStopPlugin.reseths!"$%!!!r'c:|jdkrdS|j|jz S)NrM)rrr+s r$r.zCjkInvalidStopPlugin.ratiols&  $r ) )3%(AAAr'r/) r0r1r2r3rAr4r5r%r)r,r6r7r.r8r'r$rrTs ++++#$+c+d++++&&&&BuBBBXBBBr'rcZeZdZd dZdedefdZdeddfdZd dZe de fdZ dS) ArchaicUpperLowerPluginrNchd|_d|_d|_d|_d|_d|_d|_dS)NFrT)_buf_character_count_since_last_sep_successive_upper_lower_count#_successive_upper_lower_count_finalr>_last_alpha_seen_current_ascii_onlyr+s r$rAz ArchaicUpperLowerPlugin.__init__ts? 45,23*890%&/3)-   r'rcdSrbr8r"s r$r%z ArchaicUpperLowerPlugin.eligiblercr'c|ot|}|du}|r|jdkrt|jdkr4|dur|jdur|xj|jz c_d|_d|_d|_d|_|xj dz c_ d|_dS|jdurt|durd|_|j| r|j s-| rB|j r)|jdur|xjdz c_d|_nd|_nd|_|xj dz c_ |xjdz c_||_dS)NFr@rTrH) rVr rrIrrrrrr>r roislower)r#r is_concerned chunk_seps r$r)zArchaicUpperLowerPlugin.feeds ((**J/? /J/J  E)  =AA4::%%''500,558868823D .34D 0$(D !DI  ! !Q & ! !'+D $ F  #t + +0C0Cu0L0L',D $  ,!!## "(=(E(E(G(G "!!## "(,(=(E(E(G(G "9$$66!;66 %DII $DII!  " ,,1,, )r'chd|_d|_d|_d|_d|_d|_d|_dS)NrFT)r>rrrrrrr+s r$r,zArchaicUpperLowerPlugin.resets? !/0,-.*340 $ #'   r'c:|jdkrdS|j|jz S)NrrM)r>rr+s r$r.zArchaicUpperLowerPlugin.ratios&  A % %37$:OOOr'r/rOr8r'r$rrss . . . .#$(*c(*d(*(*(*(*T((((PuPPPXPPPr'r)maxsizer{r|rc||dS||krdSd|vrd|vrdSd|vsd|vrdSd|vsd|vr d|vsd|vrdS|d|d}}|D]}|tvr ||vrdS|dv|dv}}|s|r d |vsd |vrdS|r|rdSd |vsd |vrd |vsd |vrdS|d ks|d krdSd |vs d |vs|d vr|d vrd |vsd |vrdSd|vsd|vrdSdS)za Determine if two Unicode range seen next to each other can be considered as suspicious. NTFLatin Emoticons Combining )HiraganaKatakanaCJKHangulz Basic Latin)rr PunctuationForms)splitr )r{r|keywords_range_akeywords_range_belrange_a_jp_charsrange_b_jp_charss r$rzrzs/"9t/))u/!!g&@&@uo%%)G)Gu ?""g&@&@&&+*H*Hu)8)>)> **S!!' 0 0 0  ! ! !55 "   33 ' ,   E_$<$<u,u?""h/&A&A O # #u'?'?5 m + +-/O/O5   E_$<$<333 7 7 7 O + +}/O/O5 o % %O)C)C5 4r'i皙?Fdecoded_sequencemaximum_thresholddebugc ZdtD}t|dz}d}|dkrd}n |dkrd}nd}t|d zt |D]m\}}|D],} | |r| |-|d kr ||zd ks ||dz kr!td |D}||krnn|rtd } | td |d|d|t|dkrL| td|dd| td|dd|D],} | t| j d| j -t|dS)zw Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier. c"g|] }| Sr8r8).0md_classs r$ zmess_ratio..s++++ +++r'rrMi rr rc3$K|] }|jV dSrC)r.)rdts r$ zmess_ratio..%s$!?!?r"(!?!?!?!?!?!?r'charset_normalizerzIMess-detector extended-analysis start. intermediary_mean_mess_ratio_calc=z mean_mess_ratio=z maximum_threshold=rzStarting with: Nz Ending with: iz: )r__subclasses__rzipranger%r)sumrlogr __class__r.round) rrr detectorslengthmean_mess_ratio!intermediary_mean_mess_ratio_calcrindexdetectorloggerrs r$ mess_ratiors1++#5#D#D#F#F+++I&''!+F O ||13)) 4,.)),/) 04 7vGG   5! ) )H  ++ ) i((( AII%"CCqHH fqj !!?!?Y!?!?!???O"333 =/00  51R 5 5et 5 5!2 5 5     2 % % JJuG0@"0EGG H H H JJuG.>suu.EGG H H H = =B JJu;;;; < < < < ! $ $$r'N)rF)* functoolsrloggingrtypingrrconstantrr r utilsr r r rrrrrrrrrrrrrrr:rQr^rhrsrrrr4r5rzr7rr8r'r$rs!!!!!!!! (""""""""D,L,L,L,L,L'9,L,L,L^OOOOO1OOO4EEEEE*EEE0"D"D"D"D"D&8"D"D"DJ1/1/1/1/1/(1/1/1/hWAWAWAWAWA-WAWAWAtBBBBB-BBB>IPIPIPIPIP0IPIPIPX 4Cc]C5=c]C CCCCL 4IN4%4%4%.34%BF4% 4%4%4%4%4%4%r'