Differences

This shows you the differences between two versions of the page.

--- programming:multilingual_support [2025/04/18 16:49] – [Language Detection] fasttext repo with more recent updates karelkubicek
+++ programming:multilingual_support [2026/08/27 15:26] (current) – Rebuild the stub into a corpus-backed instrument page: 53/1080 crawled papers handled a non-English language, dated detector/translator families, LibreTranslate compose digest-pinned, published html-lang checker, keep-it-in-the-source-language classificat karel.kubicek.claude
@@ Line 1: / Line 1: @@
 ====== Multilingual Support ======
-Much of the web research focuses on English websites, often resonating with the target group of English-written publications. However, this focus causes our research field to neglect other target audiences, especially those where security and privacy issues may have larger implications than in the Western world. This limitation in research scope was also identified by Mhaidli et al. in Sec. 4.4.2 of {[mhaidli2023researchers]}.
+You are about to crawl sites that are not in English, or classify text that is not. This page is the **instrument**: detect the language you actually fetched, switch the site into the language you meant, translate only when you have to, and keep the rest of the pipeline in the source language when you can.
-Below, we cover three aspects of multilingual crawling: language detection, machine translation, and full-pipeline multilingual support. Additionally, you should consider sampling websites from an [[Design:Website selection|appropriate list with local data]] and using other data sources that better fit the hypothetical user profiles of websites.
+The field mostly does not. Of the **1,080 crawled papers with a ''crawlConfig''** in this corpus (seven venues, 2010–2026, 5,859 extracted papers), **136 (12.6%)** name any language at all. **77 of those 136** name only English — against the schema's own rule that English-only stays empty. **53 papers (4.9% of 1,080)** handled a non-English natural language. Sampling from a local list is a different page ([[Design:Website selection]], [[Programming:CrUX]]); this one starts after you have URLs.
-===== Language Detection =====
+The English-centric gap is also a publishing fact. Mhaidli et al. recorded it as a tool and community problem, not a crawling one {[mhaidli2023researchers]}:
-Several libraries can automatically detect the language of input text, such as [[https://github.com/shuyo/language-detection|langdetect]], Google's [[https://github.com/google/cld3|CLD3]], and Facebook's [[https://github.com/simon-ging/fasttext-numpy2|fasttext]]. A useful comparison of these tools, along with code examples, can be found at https://modelpredict.com/language-identification-survey.
+> most work that is being done is only focused on privacy policies that are in English. Yeah, I see that as a big problem personally
-Key takeaways indicate that the clear winner is fasttext:
+If you read one measurement paper first, make it Bhuiyan et al. {[bhuiyan2025_visitors]} (IMC 2025): 120,000 sites across 12 non-Latin-script languages, crawled through country VPNs, language decided on **visible text**, not on ''<html lang>''.
-  * It is the most accurate, followed only by the considerably slower langdetect.
-  * While pycld2 is about twice as fast, it loses 10% accuracy (98% vs. 87%).
-  * If a small memory footprint is required (e.g., for parallel crawls), you can use fasttext-compressed, though it has slightly worse accuracy and is marginally slower.((If the memory footprint over multiple parallel crawlers is an issue, consider moving the language detection to a single separate service serving all crawlers.))
-This performance assessment might be skewed by evaluation on specific datasets, as fasttext uses one of the benchmarks as training data. However, fasttext also performs best on other benchmarks. Hosseini et al. {[hosseini2021unifying]} reported the best performance on long text with the langdetect library. Thus, you might want to adopt Hosseini et al.'s approach: running an ensemble of multiple methods and selecting the top result. Note that this approach increases runtime and memory usage compared to the worst-performing method.
+<WRAP important>
+**''<html lang>'' is a publisher declaration, not a measurement.** Trust visible text (a Unicode-script heuristic, then a detector). Mixed-language pages are the common case, not the edge: Bhuiyan et al. kept a site if 50% of visible text was in the target language, and the median site was **88%** native {[bhuiyan2025_visitors]}. Translating everything into English is the expensive path the corpus actually takes (Google Translate is **49 of 135** papers that used a language/translation tool). The 2026 method that is current for classification is **stay in the source language** {[nenadic2026_swiss]} — validated per language, because their F1 above 0.90 holds for most language/disclosure pairs, not all.
+</WRAP>
-Avoid using the language definition provided by websites in the ''<html>'' tag. Most non-English websites incorrectly indicate English (EN) because of CMS or template defaults.
+===== What this page is for =====
-===== Machine Translation =====
+  * **Here:** detect language, switch locale, translate or don't, keep keywords and classifiers in-language. The LibreTranslate compose is the self-hosted translation sidecar.
+  * **[[Design:Website selection]] / [[Programming:CrUX]]:** a popularity list is not a language list. LangCrUX is CrUX **plus** a language filter {[bhuiyan2025_visitors]}.
+  * **[[Design:Crawling location]]:** many sites pick a language from the client IP. A US cloud VM fetches the English CDN site. VPN localisation belongs there; this page only names it as a language switch.
+  * **[[Programming:Interaction]]:** clicking a language switcher is a click. Building the keyword list for that click is here.
+  * **[[Programming:Docker]]:** pin the image digest and size ''/dev/shm''. The compose below is a translator, not a crawler; ''check_crawler_compose.py --all'' still digest-pins it.
+  * **[[Design:Website classification]]:** topic taxonomies, including Yahoo's multilingual production classifier. Not language ID.
-A straightforward way to support multiple languages in your study is to use machine translation. While many paid translation APIs exist (e.g., Google Translate, DeepL), they quickly become prohibitively expensive at crawl scale. Although Google Translate offers a free API, its Terms of Service prohibit automated processing, and its reliability due t obot detection varies. Therefore, this section focuses on open-source, self-hosted translation methods.
+===== What to read first =====
-Note that these methods are often slow.((In our experience, LibreTranslate utilizes 20-50% of computational resources during crawls. --Karel Kubicek)) Using LLMs is even slower, so you might want to explore the next section on keeping the whole process multilingual, which is significantly more efficient.
+^ Paper ^ Why ^
+| Bhuiyan et al., IMC 2025, //Not All Visitors are Bilingual// {[bhuiyan2025_visitors]} | How to **sample** a multilingual web: CrUX × Unicode-script filter × country VPN. Mixed pages, ''lang'' attributes as accessibility metadata. |
+| Bouhoula, Kubicek et al., USENIX Sec 2024 {[bouhoula2024_automated]} | The self-hosted pipeline this page used to recommend without a paper: ''langdetect'' + LibreTranslate on **11 EU languages**, 97k sites. Greek was dropped because the translator failed on short button text. |
+| Nenadić, Rodriguez and Calandrino, PoPETs 2026 {[nenadic2026_swiss]} | Classification **without** translation: one LLM inference over German, French, Italian and English privacy policies. F1 above 0.90 for **most** language/disclosure pairs on 120 expert-annotated policies (30 per language), not all pairs. |
+| Tang et al., PoPETs 2024 {[tang2024_automatic]} | Detection at 103-language scale: Lingua, CLD3 as fallback on the 15.22% Lingua does not support, 99.29% identified. |
+| Hosseini et al., PoPETs 2021 {[hosseini2021unifying]} | **Not in this extraction** (in the bibliographic index; PDF fetched). The eight-library ensemble later papers copy. On long privacy-policy text, langdetect's Cramér's V with the ensemble-determined language was **0.97**, the highest of the eight. |
+| Degeling et al., NDSS 2019 {[degeling2019_value]} | 24 official EU languages on a consent crawl, before anyone in this corpus self-hosted a translator. |
-==== LibreTranslate (Argos) ====
+===== What the corpus actually reports =====
-[[https://github.com/LibreTranslate/LibreTranslate|LibreTranslate]] is an interface for the [[https://github.com/argosopentech/argos-translate/|Argos Translate]] project, simplifying the handling of multiple language models in parallel. We recommend starting with it using [[Programming:Docker]].
+Two signals, kept apart. ''crawlConfig.languages'' is "languages or locales explicitly handled; empty if only English or unstated." A used ''language-detection'' or ''translation'' tool is a different question, and **most of those papers did not crawl**.
-<file yaml docker-compose.yaml>
+Of **1,080** crawled papers with a ''crawlConfig'':
+^ Role of ''languages[]'' ^ Papers ^ Share of 1,080 ^
+| multilingual (two or more natural languages, or a count-phrase such as "103 languages") | 22 | 2.0% |
+| language-specific (exactly one non-English natural language) | 31 | 2.9% |
+| **handled non-English (union of the two)** | **53** | **4.9%** |
+| english-only (schema said leave empty) | 77 | 7.1% |
+| locale-knob (POSIX ''ru_RU.UTF-8'' and friends — a fingerprinting setting, not a language study) | 1 | 0.1% |
+| vague ("foreign-language websites", "multilingual webpages", …) | 4 | 0.4% |
+| programming (''languages: ["C"]'' — GitHub awesome-c, not a natural language) | 1 | 0.1% |
+| empty | 944 | 87.4% |
+Among the 136 non-empty rows, English still appears in **110 (80.9%)** because multilingual papers list it alongside the others. After English, the folded families are Chinese 25, German 17, French 12, Russian 11, Spanish 9. Residue of the language-name fold: **0**.
+**135 of 5,859 papers (2.3%)** used a language-detection or translation tool; **67 of those are crawled (6.0% of 1,120)**. Folded used-tool families, of the 135 (a paper may hit several):
+^ Family ^ Papers of 135 ^ Kind ^ Current in 2026? ^
+| Google Translate (API / Cloud / GNMT) | 49 | translation | **What the field uses.** Paid; ToS and cost are why the next rows exist. |
+| langdetect (incl. Shuyo's Java library) | 25 | detection | Live (PyPI port pushed 2025-03). Default in LibreTranslate for text ≥20 characters. |
+| googletrans / pygtrans | 8 | translation | Unofficial Google wrappers. ToS of the web UI forbid automated use. |
+| langid | 8 | detection | Live; older than langdetect's neural neighbours. |
+| DeepL | 7 | translation | Paid; same cost problem at crawl scale. |
+| fastText | 5 | detection | **Archived** 2024-03-22 (''facebookresearch/fastText''). |
+| CLD2 | 5 | detection | Historical; CLD3 replaced it in Chrome and is itself archived. |
+| NLLB / Opus-MT / EasyNMT | 5 | translation | Open neural MT. Appears late in the window. |
+| Lingua | 4 | detection | Live (''pemistahl/lingua-py'' pushed 2026-07-20). Short and mixed text. |
+| LibreTranslate / Argos | 3 | translation | Self-hosted. One of the three is {[bouhoula2024_automated]}. |
+| CLD3 | 3 | detection | **Archived** 2023-05-24 (''google/cld3''). |
+Five tool names did not fold ("Detect Language", "translation software", "Language Identification", "Translate", "language classifier [21]"). They stay residue in the report, not in this table.
+Exact-string "Google Translate" undercounts: the family is 49, plus 8 unofficial wrappers, **56 / 135 (41.5%)** once both are unioned. A page that ranked detectors by what a 2019 blog called the accuracy winner, without this table, would tell you to start with fastText. The corpus starts with langdetect; the live short-text library in 2026 is Lingua; fastText and CLD3 are archived.
+Year shape of the 53, of papers with a ''crawlConfig''. 2025–2026 is provisional ([[Literature:Corpus]]):
+^ Bucket ^ With ''crawlConfig'' ^ Named any language ^ Handled non-English ^
+| 2010–2013 | 98 | 2 | 2 |
+| 2014–2017 | 160 | 14 | 6 |
+| 2018–2021 | 297 | 39 | 14 |
+| 2022–2024 | 336 | 55 | 23 |
+| 2025–2026* | 189 | 26 | 8 |
+These are reporting rates. A paper can run langdetect and never write the word in the PDF.
+===== Language detection =====
+==== Dated, not ranked as current practice ====
+^ When ^ What people used ^ What to do with it now ^
+| 2010–2018 | Shuyo's Language Detection Library / ''langdetect'', langid, CLD2 | Still what **this corpus** uses. Fine for long page text in the languages those libraries cover. |
+| 2018–2023 | CLD3, fastText LID, occasional ensembles | CLD3 and fastText are archived. Do not start a 2026 crawl on an archived Facebook snapshot because a blog said it won a benchmark. |
+| 2021 | Hosseini et al.'s eight-library majority vote on privacy policies {[hosseini2021unifying]} | Copy the **ensemble idea** for a gold set. Do not run eight detectors in the inner loop of a crawl. Langdetect's Cramér's V with the determined language was 0.97 on that long-document corpus. |
+| 2024–2026 | Lingua, then CLD3 as fallback {[tang2024_automatic]}; Unicode-script filters for non-Latin targets {[bhuiyan2025_visitors]}; eight-library vote used to **discard** non-English {[xie2025_evaluating]} | Lingua for short/mixed text. Script heuristics when you already know the target language's writing system. An ensemble that exists only to throw away 48,178 non-English policies is a language filter, not multilingual support. |
+The undated comparison at [[https://modelpredict.com/language-identification-survey|modelpredict.com]] is still up (HTTP 200 on 2026-08-27) and is the source of the previous version of this page calling fastText the "clear winner". It is a blog with no date. Treat it as a historical speed/accuracy table, not as 2026 advice. Hosseini et al. is the peer-reviewed comparison this corpus actually cites, and it is a comparison on **privacy policies**, which are long.
+LibreTranslate's own detector, read from ''libretranslate/detect.py'' on 2026-08-27: **langdetect** for text of 20 or more characters, **lexilang** below that. The previous version of this page said LibreTranslate uses langdetect. That is still true for page-length strings.
+==== Do not trust the HTML language tag ====
+Full-text ''<html lang'' / ''document.documentElement.lang'' matches **1 of 1,120 crawled papers**. ''hreflang'' matches **1**. The attribute is almost unreported because almost nobody treated it as data.
+Two different mistakes get collapsed into "don't use ''lang''":
+  - **CMS default.** Many templates ship ''lang="en"'' on a site whose visible text is not English. That is an operator observation, not a corpus measurement.((Karel Kubicek, from crawls that compared the tag to detected text. Not a counted figure in this extraction.))
+  - **Declared language ≠ visible language, even when someone filled the attribute.** Bhuiyan et al. treat ''lang'' attributes as accessibility metadata that screen readers consume, and find they often fail to reflect the language of the visible content {[bhuiyan2025_visitors]}. That is a measurement of assistive-technology metadata, and it is the closest peer-reviewed statement we have. It is not a crawl of ''<html lang>'' versus a detector.
+The script below compares the BCP-47 primary tag on ''<html lang>'' with a Unicode-script majority over visible text (the LangCrUX idea, not their VPN crawl). A missing tag is reported, not a failure — absence is not a mismatch. An unknown primary tag is a failure: the checker does not silently agree with a language it cannot score. Hidden, ''aria-hidden'', and inline ''display:none'' / ''visibility:hidden'' subtrees are dropped. Japanese is scored as Han + hiragana + katakana together, so kana and kanji are not split against Latin. ''--demo'' agrees on a German page, fails on ''lang="en"'' with Han visible text, reports undeclared on a tagless English page, ignores a hidden Han paragraph on an English page, fails ''lang="ka"'' as unsupported, accepts Serbian Cyrillic for ''lang="sr"'', and treats a Japanese page whose kana+kanji outnumber Latin as a match.
+<file python check_page_language.py>
+#!/usr/bin/env python3
+"""Compare the HTML lang declaration with a Unicode-script heuristic.
+``<html lang>`` is a publisher claim. Bhuiyan et al. (IMC 2025) built
+LangCrUX by classifying *visible text* into Unicode blocks, not by reading
+the attribute. This checker makes that disagreement visible on one file.
+    python3 check_page_language.py page.html
+    python3 check_page_language.py --demo
+Exit 1 if the declared BCP-47 language's script family disagrees with the
+script family of visible text, or if the primary tag is not in the map
+(the checker does not silently agree with an unknown tag). A missing or
+empty ``lang`` is reported, not a failure: absence is not a mismatch.
+Visible text drops ``script``/``style``/``head``/``noscript``/``template``,
+plus subtrees with the ``hidden`` attribute, ``aria-hidden="true"``, or
+inline ``display:none`` / ``visibility:hidden``. Japanese is scored as
+Han + hiragana + katakana together. Stdlib only.
+``--demo`` runs the fixtures below.
+"""
+from __future__ import annotations
+import argparse
+import re
+import sys
+from html.parser import HTMLParser
+# Script families used on programming:multilingual_support, aligned with
+# Bhuiyan et al. IMC 2025 (Devanagari, Hangul, Cyrillic, Arabic, …).
+SCRIPTS: list[tuple[str, list[tuple[int, int]]]] = [
+    ("latin", [(0x0041, 0x024F), (0x1E00, 0x1EFF)]),
+    ("cyrillic", [(0x0400, 0x04FF)]),
+    ("greek", [(0x0370, 0x03FF)]),
+    ("arabic", [(0x0600, 0x06FF), (0x0750, 0x077F)]),
+    ("devanagari", [(0x0900, 0x097F)]),
+    ("bengali", [(0x0980, 0x09FF)]),
+    ("thai", [(0x0E00, 0x0E7F)]),
+    ("hangul", [(0xAC00, 0xD7AF), (0x1100, 0x11FF)]),
+    ("han", [(0x4E00, 0x9FFF)]),
+    ("hiragana", [(0x3040, 0x309F)]),
+    ("katakana", [(0x30A0, 0x30FF)]),
+    ("hebrew", [(0x0590, 0x05FF)]),
+]
+# Primary BCP-47 tag -> script families that count as a match when summed.
+# Dual-script languages list both; Japanese sums Han and kana so they are
+# not split against Latin. A tag missing from this map is unsupported.
+LANG_TO_SCRIPTS: dict[str, frozenset[str]] = {
+    "en": frozenset({"latin"}),
+    "de": frozenset({"latin"}),
+    "fr": frozenset({"latin"}),
+    "es": frozenset({"latin"}),
+    "it": frozenset({"latin"}),
+    "pt": frozenset({"latin"}),
+    "nl": frozenset({"latin"}),
+    "pl": frozenset({"latin"}),
+    "sv": frozenset({"latin"}),
+    "da": frozenset({"latin"}),
+    "fi": frozenset({"latin"}),
+    "tr": frozenset({"latin"}),
+    "vi": frozenset({"latin"}),
+    "id": frozenset({"latin"}),
+    "ro": frozenset({"latin"}),
+    "cs": frozenset({"latin"}),
+    "hr": frozenset({"latin"}),
+    "ru": frozenset({"cyrillic"}),
+    "uk": frozenset({"cyrillic"}),
+    "bg": frozenset({"cyrillic"}),
+    "sr": frozenset({"latin", "cyrillic"}),
+    "bs": frozenset({"latin", "cyrillic"}),
+    "el": frozenset({"greek"}),
+    "ar": frozenset({"arabic"}),
+    "fa": frozenset({"arabic"}),
+    "ur": frozenset({"arabic"}),
+    "hi": frozenset({"devanagari"}),
+    "bn": frozenset({"bengali"}),
+    "th": frozenset({"thai"}),
+    "ko": frozenset({"hangul"}),
+    "zh": frozenset({"han"}),
+    "ja": frozenset({"han", "hiragana", "katakana"}),
+    "he": frozenset({"hebrew"}),
+}
+HTML_LANG_RE = re.compile(r"<html\b[^>]*\blang\s*=\s*['\"]([^'\"]+)['\"]", re.I)
+HIDDEN_STYLE_RE = re.compile(r"(?:display\s*:\s*none|visibility\s*:\s*hidden)", re.I)
+SKIP_TAGS = frozenset({"script", "style", "head", "noscript", "template"})
+VOID_TAGS = frozenset(
+    {
+        "area",
+        "base",
+        "br",
+        "col",
+        "embed",
+        "hr",
+        "img",
+        "input",
+        "link",
+        "meta",
+        "param",
+        "source",
+        "track",
+        "wbr",
+    }
+)
+class VisibleTextParser(HTMLParser):
+    """Collect text nodes that a layout-less heuristic treats as visible."""
+    def __init__(self) -> None:
+        super().__init__(convert_charrefs=True)
+        self.parts: list[str] = []
+        self.hide_stack: list[bool] = []
+    def _this_tag_hides(self, tag: str, attrs: list[tuple[str, str | None]]) -> bool:
+        if tag in SKIP_TAGS:
+            return True
+        for k, v in attrs:
+            kl = k.lower()
+            vl = v if v is not None else ""
+            if kl == "hidden":
+                return True
+            if kl == "aria-hidden" and vl.strip().lower() == "true":
+                return True
+            if kl == "style" and HIDDEN_STYLE_RE.search(vl):
+                return True
+        return False
+    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
+        tag = tag.lower()
+        hidden = self._this_tag_hides(tag, attrs) or any(self.hide_stack)
+        if tag not in VOID_TAGS:
+            self.hide_stack.append(hidden)
+    def handle_endtag(self, tag: str) -> None:
+        tag = tag.lower()
+        if tag in VOID_TAGS:
+            return
+        if self.hide_stack:
+            self.hide_stack.pop()
+    def handle_data(self, data: str) -> None:
+        if self.hide_stack and self.hide_stack[-1]:
+            return
+        self.parts.append(data)
+def script_of(ch: str) -> str | None:
+    o = ord(ch)
+    for name, ranges in SCRIPTS:
+        for lo, hi in ranges:
+            if lo <= o <= hi:
+                return name
+    return None
+def declared_lang(html: str) -> str | None:
+    m = HTML_LANG_RE.search(html)
+    if not m:
+        return None
+    tag = m.group(1).strip()
+    if not tag:
+        return None
+    return tag.split("-")[0].lower()
+def visible_text(html: str) -> str:
+    parser = VisibleTextParser()
+    parser.feed(html)
+    parser.close()
+    return " ".join(parser.parts)
+def majority_script(text: str) -> tuple[str | None, dict[str, int]]:
+    counts: dict[str, int] = {name: 0 for name, _ranges in SCRIPTS}
+    for ch in text:
+        s = script_of(ch)
+        if s is None:
+            continue
+        counts[s] += 1
+    nonzero = {k: v for k, v in counts.items() if v > 0}
+    if not nonzero:
+        return None, nonzero
+    winner = max(nonzero.items(), key=lambda kv: (kv[1], kv[0]))[0]
+    return winner, nonzero
+def check(html: str) -> dict:
+    lang = declared_lang(html)
+    text = visible_text(html)
+    maj, counts = majority_script(text)
+    expected: frozenset[str] | None = None
+    unsupported = False
+    if lang is not None:
+        try:
+            expected = LANG_TO_SCRIPTS[lang]
+        except KeyError:
+            unsupported = True  # SafeMethodExc: unknown BCP-47 primary tag — HTML lang, not a schema key — 2026-08-27
+    mismatch = False
+    expected_n = 0
+    other_max = 0
+    if expected is not None and counts:
+        expected_n = 0
+        other_max = 0
+        for name, n in counts.items():
+            if name in expected:
+                expected_n += n
+            elif n > other_max:
+                other_max = n
+        mismatch = expected_n < other_max
+    return {
+        "declared": lang,
+        "majority_script": maj,
+        "expected_scripts": sorted(expected) if expected is not None else None,
+        "unsupported_lang": unsupported,
+        "mismatch": mismatch,
+        "expected_n": expected_n,
+        "other_max": other_max,
+        "counts": counts,
+    }
+DEMO = {
+    "agree": '<html lang="de"><p>Datenschutz und Einwilligung auf dieser Seite.</p></html>',
+    "mismatch": '<html lang="en"><p>这是中文正文，导航却声称 English。</p></html>',
+    "undeclared": "<html><p>Privacy policy of the example site.</p></html>",
+    "hidden": (
+        '<html lang="en"><p>Privacy policy of the example site.</p>'
+        "<p hidden>这是中文正文，导航却声称 English。</p></html>"
+    ),
+    "unsupported": '<html lang="ka"><p>ეს არის ქართული ტექსტი.</p></html>',
+    "japanese": '<html lang="ja"><p>あいうえ漢字カタカナAbcde</p></html>',
+    "serbian": '<html lang="sr"><p>Ово је српски текст на ћирилици.</p></html>',
+}
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("html_path", nargs="?")
+    ap.add_argument("--demo", action="store_true")
+    args = ap.parse_args()
+    if args.demo:
+        failed = False
+        r_agree = check(DEMO["agree"])
+        r_mismatch = check(DEMO["mismatch"])
+        r_undeclared = check(DEMO["undeclared"])
+        r_hidden = check(DEMO["hidden"])
+        r_unsupported = check(DEMO["unsupported"])
+        r_japanese = check(DEMO["japanese"])
+        r_serbian = check(DEMO["serbian"])
+        print("agree", r_agree)
+        print("mismatch", r_mismatch)
+        print("undeclared", r_undeclared)
+        print("hidden", r_hidden)
+        print("unsupported", r_unsupported)
+        print("japanese", r_japanese)
+        print("serbian", r_serbian)
+        han = r_mismatch["counts"]["han"]
+        latin = r_mismatch["counts"]["latin"]
+        print(f"mismatch fixture Han={han} Latin={latin}")
+        if r_agree["mismatch"] or r_agree["unsupported_lang"]:
+            print("FAILURE: German agree fixture mismatched or unsupported")
+            failed = True
+        if not r_mismatch["mismatch"]:
+            print("FAILURE: en+Han fixture did not mismatch")
+            failed = True
+        if han != 11 or latin != 7:
+            print("FAILURE: mismatch fixture is not 11 Han vs 7 Latin")
+            failed = True
+        if r_undeclared["declared"] is not None:
+            print("FAILURE: undeclared fixture has a lang")
+            failed = True
+        if r_hidden["mismatch"] or r_hidden["unsupported_lang"]:
+            print("FAILURE: hidden Han on an English page was treated as visible")
+            failed = True
+        if not r_unsupported["unsupported_lang"]:
+            print("FAILURE: lang=ka was not reported as unsupported")
+            failed = True
+        if r_unsupported["mismatch"]:
+            print("FAILURE: unsupported tag was also labelled a script mismatch")
+            failed = True
+        if r_japanese["mismatch"] or r_japanese["unsupported_lang"]:
+            print("FAILURE: Japanese kana+kanji lost to Latin because scripts were split")
+            failed = True
+        if r_serbian["mismatch"] or r_serbian["unsupported_lang"]:
+            print("FAILURE: Serbian Cyrillic was not accepted for lang=sr")
+            failed = True
+        return 1 if failed else 0
+    if not args.html_path:
+        ap.error("html_path or --demo")
+    html = open(args.html_path, encoding="utf-8").read()
+    r = check(html)
+    print(
+        f"declared={r['declared']!r} majority_script={r['majority_script']!r} "
+        f"expected_scripts={r['expected_scripts']!r} unsupported_lang={r['unsupported_lang']} "
+        f"mismatch={r['mismatch']}"
+    )
+    if r["unsupported_lang"]:
+        print("FAILURE: html lang primary tag is not in the script map")
+        return 1
+    if r["mismatch"]:
+        print("FAILURE: html lang script disagrees with visible text")
+        return 1
+    return 0
+if __name__ == "__main__":
+    sys.exit(main())
+</file>
+Ran here on 2026-08-27: ''--demo'' exit 0; the mismatch fixture is 11 Han vs 7 Latin characters. Extra fixtures: hidden Han ignored; ''lang="ka"'' unsupported; Japanese family 10 vs Latin 5; Serbian Cyrillic accepted.
+For long text after you know you are looking at prose, run langdetect or Lingua on the extracted visible text, not on the raw HTML. Hosseini et al. stripped URLs and required ten tokens before trusting a label {[hosseini2021unifying]}.
+===== Switching the site into a language =====
+Detection tells you what you fetched. It does not make the site serve the language you wanted. Four knobs, in the order this literature actually uses them:
+  - **Vantage point.** Bhuiyan et al. routed Puppeteer through country VPNs because sites "dynamically serve content-including language settings, layout, or accessibility features-based on the user's IP location." Without VPN-based localization, "web crawlers risk accessing global or English-dominant versions" {[bhuiyan2025_visitors]}. The instrument is on [[Design:Crawling location]].
+  - **Browser locale / ''Accept-Language''.** Full-text ''Accept-Language'' hits **8 of 1,120 crawled papers (0.7%)**; ''browser locale'' / ''navigator.language'' hits **13 (1.2%)**. Hils, Woods and Böhme set ''en-US'' as the preferred browser language on a preference-signal crawl {[hils2021_privacy]}. Yeung et al. changed the browser language to Ukrainian, Russian, and English to match vantage points in Ukraine, Russia and the US {[yeung2023_online]}. POSIX locales (''ru_RU.UTF-8'') in this corpus are a fingerprinting knob, not a language sample — one paper, role ''locale-knob'' above.
+  - **URL locale.** ''/de/'', ''?hl=ja'', ''www.google.fr''. Cheap when the site uses it; many CDNs ignore it once the cookie or IP has voted.
+  - **Click the language switcher.** An [[Programming:Interaction]] problem whose keywords you have to collect per language. See below.
+Pick the knob that matches how the site localises. A Tranco crawl from a US VM with default ''en-US'' is an English crawl of internationally branded domains, even if the company's users are not.
+===== Translation =====
+==== What the field uses, versus what you can run ====
+**56 of 135 tool-using papers (41.5%)** go through Google Translate or an unofficial wrapper. DeepL is 7. LibreTranslate / Argos is **3**, one of them this page's authors {[bouhoula2024_automated]}. Paid APIs are current practice and they are the reason a methods section that says "we translated with Google" is incomplete without a cost and a ToS sentence.
+Google Translate's consumer interface forbids automated processing; unofficial libraries (''googletrans'', ''pygtrans'') wrap that interface and get blocked. The Cloud Translation API is the licensed path and is the cost problem Bouhoula et al. named: "their costs scale poorly with crawling tens of thousands of websites" {[bouhoula2024_automated]}.
+==== LibreTranslate (Argos) — the self-hosted path ====
+[[https://github.com/LibreTranslate/LibreTranslate|LibreTranslate]] is an HTTP API in front of [[https://github.com/argosopentech/argos-translate/|Argos Translate]]. README fetched 2026-08-27: the engine is still Argos, not Google. Detection is still langdetect for long strings (see above).
+The previous compose on this page used ''libretranslate/libretranslate:latest''. That tag's Hub ''last_updated'' on 2026-08-27 was **2026-08-23**. A tag that moved four days ago is not a pin. [[Programming:Docker]] is the digest page; ''check_crawler_compose.py --all'' digest-pins every service, including this one. The compose below pins the **arm64** digest from that day (this host is aarch64). amd64 that day was ''sha256:8402eea7845b6890acdd4b1fb70320c130d2771935d36d697a901e8c149682ca''. Re-fetch before you copy either line.
+<file yaml libretranslate.compose.yaml>
 services:
   libretranslate:
-    image: libretranslate/libretranslate:latest  # Use libretranslate/libretranslate:latest-cuda for CUDA support
+    # Per-architecture digest of libretranslate/libretranslate:latest
+    # fetched 2026-08-27 from Hub. The tag last moved 2026-08-23.
+    # Re-fetch before you pin; this host is aarch64 so the arm64 line is
+    # the one that would run here. amd64:
+    #   sha256:8402eea7845b6890acdd4b1fb70320c130d2771935d36d697a901e8c149682ca
+    image: libretranslate/libretranslate:latest@sha256:110905ed2b68530de438c9126c951b0bb1c79669b8cd59d2f3bf2ecf679ea534
     restart: unless-stopped
     ports:
@@ Line 38: / Line 462: @@
       test: ['CMD-SHELL', './venv/bin/python scripts/healthcheck.py']
     environment:
-      LT_THREADS: 8  # Set this to the number of your CPUs
+      LT_THREADS: 8
-      LT_FRONTEND_TIMEOUT: 180  # 3 minutes timeout
+      LT_FRONTEND_TIMEOUT: 180
-      # More options here: https://github.com/LibreTranslate/LibreTranslate?tab=readme-ov-file#settings--flags
     volumes:
       - lt-local:/home/libretranslate/.local
-    # For CUDA support, uncomment the following lines:
+    # CUDA: image libretranslate/libretranslate:latest-cuda plus deploy.resources.
-    #deploy:
-    #  resources:
-    #    reservations:
-    #      devices:
-    #        - driver: nvidia
-    #          count: 1
-    #          capabilities: [gpu]
 volumes:
@@ Line 58: / Line 474: @@
 </file>
-Save this file to a new folder and run: ''docker compose up --detach''. The initial model download may take a while. You can check the status using ''docker ps''. Once the ''libretranslate'' service is listed as ''healthy'', it is operational.
+''docker compose up --detach'', wait until the service is ''healthy'' (first model download is slow), then use the documented API. LibreTranslate will detect-and-translate in one call; that detection is langdetect, not Lingua.
+This host has no Docker binary. The digest was fetched from the Hub API, not by pulling the image. We did not run a translation through this compose here.
+Two measured failure modes, from the paper that actually used this stack at crawl scale {[bouhoula2024_automated]}:
+  * **Short interactive text.** "LibreTranslate has trouble translating the short texts of interactive elements in some languages. For example, the Greek version of "Accept all cookies" translates to "Cookie policy" in English." They dropped Greek entirely.
+  * **Language coverage is the crawl's coverage.** spaCy + LibreTranslate supported their eleven EU languages; missing a language is a sampling decision, not a later filter.
+LibreTranslate is not free in CPU. In our crawls it used on the order of 20–50% of the machine.((Karel Kubicek, operator measurement on the cookie-notice crawl, not a corpus figure.)) Budget it as a sidecar, or you will think the crawler is slow.
+==== LLMs, and the case for not translating ====
+Local LLMs (Ollama and friends) translate short strings better than Argos and long documents worse: attention drops clauses, repeats, or leaves runs untranslated. They are much slower than LibreTranslate, depending on the model.((Operator observation, same source as the CPU footnote. Not a paper figure.))
+For **classification**, translating first is the 2024 answer and not the 2026 one. Nenadić et al. extract legally relevant labels from German, French, Italian and English privacy policies "in a single inference without requiring translation" and report F1 above 0.90 for most language/disclosure pairs on 120 expert-annotated policies {[nenadic2026_swiss]}. Tang et al. already noted that BERTopic "works with multilingual input and does not need translation" {[tang2024_automatic]}. If the downstream model speaks the language, skip the translator; you cannot debug a bad label that was produced from a bad translation of a short button.
-For translating text, use the documented API examples on [[https://github.com/LibreTranslate/LibreTranslate?tab=readme-ov-file#api-examples|LibreTranslate GitHub]] or one of the [[https://github.com/LibreTranslate/LibreTranslate?tab=readme-ov-file#language-bindings|API libraries]]. LibreTranslate also supports translation with automated language detection, currently using langdetect for this purpose.
+===== Keep the pipeline in the source language =====
-==== LLMs ====
+The previous version of this page listed five jobs and left interaction as a FIXME. The jobs are still the right list. The missing citations are the two papers above plus keyword practice from the cookie-notice crawl.
-Most large language models (LLMs) are multilingual and outperform traditional translation methods. To run LLMs locally, consider the [[https://github.com/ollama/ollama?tab=readme-ov-file|Ollama library]] and language API bindings. Prompts can be as simple as ''Translate from language <lang-A> to language <lang-B> the following text: <text>'' or ''Translate the following to <lang>: <text>''.
+  - **Detect** (above).
+  - **Switch** the site (above).
+  - **Interact** with language-specific keywords.
+  - **Classify** in-language, or translate then classify, and say which.
+  - **Encode** the bytes you store.
-Expect throughput to be 50-1000 times slower than LibreTranslate, depending on the model. For shorter texts, LLMs often provide much higher accuracy. However, for longer texts, attention mechanisms may fail, resulting in issues like untranslated text, repeated words, shortened translations, or loss of meaning.
+==== Interaction ====
-===== Multilingual Pipeline =====
+Most crawlers find a cookie notice, a login, or a language switcher by matching keywords on links and buttons. Those keywords are not one Google-translated list.
-To make your crawl multilingual, address the following tasks:
+Bouhoula et al. detect the notice with language-specific spaCy models and language-specific variants of "cookie", then translate interactive-element text into English and classify accept / reject / close / save with one English model {[bouhoula2024_automated]}. When LibreTranslate mangled short strings, they hand-corrected the German and Danish translations and dropped Greek entirely. That is the method: **do not ship an uncorrected English keyword list through a translator**. Translating an English accept-button list through LibreTranslate is how you get "Cookie policy" as the accept button.
-  - Language detection (see above).
+A looser alternative is to crawl first in English-only and claim a limitation. Xie et al. ran Hosseini's eight-library vote and then **excluded** domains whose homepage or policies were non-English — **48,178 of 101,249** domains in their funnel {[xie2025_evaluating]}. That is a valid English-only study. It is not multilingual support. Write it as a sampling cut.
-  - Switching websites to supported languages by changing browser locale, modifying URL locale strings, or clicking links/buttons labeled with the target language name.
-  - Interacting with websites based on detection of specific keywords in links or forms.
-  - Classifying multilingual content.
-  - Processing multilingual data.
-=== Interaction ===
+==== Classification ====
-FIXME: This section is subjective. Can we find references?
+  * **2026, current for documents:** multilingual LLM, no translation {[nenadic2026_swiss]}. Fine-tune or prompt in the source language. Validate per language; they annotated 30 policies in each of four languages, and F1 above 0.90 is for most language/disclosure pairs, not a licence to skip that gold set.
+  * **2024, current for short UI text if you must share one English classifier:** translate, then BERT {[bouhoula2024_automated]}. Budget the translator, and do not trust it on short buttons.
+  * **Historical:** mBERT / XLM-R fine-tuned on multilingual training data. Still faster than a local LLM at crawl volume. Yahoo's production page classifier (outside this corpus; see [[Design:Website classification]]) takes full content **or** the URL alone.
-Most crawlers interact with websites by detecting specific keywords on pages (e.g., in links or form fields). To construct keyword lists, navigate target-language websites and collect keywords with the help of translation tools or native speakers (e.g., via Amazon MTurk). Alternatively, translate keywords—while simple translation tools may result in unsuitable synonyms, LLMs with task-specific context can perform reasonably well.
+Generating multilingual training data with an LLM from an English gold set is a data-augmentation trick, not a substitute for per-language validation.
-=== Classification ===
+==== Bytes ====
-Many NLP models now support multiple languages (e.g., multilingual BERT). These models perform better when fine-tuned on multilingual training data. You can collect such data from a sample crawl and use translation for data labeling. Alternatively, use LLMs to generate multilingual training data from a single-language annotated dataset and train a multilingual model. This approach balances performance with computational costs, as BERT is faster than traditional LLMs.
+UTF-8 end to end. NFC if you will compare strings. A Latin-1 round-trip on a UTF-8 page is a silent corruption of every non-ASCII token, which on this page is the object of study. The extraction behind this wiki reads ''paper.cols.txt'' as latin1 **on purpose** (byte-preserving); do not copy that for page content you will language-detect.
-=== Postprocessing ===
+===== What to report =====
-Watch for potential pitfalls, such as:
+A methods paragraph that names "the multilingual web" without these is not reconstructable:
-  * Proper encoding support for non-Latin alphabets.
+  * **Which languages, with ISO 639-1 / BCP-47 tags**, and whether English was a target or a fallback.
+  * **How the site was switched** into that language: vantage (with provider), ''Accept-Language'' / browser locale, URL, click. "We crawled from the EU" is a location, not a language.
+  * **How language was decided on each page:** detector (and version), ensemble, Unicode-script rule, or a human. The threshold (LangCrUX: 50% visible text; Hosseini: ≥10 tokens).
+  * **Whether you translated**, with the engine, the pin (digest, not '':latest''), and which languages you **dropped** because the engine failed.
+  * **Per-language validation** on the labels you publish, not a micro-F1 pooled across German and Greek.
-====== References ======
+===== Related pages =====
-/*
+  * [[Design:Website selection]] — lists. LangCrUX is a language-filtered CrUX, not a Tranco of "the web".
-To insert citations, follow these steps:
+  * [[Design:Crawling location]] — the IP that decides the language you are served.
+  * [[Programming:Interaction]] — the click; this page is the keyword list behind it.
+  * [[Programming:Docker]] — digest and ''/dev/shm''. This compose is a translator sidecar.
+  * [[Privacy:Consent]] — cookie notices are the worked example of multilingual UI text {[bouhoula2024_automated]}.
+  * [[Design:Website classification]] — topic, not language ID.
+  * [[Literature:Corpus]] — venue window and provisional 2025–2026.
-  - Verify the BibTeX entry exists in https://measuretheweb.org/literature/bibliography. If not, add it there.
+===== Methodology and limitations of these figures =====
-  - Use {[CitationKey]} where needed in the text; it will render as a numbered reference.
-  - Keep this section unchanged to display the bibliography.
-If any step fails, a purple warning will appear on the page.
+Paper counts from the 5,859-paper extraction (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P, 2010–2026). Denominator for language handling is **1,080 crawled papers with a ''crawlConfig''**, not 5,859 and not 1,120. Tool counts are of papers that **used** a ''language-detection'' or ''translation'' tool. Sentinels are not answers. Language names and tool names were folded; residue is in the report. 2025–2026 venue-years are incomplete by construction. Queries, the fold, Hub digests and the archived-repo checks: [[provenance:programming:multilingual_support]]. Hosseini et al. {[hosseini2021unifying]} is in the bibliographic index and **not** in ''extractions.jsonl''; figures from it are from the publisher PDF.
-*/
 <bibtex bibliography></bibtex>
-/* This enables discussion under this article. */
 ~~DISCUSSION~~