Table of Contents
Multilingual Support
You are about to crawl sites that are not in English, or classify text that is not. This page is the instrument: detect the language you actually fetched, switch the site into the language you meant, translate only when you have to, and keep the rest of the pipeline in the source language when you can.
The field mostly does not. Of the 1,080 crawled papers with a crawlConfig in this corpus (seven venues, 2010–2026, 5,859 extracted papers), 136 (12.6%) name any language at all. 77 of those 136 name only English — against the schema's own rule that English-only stays empty. 53 papers (4.9% of 1,080) handled a non-English natural language. Sampling from a local list is a different page (Website selection, CrUX); this one starts after you have URLs.
The English-centric gap is also a publishing fact. Mhaidli et al. recorded it as a tool and community problem, not a crawling one [1Mhaidli, Abraham; Fidan, Selin; Doan, An; Herakovic, Gina; Srinath, Mukund; Matheson, Lee; Wilson, Shomir; Schaub, Florian (2023): "Researchers’ Experiences in Analyzing Privacy Policies: Challenges and Opportunities", Proceedings on Privacy Enhancing Technologies 2023:287-305. (DOI) (Link)]:
most work that is being done is only focused on privacy policies that are in English. Yeah, I see that as a big problem personally
If you read one measurement paper first, make it Bhuiyan et al. [2Bhuiyan, Masudul Hasan Masud; Varvello, Matteo; Zaki, Yasir; Staicu, Cristian-Alexandru (2025): "Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] (IMC 2025): 120,000 sites across 12 non-Latin-script languages, crawled through country VPNs, language decided on visible text, not on <html lang>.
<html lang> is a publisher declaration, not a measurement. Trust visible text (a Unicode-script heuristic, then a detector). Mixed-language pages are the common case, not the edge: Bhuiyan et al. kept a site if 50% of visible text was in the target language, and the median site was 88% native [2Bhuiyan, Masudul Hasan Masud; Varvello, Matteo; Zaki, Yasir; Staicu, Cristian-Alexandru (2025): "Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]. Translating everything into English is the expensive path the corpus actually takes (Google Translate is 49 of 135 papers that used a language/translation tool). The 2026 method that is current for classification is stay in the source language [3Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)] — validated per language, because their F1 above 0.90 holds for most language/disclosure pairs, not all.
What this page is for
- Here: detect language, switch locale, translate or don't, keep keywords and classifiers in-language. The LibreTranslate compose is the self-hosted translation sidecar.
- Website selection / CrUX: a popularity list is not a language list. LangCrUX is CrUX plus a language filter [2Bhuiyan, Masudul Hasan Masud; Varvello, Matteo; Zaki, Yasir; Staicu, Cristian-Alexandru (2025): "Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective", in: Proceedings of the ACM Internet Measurement Conference. (DOI)].
- Crawling location: many sites pick a language from the client IP. A US cloud VM fetches the English CDN site. VPN localisation belongs there; this page only names it as a language switch.
- Interaction: clicking a language switcher is a click. Building the keyword list for that click is here.
- Docker: pin the image digest and size
/dev/shm. The compose below is a translator, not a crawler;check_crawler_compose.py –allstill digest-pins it. - Website classification: topic taxonomies, including Yahoo's multilingual production classifier. Not language ID.
What to read first
| Paper | Why |
|---|---|
| Bhuiyan et al., IMC 2025, Not All Visitors are Bilingual [2Bhuiyan, Masudul Hasan Masud; Varvello, Matteo; Zaki, Yasir; Staicu, Cristian-Alexandru (2025): "Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] | How to sample a multilingual web: CrUX × Unicode-script filter × country VPN. Mixed pages, lang attributes as accessibility metadata. |
| Bouhoula, Kubicek et al., USENIX Sec 2024 [4Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)] | The self-hosted pipeline this page used to recommend without a paper: langdetect + LibreTranslate on 11 EU languages, 97k sites. Greek was dropped because the translator failed on short button text. |
| Nenadić, Rodriguez and Calandrino, PoPETs 2026 [3Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)] | Classification without translation: one LLM inference over German, French, Italian and English privacy policies. F1 above 0.90 for most language/disclosure pairs on 120 expert-annotated policies (30 per language), not all pairs. |
| Tang et al., PoPETs 2024 [5Tang, Jenny; Alvarez, Léo; Brar, Arjun; Hoang, Nguyen Phong; Christin, Nicolas (2024): "Automatic Generation of Web Censorship Probe Lists", Proceedings on Privacy Enhancing Technologies 2024(4):44-60. (DOI)] | Detection at 103-language scale: Lingua, CLD3 as fallback on the 15.22% Lingua does not support, 99.29% identified. |
| Hosseini et al., PoPETs 2021 [6Hosseini, Henry; Degeling, Martin; Utz, Christine; Hupperich, Thomas (2021): "Unifying privacy policy detection", Proceedings on Privacy Enhancing Technologies 2021:480–499. (DOI)] | Not in this extraction (in the bibliographic index; PDF fetched). The eight-library ensemble later papers copy. On long privacy-policy text, langdetect's Cramér's V with the ensemble-determined language was 0.97, the highest of the eight. |
| Degeling et al., NDSS 2019 [7Degeling, Martin; Utz, Christine; Lentzsch, Christopher; Hosseini, Henry; Schaub, Florian; Holz, Thorsten (2019): "We Value Your Privacy ... Now Take Some Cookies: Measuring the GDPR's Impact on Web Privacy", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] | 24 official EU languages on a consent crawl, before anyone in this corpus self-hosted a translator. |
What the corpus actually reports
Two signals, kept apart. crawlConfig.languages is “languages or locales explicitly handled; empty if only English or unstated.” A used language-detection or translation tool is a different question, and most of those papers did not crawl.
Of 1,080 crawled papers with a crawlConfig:
Role of languages[] | Papers | Share of 1,080 |
|---|---|---|
| multilingual (two or more natural languages, or a count-phrase such as “103 languages”) | 22 | 2.0% |
| language-specific (exactly one non-English natural language) | 31 | 2.9% |
| handled non-English (union of the two) | 53 | 4.9% |
| english-only (schema said leave empty) | 77 | 7.1% |
locale-knob (POSIX ru_RU.UTF-8 and friends — a fingerprinting setting, not a language study) | 1 | 0.1% |
| vague (“foreign-language websites”, “multilingual webpages”, …) | 4 | 0.4% |
programming (languages: [“C”] — GitHub awesome-c, not a natural language) | 1 | 0.1% |
| empty | 944 | 87.4% |
Among the 136 non-empty rows, English still appears in 110 (80.9%) because multilingual papers list it alongside the others. After English, the folded families are Chinese 25, German 17, French 12, Russian 11, Spanish 9. Residue of the language-name fold: 0.
135 of 5,859 papers (2.3%) used a language-detection or translation tool; 67 of those are crawled (6.0% of 1,120). Folded used-tool families, of the 135 (a paper may hit several):
| Family | Papers of 135 | Kind | Current in 2026? |
|---|---|---|---|
| Google Translate (API / Cloud / GNMT) | 49 | translation | What the field uses. Paid; ToS and cost are why the next rows exist. |
| langdetect (incl. Shuyo's Java library) | 25 | detection | Live (PyPI port pushed 2025-03). Default in LibreTranslate for text ≥20 characters. |
| googletrans / pygtrans | 8 | translation | Unofficial Google wrappers. ToS of the web UI forbid automated use. |
| langid | 8 | detection | Live; older than langdetect's neural neighbours. |
| DeepL | 7 | translation | Paid; same cost problem at crawl scale. |
| fastText | 5 | detection | Archived 2024-03-22 (facebookresearch/fastText). |
| CLD2 | 5 | detection | Historical; CLD3 replaced it in Chrome and is itself archived. |
| NLLB / Opus-MT / EasyNMT | 5 | translation | Open neural MT. Appears late in the window. |
| Lingua | 4 | detection | Live (pemistahl/lingua-py pushed 2026-07-20). Short and mixed text. |
| LibreTranslate / Argos | 3 | translation | Self-hosted. One of the three is [4Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)]. |
| CLD3 | 3 | detection | Archived 2023-05-24 (google/cld3). |
Five tool names did not fold (“Detect Language”, “translation software”, “Language Identification”, “Translate”, “language classifier [21]”). They stay residue in the report, not in this table.
Exact-string “Google Translate” undercounts: the family is 49, plus 8 unofficial wrappers, 56 / 135 (41.5%) once both are unioned. A page that ranked detectors by what a 2019 blog called the accuracy winner, without this table, would tell you to start with fastText. The corpus starts with langdetect; the live short-text library in 2026 is Lingua; fastText and CLD3 are archived.
Year shape of the 53, of papers with a crawlConfig. 2025–2026 is provisional (Corpus):
| Bucket | With crawlConfig | Named any language | Handled non-English |
|---|---|---|---|
| 2010–2013 | 98 | 2 | 2 |
| 2014–2017 | 160 | 14 | 6 |
| 2018–2021 | 297 | 39 | 14 |
| 2022–2024 | 336 | 55 | 23 |
| 2025–2026* | 189 | 26 | 8 |
These are reporting rates. A paper can run langdetect and never write the word in the PDF.
Language detection
Dated, not ranked as current practice
| When | What people used | What to do with it now |
|---|---|---|
| 2010–2018 | Shuyo's Language Detection Library / langdetect, langid, CLD2 | Still what this corpus uses. Fine for long page text in the languages those libraries cover. |
| 2018–2023 | CLD3, fastText LID, occasional ensembles | CLD3 and fastText are archived. Do not start a 2026 crawl on an archived Facebook snapshot because a blog said it won a benchmark. |
| 2021 | Hosseini et al.'s eight-library majority vote on privacy policies [6Hosseini, Henry; Degeling, Martin; Utz, Christine; Hupperich, Thomas (2021): "Unifying privacy policy detection", Proceedings on Privacy Enhancing Technologies 2021:480–499. (DOI)] | Copy the ensemble idea for a gold set. Do not run eight detectors in the inner loop of a crawl. Langdetect's Cramér's V with the determined language was 0.97 on that long-document corpus. |
| 2024–2026 | Lingua, then CLD3 as fallback [5Tang, Jenny; Alvarez, Léo; Brar, Arjun; Hoang, Nguyen Phong; Christin, Nicolas (2024): "Automatic Generation of Web Censorship Probe Lists", Proceedings on Privacy Enhancing Technologies 2024(4):44-60. (DOI)]; Unicode-script filters for non-Latin targets [2Bhuiyan, Masudul Hasan Masud; Varvello, Matteo; Zaki, Yasir; Staicu, Cristian-Alexandru (2025): "Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]; eight-library vote used to discard non-English [8Xie, Qinge; Ramakrishnan, Karthik; Li, Frank (2025): "Evaluating Privacy Policies under Modern Privacy Laws At Scale: An LLM-Based Automated Approach", in: Proceedings of the USENIX Security Symposium, pp. 5797-5816. (Link)] | Lingua for short/mixed text. Script heuristics when you already know the target language's writing system. An ensemble that exists only to throw away 48,178 non-English policies is a language filter, not multilingual support. |
The undated comparison at modelpredict.com is still up (HTTP 200 on 2026-08-27) and is the source of the previous version of this page calling fastText the “clear winner”. It is a blog with no date. Treat it as a historical speed/accuracy table, not as 2026 advice. Hosseini et al. is the peer-reviewed comparison this corpus actually cites, and it is a comparison on privacy policies, which are long.
LibreTranslate's own detector, read from libretranslate/detect.py on 2026-08-27: langdetect for text of 20 or more characters, lexilang below that. The previous version of this page said LibreTranslate uses langdetect. That is still true for page-length strings.
Do not trust the HTML language tag
Full-text <html lang / document.documentElement.lang matches 1 of 1,120 crawled papers. hreflang matches 1. The attribute is almost unreported because almost nobody treated it as data.
Two different mistakes get collapsed into “don't use lang”:
- CMS default. Many templates ship
lang=“en”on a site whose visible text is not English. That is an operator observation, not a corpus measurement.1) - Declared language ≠ visible language, even when someone filled the attribute. Bhuiyan et al. treat
langattributes as accessibility metadata that screen readers consume, and find they often fail to reflect the language of the visible content [2Bhuiyan, Masudul Hasan Masud; Varvello, Matteo; Zaki, Yasir; Staicu, Cristian-Alexandru (2025): "Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]. That is a measurement of assistive-technology metadata, and it is the closest peer-reviewed statement we have. It is not a crawl of<html lang>versus a detector.
The script below compares the BCP-47 primary tag on <html lang> with a Unicode-script majority over visible text (the LangCrUX idea, not their VPN crawl). A missing tag is reported, not a failure — absence is not a mismatch. An unknown primary tag is a failure: the checker does not silently agree with a language it cannot score. Hidden, aria-hidden, and inline display:none / visibility:hidden subtrees are dropped. Japanese is scored as Han + hiragana + katakana together, so kana and kanji are not split against Latin. –demo agrees on a German page, fails on lang=“en” with Han visible text, reports undeclared on a tagless English page, ignores a hidden Han paragraph on an English page, fails lang=“ka” as unsupported, accepts Serbian Cyrillic for lang=“sr”, and treats a Japanese page whose kana+kanji outnumber Latin as a match.
- check_page_language.py
#!/usr/bin/env python3 """Compare the HTML lang declaration with a Unicode-script heuristic. ``<html lang>`` is a publisher claim. Bhuiyan et al. (IMC 2025) built LangCrUX by classifying *visible text* into Unicode blocks, not by reading the attribute. This checker makes that disagreement visible on one file. python3 check_page_language.py page.html python3 check_page_language.py --demo Exit 1 if the declared BCP-47 language's script family disagrees with the script family of visible text, or if the primary tag is not in the map (the checker does not silently agree with an unknown tag). A missing or empty ``lang`` is reported, not a failure: absence is not a mismatch. Visible text drops ``script``/``style``/``head``/``noscript``/``template``, plus subtrees with the ``hidden`` attribute, ``aria-hidden="true"``, or inline ``display:none`` / ``visibility:hidden``. Japanese is scored as Han + hiragana + katakana together. Stdlib only. ``--demo`` runs the fixtures below. """ from __future__ import annotations import argparse import re import sys from html.parser import HTMLParser # Script families used on programming:multilingual_support, aligned with # Bhuiyan et al. IMC 2025 (Devanagari, Hangul, Cyrillic, Arabic, …). SCRIPTS: list[tuple[str, list[tuple[int, int]]]] = [ ("latin", [(0x0041, 0x024F), (0x1E00, 0x1EFF)]), ("cyrillic", [(0x0400, 0x04FF)]), ("greek", [(0x0370, 0x03FF)]), ("arabic", [(0x0600, 0x06FF), (0x0750, 0x077F)]), ("devanagari", [(0x0900, 0x097F)]), ("bengali", [(0x0980, 0x09FF)]), ("thai", [(0x0E00, 0x0E7F)]), ("hangul", [(0xAC00, 0xD7AF), (0x1100, 0x11FF)]), ("han", [(0x4E00, 0x9FFF)]), ("hiragana", [(0x3040, 0x309F)]), ("katakana", [(0x30A0, 0x30FF)]), ("hebrew", [(0x0590, 0x05FF)]), ] # Primary BCP-47 tag -> script families that count as a match when summed. # Dual-script languages list both; Japanese sums Han and kana so they are # not split against Latin. A tag missing from this map is unsupported. LANG_TO_SCRIPTS: dict[str, frozenset[str]] = { "en": frozenset({"latin"}), "de": frozenset({"latin"}), "fr": frozenset({"latin"}), "es": frozenset({"latin"}), "it": frozenset({"latin"}), "pt": frozenset({"latin"}), "nl": frozenset({"latin"}), "pl": frozenset({"latin"}), "sv": frozenset({"latin"}), "da": frozenset({"latin"}), "fi": frozenset({"latin"}), "tr": frozenset({"latin"}), "vi": frozenset({"latin"}), "id": frozenset({"latin"}), "ro": frozenset({"latin"}), "cs": frozenset({"latin"}), "hr": frozenset({"latin"}), "ru": frozenset({"cyrillic"}), "uk": frozenset({"cyrillic"}), "bg": frozenset({"cyrillic"}), "sr": frozenset({"latin", "cyrillic"}), "bs": frozenset({"latin", "cyrillic"}), "el": frozenset({"greek"}), "ar": frozenset({"arabic"}), "fa": frozenset({"arabic"}), "ur": frozenset({"arabic"}), "hi": frozenset({"devanagari"}), "bn": frozenset({"bengali"}), "th": frozenset({"thai"}), "ko": frozenset({"hangul"}), "zh": frozenset({"han"}), "ja": frozenset({"han", "hiragana", "katakana"}), "he": frozenset({"hebrew"}), } HTML_LANG_RE = re.compile(r"<html\b[^>]*\blang\s*=\s*['\"]([^'\"]+)['\"]", re.I) HIDDEN_STYLE_RE = re.compile(r"(?:display\s*:\s*none|visibility\s*:\s*hidden)", re.I) SKIP_TAGS = frozenset({"script", "style", "head", "noscript", "template"}) VOID_TAGS = frozenset( { "area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source", "track", "wbr", } ) class VisibleTextParser(HTMLParser): """Collect text nodes that a layout-less heuristic treats as visible.""" def __init__(self) -> None: super().__init__(convert_charrefs=True) self.parts: list[str] = [] self.hide_stack: list[bool] = [] def _this_tag_hides(self, tag: str, attrs: list[tuple[str, str | None]]) -> bool: if tag in SKIP_TAGS: return True for k, v in attrs: kl = k.lower() vl = v if v is not None else "" if kl == "hidden": return True if kl == "aria-hidden" and vl.strip().lower() == "true": return True if kl == "style" and HIDDEN_STYLE_RE.search(vl): return True return False def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: tag = tag.lower() hidden = self._this_tag_hides(tag, attrs) or any(self.hide_stack) if tag not in VOID_TAGS: self.hide_stack.append(hidden) def handle_endtag(self, tag: str) -> None: tag = tag.lower() if tag in VOID_TAGS: return if self.hide_stack: self.hide_stack.pop() def handle_data(self, data: str) -> None: if self.hide_stack and self.hide_stack[-1]: return self.parts.append(data) def script_of(ch: str) -> str | None: o = ord(ch) for name, ranges in SCRIPTS: for lo, hi in ranges: if lo <= o <= hi: return name return None def declared_lang(html: str) -> str | None: m = HTML_LANG_RE.search(html) if not m: return None tag = m.group(1).strip() if not tag: return None return tag.split("-")[0].lower() def visible_text(html: str) -> str: parser = VisibleTextParser() parser.feed(html) parser.close() return " ".join(parser.parts) def majority_script(text: str) -> tuple[str | None, dict[str, int]]: counts: dict[str, int] = {name: 0 for name, _ranges in SCRIPTS} for ch in text: s = script_of(ch) if s is None: continue counts[s] += 1 nonzero = {k: v for k, v in counts.items() if v > 0} if not nonzero: return None, nonzero winner = max(nonzero.items(), key=lambda kv: (kv[1], kv[0]))[0] return winner, nonzero def check(html: str) -> dict: lang = declared_lang(html) text = visible_text(html) maj, counts = majority_script(text) expected: frozenset[str] | None = None unsupported = False if lang is not None: try: expected = LANG_TO_SCRIPTS[lang] except KeyError: unsupported = True # SafeMethodExc: unknown BCP-47 primary tag — HTML lang, not a schema key — 2026-08-27 mismatch = False expected_n = 0 other_max = 0 if expected is not None and counts: expected_n = 0 other_max = 0 for name, n in counts.items(): if name in expected: expected_n += n elif n > other_max: other_max = n mismatch = expected_n < other_max return { "declared": lang, "majority_script": maj, "expected_scripts": sorted(expected) if expected is not None else None, "unsupported_lang": unsupported, "mismatch": mismatch, "expected_n": expected_n, "other_max": other_max, "counts": counts, } DEMO = { "agree": '<html lang="de"><p>Datenschutz und Einwilligung auf dieser Seite.</p></html>', "mismatch": '<html lang="en"><p>这是中文正文，导航却声称 English。</p></html>', "undeclared": "<html><p>Privacy policy of the example site.</p></html>", "hidden": ( '<html lang="en"><p>Privacy policy of the example site.</p>' "<p hidden>这是中文正文，导航却声称 English。</p></html>" ), "unsupported": '<html lang="ka"><p>ეს არის ქართული ტექსტი.</p></html>', "japanese": '<html lang="ja"><p>あいうえ漢字カタカナAbcde</p></html>', "serbian": '<html lang="sr"><p>Ово је српски текст на ћирилици.</p></html>', } def main() -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("html_path", nargs="?") ap.add_argument("--demo", action="store_true") args = ap.parse_args() if args.demo: failed = False r_agree = check(DEMO["agree"]) r_mismatch = check(DEMO["mismatch"]) r_undeclared = check(DEMO["undeclared"]) r_hidden = check(DEMO["hidden"]) r_unsupported = check(DEMO["unsupported"]) r_japanese = check(DEMO["japanese"]) r_serbian = check(DEMO["serbian"]) print("agree", r_agree) print("mismatch", r_mismatch) print("undeclared", r_undeclared) print("hidden", r_hidden) print("unsupported", r_unsupported) print("japanese", r_japanese) print("serbian", r_serbian) han = r_mismatch["counts"]["han"] latin = r_mismatch["counts"]["latin"] print(f"mismatch fixture Han={han} Latin={latin}") if r_agree["mismatch"] or r_agree["unsupported_lang"]: print("FAILURE: German agree fixture mismatched or unsupported") failed = True if not r_mismatch["mismatch"]: print("FAILURE: en+Han fixture did not mismatch") failed = True if han != 11 or latin != 7: print("FAILURE: mismatch fixture is not 11 Han vs 7 Latin") failed = True if r_undeclared["declared"] is not None: print("FAILURE: undeclared fixture has a lang") failed = True if r_hidden["mismatch"] or r_hidden["unsupported_lang"]: print("FAILURE: hidden Han on an English page was treated as visible") failed = True if not r_unsupported["unsupported_lang"]: print("FAILURE: lang=ka was not reported as unsupported") failed = True if r_unsupported["mismatch"]: print("FAILURE: unsupported tag was also labelled a script mismatch") failed = True if r_japanese["mismatch"] or r_japanese["unsupported_lang"]: print("FAILURE: Japanese kana+kanji lost to Latin because scripts were split") failed = True if r_serbian["mismatch"] or r_serbian["unsupported_lang"]: print("FAILURE: Serbian Cyrillic was not accepted for lang=sr") failed = True return 1 if failed else 0 if not args.html_path: ap.error("html_path or --demo") html = open(args.html_path, encoding="utf-8").read() r = check(html) print( f"declared={r['declared']!r} majority_script={r['majority_script']!r} " f"expected_scripts={r['expected_scripts']!r} unsupported_lang={r['unsupported_lang']} " f"mismatch={r['mismatch']}" ) if r["unsupported_lang"]: print("FAILURE: html lang primary tag is not in the script map") return 1 if r["mismatch"]: print("FAILURE: html lang script disagrees with visible text") return 1 return 0 if __name__ == "__main__": sys.exit(main())
Ran here on 2026-08-27: –demo exit 0; the mismatch fixture is 11 Han vs 7 Latin characters. Extra fixtures: hidden Han ignored; lang=“ka” unsupported; Japanese family 10 vs Latin 5; Serbian Cyrillic accepted.
For long text after you know you are looking at prose, run langdetect or Lingua on the extracted visible text, not on the raw HTML. Hosseini et al. stripped URLs and required ten tokens before trusting a label [6Hosseini, Henry; Degeling, Martin; Utz, Christine; Hupperich, Thomas (2021): "Unifying privacy policy detection", Proceedings on Privacy Enhancing Technologies 2021:480–499. (DOI)].
Switching the site into a language
Detection tells you what you fetched. It does not make the site serve the language you wanted. Four knobs, in the order this literature actually uses them:
- Vantage point. Bhuiyan et al. routed Puppeteer through country VPNs because sites “dynamically serve content-including language settings, layout, or accessibility features-based on the user's IP location.” Without VPN-based localization, “web crawlers risk accessing global or English-dominant versions” [2Bhuiyan, Masudul Hasan Masud; Varvello, Matteo; Zaki, Yasir; Staicu, Cristian-Alexandru (2025): "Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]. The instrument is on Crawling location.
- Browser locale /
Accept-Language. Full-textAccept-Languagehits 8 of 1,120 crawled papers (0.7%);browser locale/navigator.languagehits 13 (1.2%). Hils, Woods and Böhme seten-USas the preferred browser language on a preference-signal crawl [9Hils, Maximilian; Woods, Daniel W.; Böhme, Rainer (2021): "Privacy Preference Signals: Past, Present and Future", in: Proceedings on Privacy Enhancing Technologies. (DOI)]. Yeung et al. changed the browser language to Ukrainian, Russian, and English to match vantage points in Ukraine, Russia and the US [10Yeung, Christina; Iqbal, Umar; O'Neil, Yekaterina Tsipenyuk; Kohno, Tadayoshi; Roesner, Franziska (2023): "Online Advertising in Ukraine and Russia During the 2022 Russian Invasion", in: Proceedings of the ACM Web Conference. (DOI)]. POSIX locales (ru_RU.UTF-8) in this corpus are a fingerprinting knob, not a language sample — one paper, rolelocale-knobabove. - URL locale.
/de/,?hl=ja,www.google.fr. Cheap when the site uses it; many CDNs ignore it once the cookie or IP has voted. - Click the language switcher. An Interaction problem whose keywords you have to collect per language. See below.
Pick the knob that matches how the site localises. A Tranco crawl from a US VM with default en-US is an English crawl of internationally branded domains, even if the company's users are not.
Translation
What the field uses, versus what you can run
56 of 135 tool-using papers (41.5%) go through Google Translate or an unofficial wrapper. DeepL is 7. LibreTranslate / Argos is 3, one of them this page's authors [4Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)]. Paid APIs are current practice and they are the reason a methods section that says “we translated with Google” is incomplete without a cost and a ToS sentence.
Google Translate's consumer interface forbids automated processing; unofficial libraries (googletrans, pygtrans) wrap that interface and get blocked. The Cloud Translation API is the licensed path and is the cost problem Bouhoula et al. named: “their costs scale poorly with crawling tens of thousands of websites” [4Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)].
LibreTranslate (Argos) — the self-hosted path
LibreTranslate is an HTTP API in front of Argos Translate. README fetched 2026-08-27: the engine is still Argos, not Google. Detection is still langdetect for long strings (see above).
The previous compose on this page used libretranslate/libretranslate:latest. That tag's Hub last_updated on 2026-08-27 was 2026-08-23. A tag that moved four days ago is not a pin. Docker is the digest page; check_crawler_compose.py –all digest-pins every service, including this one. The compose below pins the arm64 digest from that day (this host is aarch64). amd64 that day was sha256:8402eea7845b6890acdd4b1fb70320c130d2771935d36d697a901e8c149682ca. Re-fetch before you copy either line.
- libretranslate.compose.yaml
services: libretranslate: # Per-architecture digest of libretranslate/libretranslate:latest # fetched 2026-08-27 from Hub. The tag last moved 2026-08-23. # Re-fetch before you pin; this host is aarch64 so the arm64 line is # the one that would run here. amd64: # sha256:8402eea7845b6890acdd4b1fb70320c130d2771935d36d697a901e8c149682ca image: libretranslate/libretranslate:latest@sha256:110905ed2b68530de438c9126c951b0bb1c79669b8cd59d2f3bf2ecf679ea534 restart: unless-stopped ports: - 5000:5000 healthcheck: test: ['CMD-SHELL', './venv/bin/python scripts/healthcheck.py'] environment: LT_THREADS: 8 LT_FRONTEND_TIMEOUT: 180 volumes: - lt-local:/home/libretranslate/.local # CUDA: image libretranslate/libretranslate:latest-cuda plus deploy.resources. volumes: lt-local: name: lt-local external: true
docker compose up –detach, wait until the service is healthy (first model download is slow), then use the documented API. LibreTranslate will detect-and-translate in one call; that detection is langdetect, not Lingua.
This host has no Docker binary. The digest was fetched from the Hub API, not by pulling the image. We did not run a translation through this compose here.
Two measured failure modes, from the paper that actually used this stack at crawl scale [4Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)]:
- Short interactive text. “LibreTranslate has trouble translating the short texts of interactive elements in some languages. For example, the Greek version of “Accept all cookies” translates to “Cookie policy” in English.” They dropped Greek entirely.
- Language coverage is the crawl's coverage. spaCy + LibreTranslate supported their eleven EU languages; missing a language is a sampling decision, not a later filter.
LibreTranslate is not free in CPU. In our crawls it used on the order of 20–50% of the machine.2) Budget it as a sidecar, or you will think the crawler is slow.
LLMs, and the case for not translating
Local LLMs (Ollama and friends) translate short strings better than Argos and long documents worse: attention drops clauses, repeats, or leaves runs untranslated. They are much slower than LibreTranslate, depending on the model.3)
For classification, translating first is the 2024 answer and not the 2026 one. Nenadić et al. extract legally relevant labels from German, French, Italian and English privacy policies “in a single inference without requiring translation” and report F1 above 0.90 for most language/disclosure pairs on 120 expert-annotated policies [3Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)]. Tang et al. already noted that BERTopic “works with multilingual input and does not need translation” [5Tang, Jenny; Alvarez, Léo; Brar, Arjun; Hoang, Nguyen Phong; Christin, Nicolas (2024): "Automatic Generation of Web Censorship Probe Lists", Proceedings on Privacy Enhancing Technologies 2024(4):44-60. (DOI)]. If the downstream model speaks the language, skip the translator; you cannot debug a bad label that was produced from a bad translation of a short button.
Keep the pipeline in the source language
The previous version of this page listed five jobs and left interaction as a 
. The jobs are still the right list. The missing citations are the two papers above plus keyword practice from the cookie-notice crawl.
- Detect (above).
- Switch the site (above).
- Interact with language-specific keywords.
- Classify in-language, or translate then classify, and say which.
- Encode the bytes you store.
Interaction
Most crawlers find a cookie notice, a login, or a language switcher by matching keywords on links and buttons. Those keywords are not one Google-translated list.
Bouhoula et al. detect the notice with language-specific spaCy models and language-specific variants of “cookie”, then translate interactive-element text into English and classify accept / reject / close / save with one English model [4Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)]. When LibreTranslate mangled short strings, they hand-corrected the German and Danish translations and dropped Greek entirely. That is the method: do not ship an uncorrected English keyword list through a translator. Translating an English accept-button list through LibreTranslate is how you get “Cookie policy” as the accept button.
A looser alternative is to crawl first in English-only and claim a limitation. Xie et al. ran Hosseini's eight-library vote and then excluded domains whose homepage or policies were non-English — 48,178 of 101,249 domains in their funnel [8Xie, Qinge; Ramakrishnan, Karthik; Li, Frank (2025): "Evaluating Privacy Policies under Modern Privacy Laws At Scale: An LLM-Based Automated Approach", in: Proceedings of the USENIX Security Symposium, pp. 5797-5816. (Link)]. That is a valid English-only study. It is not multilingual support. Write it as a sampling cut.
Classification
- 2026, current for documents: multilingual LLM, no translation [3Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)]. Fine-tune or prompt in the source language. Validate per language; they annotated 30 policies in each of four languages, and F1 above 0.90 is for most language/disclosure pairs, not a licence to skip that gold set.
- 2024, current for short UI text if you must share one English classifier: translate, then BERT [4Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)]. Budget the translator, and do not trust it on short buttons.
- Historical: mBERT / XLM-R fine-tuned on multilingual training data. Still faster than a local LLM at crawl volume. Yahoo's production page classifier (outside this corpus; see Website classification) takes full content or the URL alone.
Generating multilingual training data with an LLM from an English gold set is a data-augmentation trick, not a substitute for per-language validation.
Bytes
UTF-8 end to end. NFC if you will compare strings. A Latin-1 round-trip on a UTF-8 page is a silent corruption of every non-ASCII token, which on this page is the object of study. The extraction behind this wiki reads paper.cols.txt as latin1 on purpose (byte-preserving); do not copy that for page content you will language-detect.
What to report
A methods paragraph that names “the multilingual web” without these is not reconstructable:
- Which languages, with ISO 639-1 / BCP-47 tags, and whether English was a target or a fallback.
- How the site was switched into that language: vantage (with provider),
Accept-Language/ browser locale, URL, click. “We crawled from the EU” is a location, not a language. - How language was decided on each page: detector (and version), ensemble, Unicode-script rule, or a human. The threshold (LangCrUX: 50% visible text; Hosseini: ≥10 tokens).
- Whether you translated, with the engine, the pin (digest, not
:latest), and which languages you dropped because the engine failed. - Per-language validation on the labels you publish, not a micro-F1 pooled across German and Greek.
Related pages
- Website selection — lists. LangCrUX is a language-filtered CrUX, not a Tranco of “the web”.
- Crawling location — the IP that decides the language you are served.
- Interaction — the click; this page is the keyword list behind it.
- Docker — digest and
/dev/shm. This compose is a translator sidecar. - Website classification — topic, not language ID.
- Corpus — venue window and provisional 2025–2026.
Methodology and limitations of these figures
Paper counts from the 5,859-paper extraction (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P, 2010–2026). Denominator for language handling is 1,080 crawled papers with a crawlConfig, not 5,859 and not 1,120. Tool counts are of papers that used a language-detection or translation tool. Sentinels are not answers. Language names and tool names were folded; residue is in the report. 2025–2026 venue-years are incomplete by construction. Queries, the fold, Hub digests and the archived-repo checks: multilingual_support. Hosseini et al. [6Hosseini, Henry; Degeling, Martin; Utz, Christine; Hupperich, Thomas (2021): "Unifying privacy policy detection", Proceedings on Privacy Enhancing Technologies 2021:480–499. (DOI)] is in the bibliographic index and not in extractions.jsonl; figures from it are from the publisher PDF.
- [1]
- Mhaidli, Abraham; Fidan, Selin; Doan, An; Herakovic, Gina; Srinath, Mukund; Matheson, Lee; Wilson, Shomir; Schaub, Florian (2023): "Researchers’ Experiences in Analyzing Privacy Policies: Challenges and Opportunities", Proceedings on Privacy Enhancing Technologies 2023:287-305. (DOI) (Link)
- [2]
- Bhuiyan, Masudul Hasan Masud; Varvello, Matteo; Zaki, Yasir; Staicu, Cristian-Alexandru (2025): "Not All Visitors are Bilingual: A Measurement Study of the Multilingual Web from an Accessibility Perspective", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
- [3]
- Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)
- [4]
- Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)
- [5]
- Tang, Jenny; Alvarez, Léo; Brar, Arjun; Hoang, Nguyen Phong; Christin, Nicolas (2024): "Automatic Generation of Web Censorship Probe Lists", Proceedings on Privacy Enhancing Technologies 2024(4):44-60. (DOI)
- [6]
- Hosseini, Henry; Degeling, Martin; Utz, Christine; Hupperich, Thomas (2021): "Unifying privacy policy detection", Proceedings on Privacy Enhancing Technologies 2021:480–499. (DOI)
- [7]
- Degeling, Martin; Utz, Christine; Lentzsch, Christopher; Hosseini, Henry; Schaub, Florian; Holz, Thorsten (2019): "We Value Your Privacy ... Now Take Some Cookies: Measuring the GDPR's Impact on Web Privacy", in: Proceedings of the Network and Distributed System Security Symposium. (Link)
- [8]
- Xie, Qinge; Ramakrishnan, Karthik; Li, Frank (2025): "Evaluating Privacy Policies under Modern Privacy Laws At Scale: An LLM-Based Automated Approach", in: Proceedings of the USENIX Security Symposium, pp. 5797-5816. (Link)
- [9]
- Hils, Maximilian; Woods, Daniel W.; Böhme, Rainer (2021): "Privacy Preference Signals: Past, Present and Future", in: Proceedings on Privacy Enhancing Technologies. (DOI)
- [10]
- Yeung, Christina; Iqbal, Umar; O'Neil, Yekaterina Tsipenyuk; Kohno, Tadayoshi; Roesner, Franziska (2023): "Online Advertising in Ukraine and Russia During the 2022 Russian Invasion", in: Proceedings of the ACM Web Conference. (DOI)