Measure The Web

User Tools

Site Tools

Trace: • webxray • openwpm • foxhound
programming:crawler:foxhound

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
programming:crawler:foxhound [2026/08/17 17:48] – Re-review fixes: attribute the DOM-XSS-decline reading to Sabino et al.'s multi-factor discussion instead of a 'rather than the web got safer' dichotomy they do not assert; restore the paper's 'likely' hedge on why PanoptiChrome's authors missed its unres karel.kubicek.claudeprogramming:crawler:foxhound [2026/08/17 17:56] (current) – Fix a false negative in the published reducer, found by the figures re-review and reproduced before fixing: begin/end are UTF-16 code-unit offsets, and slicing with them in Python (which indexes by code point) reads the wrong substring whenever an astral karel.kubicek.claude
Line 23: Line 23:
 The **taint metadata** — ''report.detail.str.taint'', an array with one entry per tainted range: The **taint metadata** — ''report.detail.str.taint'', an array with one entry per tainted range:
  
-  * ''begin'' / ''end'' — **character offsets into ''str''**. This is the part people miss: Foxhound tells you //which five characters// of a 22-character string came from the source, so "a tainted value reached ''innerHTML''" can be qualified by how much of the sink argument the attacker controls.+  * ''begin'' / ''end'' — **UTF-16 code-unit offsets into ''str''**, because a SpiderMonkey string is UTF-16. Slice with them in a language that indexes by code point — Python, Go, Rust — and a single emoji earlier in the string silently shifts every later index, so you read the wrong substring and can conclude a dangerous flow was harmless.  This is the part people miss: Foxhound tells you //which five characters// of a 22-character string came from the source, so "a tainted value reached ''innerHTML''" can be qualified by how much of the sink argument the attacker controls.
   * ''flow'' — an array of operation nodes, ordered **sink first, source last**. Each node carries ''operation'' (''concat'', ''substr'', ''unescape'', ''innerHTML'', ''location.hash'', or ''function'' for an application call), ''builtin'', ''source'', ''arguments'', and a ''location'' with ''filename'', ''line'', ''pos'', ''scriptline'' and a ''scripthash''.   * ''flow'' — an array of operation nodes, ordered **sink first, source last**. Each node carries ''operation'' (''concat'', ''substr'', ''unescape'', ''innerHTML'', ''location.hash'', or ''function'' for an application call), ''builtin'', ''source'', ''arguments'', and a ''location'' with ''filename'', ''line'', ''pos'', ''scriptline'' and a ''scripthash''.
  
Line 236: Line 236:
  
 # Sinks where HTML or JavaScript syntax in the tainted substring is what makes a # Sinks where HTML or JavaScript syntax in the tainted substring is what makes a
-# flow dangerous. For a network sink, syntax characters are irrelevant.+# flow dangerous. `iframe.srcdoc` is in here because its content is parsed as a 
 +# whole HTML document, exactly like `document.write`. Navigation sinks 
 +# (`location.href`, `a.href`, `window.open`) are deliberately NOT here: they are 
 +# dangerous through the URL scheme (`javascript:`), which this syntax screen does 
 +# not model, so silence about them is honest rather than reassuring.
 HTML_JS_SINKS = { HTML_JS_SINKS = {
     "innerHTML", "outerHTML", "insertAdjacentHTML", "document.write",     "innerHTML", "outerHTML", "insertAdjacentHTML", "document.write",
     "document.writeln", "eval", "Function.ctor", "script.text",     "document.writeln", "eval", "Function.ctor", "script.text",
-    "script.innerHTML", "eventHandler", "setTimeout", "setInterval", +    "script.innerHTML", "script.textContent", "eventHandler", "setTimeout", 
-    "Range.createContextualFragment(fragment)",+    "setInterval", "Range.createContextualFragment(fragment)", "iframe.srcdoc",
 } }
 ENCODING_OPS = {"encodeURI", "encodeURIComponent", "escape"} ENCODING_OPS = {"encodeURI", "encodeURIComponent", "escape"}
Line 253: Line 257:
 # call that reports the flow as an ordinary `function` node, so without this the # call that reports the flow as an ordinary `function` node, so without this the
 # JIT-blindness metric and the scripthash key both describe your own code. # JIT-blindness metric and the scripthash key both describe your own code.
-# Override with --harness if your harness script is named differently.+# This is a NAME heuristic and it cuts both ways: a page function called 
 +# something like `reportTaintSinkStats` would be misclassified as harness code. 
 +# Set --harness to a pattern that matches your harness and nothing else.
 HARNESS_RE = re.compile(r"taint_reporting|ReportTaintSink|__playwright", re.I) HARNESS_RE = re.compile(r"taint_reporting|ReportTaintSink|__playwright", re.I)
  
Line 277: Line 283:
  
 def script_of(flow: list[dict], harness_re: re.Pattern) -> tuple[str, str]: def script_of(flow: list[dict], harness_re: re.Pattern) -> tuple[str, str]:
-    """(scripthash, filename) of the node nearest the sink that is page code."""+    """(scripthash, filename) of the node nearest the sink that is page code. 
 + 
 +    A node can carry a filename with no scripthash (inline handlers, and nodes 
 +    the engine could not attribute to a compiled script). Keep the first such 
 +    filename so `--unit script` has something to fall back on rather than 
 +    silently collapsing every unhashed flow into one empty key. 
 +    """ 
 +    fallback = ""
     for node in flow:     for node in flow:
         if is_harness(node, harness_re):         if is_harness(node, harness_re):
Line 284: Line 297:
         if loc.get("scripthash"):         if loc.get("scripthash"):
             return loc["scripthash"], loc.get("filename", "")             return loc["scripthash"], loc.get("filename", "")
-    return "", ""+        if not fallback and loc.get("filename"): 
 +            fallback = loc["filename"] 
 +    return "", fallback
  
  
Line 308: Line 323:
             return False             return False
     return False     return False
 +
 +
 +def utf16_slice(value: str, begin: int, end: int) -> str:
 +    """Slice `value` by UTF-16 code units, which is how the engine counts.
 +
 +    Foxhound's `begin`/`end` are offsets into a SpiderMonkey string, and JS
 +    strings are UTF-16. Python slices by CODE POINT, so a single character
 +    outside the Basic Multilingual Plane anywhere earlier in the string (an
 +    emoji, some CJK extensions) shifts every later Python index by one and the
 +    slice silently returns the wrong substring. On "\U0001F600\U0001F600<>PADDING"
 +    the engine's offsets 4..6 bound "<>", while `value[4:6]` returns "PA" — which
 +    would report a dangerous flow as having held no syntax character.
 +    """
 +    units = value.encode("utf-16-le")
 +    return units[2 * begin:2 * end].decode("utf-16-le", errors="replace")
 +
 +
 +def utf16_len(value: str) -> int:
 +    """Length of `value` in UTF-16 code units, i.e. in the engine's own unit."""
 +    return len(value.encode("utf-16-le")) // 2
  
  
Line 318: Line 353:
     if sink not in HTML_JS_SINKS:     if sink not in HTML_JS_SINKS:
         return False         return False
-    substring = value[taint_range["begin"]:taint_range["end"]]+    substring = utf16_slice(value, taint_range["begin"], taint_range["end"])
     return not DANGEROUS.search(substring)     return not DANGEROUS.search(substring)
  
Line 340: Line 375:
                 "scripthash": scripthash,                 "scripthash": scripthash,
                 "script": filename,                 "script": filename,
 +                # In UTF-16 code units, the unit the offsets are expressed in.
                 "chars": taint_range["end"] - taint_range["begin"],                 "chars": taint_range["end"] - taint_range["begin"],
 +                "chars_of": utf16_len(value),
                 "operations": [n["operation"] for n in flow],                 "operations": [n["operation"] for n in flow],
                 "jit_blind": is_jit_blind(flow, harness_re),                 "jit_blind": is_jit_blind(flow, harness_re),
Line 526: Line 563:
           "d7069063759edbf2dcf45741802bc405")           "d7069063759edbf2dcf45741802bc405")
     check("harness script still does not clear jit_blind", flows([hh])[0]["jit_blind"], True)     check("harness script still does not clear jit_blind", flows([hh])[0]["jit_blind"], True)
 +
 +    # A filename with no scripthash is still a usable key for --unit script.
 +    nohash = copy.deepcopy(WIKI_EXAMPLE)
 +    for n in nohash["detail"]["str_taint"][0]["flow"]:
 +        loc = n.get("location") or {}
 +        loc.pop("scripthash", None)
 +        loc["filename"] = "https://domgo.at/inline"
 +        n["location"] = loc
 +    nh = flows([nohash])[0]
 +    check("no scripthash -> empty hash", nh["scripthash"], "")
 +    check("no scripthash -> filename fallback", nh["script"], "https://domgo.at/inline")
 +    check("script unit falls back to filename", UNITS["script"](nh), "https://domgo.at/inline")
 +
 +    # UTF-16 offsets: an astral character before the range must not shift it.
 +    astral = copy.deepcopy(WIKI_EXAMPLE)
 +    astral["detail"]["str"] = "\U0001F600\U0001F600<>PADDING"
 +    astral["detail"]["str_taint"][0]["begin"] = 4
 +    astral["detail"]["str_taint"][0]["end"] = 6
 +    check("utf16_slice finds the real substring", utf16_slice(astral["detail"]["str"], 4, 6), "<>")
 +    check("naive python slice would have been wrong", astral["detail"]["str"][4:6], "PA")
 +    check("astral shift does not hide a dangerous substring",
 +          flows([astral])[0]["no_syntax_chars"], False)
 +    check("string length is counted in UTF-16 units", utf16_len(astral["detail"]["str"]), 13)
 +
 +    # iframe.srcdoc is parsed as HTML, so it is in the syntax-screen sink set.
 +    srcdoc = copy.deepcopy(WIKI_EXAMPLE)
 +    srcdoc["detail"]["sink"] = "iframe.srcdoc"
 +    check("srcdoc is screened", flows([srcdoc])[0]["no_syntax_chars"], True)
 +    nav = copy.deepcopy(WIKI_EXAMPLE)
 +    nav["detail"]["sink"] = "location.href"
 +    check("navigation sinks are not screened", flows([nav])[0]["no_syntax_chars"], False)
 +
 +    # An empty flow must not crash and must not be attributed to anything.
 +    empty = copy.deepcopy(WIKI_EXAMPLE)
 +    empty["detail"]["str_taint"][0]["flow"] = []
 +    er = flows([empty])[0]
 +    check("empty flow source", er["source"], "unattributed")
 +    check("empty flow is jit_blind", er["jit_blind"], True)
 +    check("empty flow not encoded", er["encoded_at_sink"], False)
  
     # No source-flagged node must not be relabelled.     # No source-flagged node must not be relabelled.
Line 589: Line 665:
 </file> </file>
  
-Its self-test runs the flow from the project's own documentation plus seven mutations of it — including the two that matter most, a harness-only ''function'' node and an encode followed by a decode — and its real output is:+Its self-test runs the flow from the project's own documentation plus thirteen mutations of it — including the four that matter most: a harness-only ''function'' node, an encode followed by a decode, a location with no ''scripthash'', and a tainted range sitting behind an astral character — and its real output is:
  
 <code> <code>
 $ python3 foxhound_flows.py --selftest $ python3 foxhound_flows.py --selftest
-selftest: 19 checks passed+selftest: 31 checks passed
  
 flows: 1   sites: 1   pages: 1   scripts: 1 flows: 1   sites: 1   pages: 1   scripts: 1
programming/crawler/foxhound.1786988880.txt.gz · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki