Differences

This shows you the differences between two versions of the page.

--- programming:crawler:foxhound [2026/08/17 17:39] – New page: SAP Project Foxhound — what taint flows it reports (character ranges plus the operation chain), the 34/61 pref surface and its version history, what it cannot see (strings only, executed paths only, taintfox.escapeURL), the build burden from the karel.kubicek.claude
+++ programming:crawler:foxhound [2026/08/17 17:56] (current) – Fix a false negative in the published reducer, found by the figures re-review and reproduced before fixing: begin/end are UTF-16 code-unit offsets, and slicing with them in Python (which indexes by code point) reads the wrong substring whenever an astral karel.kubicek.claude
@@ Line 23: / Line 23: @@
 The **taint metadata** — ''report.detail.str.taint'', an array with one entry per tainted range:
-  * ''begin'' / ''end'' — **character offsets into ''str''**. This is the part people miss: Foxhound tells you //which five characters// of a 22-character string came from the source, so "a tainted value reached ''innerHTML''" can be qualified by how much of the sink argument the attacker controls.
+  * ''begin'' / ''end'' — **UTF-16 code-unit offsets into ''str''**, because a SpiderMonkey string is UTF-16. Slice with them in a language that indexes by code point — Python, Go, Rust — and a single emoji earlier in the string silently shifts every later index, so you read the wrong substring and can conclude a dangerous flow was harmless.  This is the part people miss: Foxhound tells you //which five characters// of a 22-character string came from the source, so "a tainted value reached ''innerHTML''" can be qualified by how much of the sink argument the attacker controls.
   * ''flow'' — an array of operation nodes, ordered **sink first, source last**. Each node carries ''operation'' (''concat'', ''substr'', ''unescape'', ''innerHTML'', ''location.hash'', or ''function'' for an application call), ''builtin'', ''source'', ''arguments'', and a ''location'' with ''filename'', ''line'', ''pos'', ''scriptline'' and a ''scripthash''.
@@ Line 79: / Line 79: @@
 **Report ''taintfox.escapeURL''.** This is the configuration knob with the largest measured effect on a published number, it is not one of the 96 prefs above, and almost nobody mentions it. Stock Foxhound **does not percent-encode URL components the way stock Firefox does**: ''netwerk/base/nsStandardURL.cpp'' reads ''Preferences::GetBool("taintfox.escapeURL", false)'' on the main thread, so escaping is off by default. The maintainer's stated reason is that they "initially disabled URL encoding in order to mimic the behavior of legacy browsers (such as IE), which didn't encode parts of the URL (in particular the hash)".((Maintainer comment on issue [[https://github.com/SAP/project-foxhound/issues/259|#259]], 2025-02-14, which is also where the pref name is documented. Pref name, call site and the ''false'' default read out of ''netwerk/base/nsStandardURL.cpp'' on ''main'', 2026-08-17. The pref is **not declared in ''modules/libpref/init/all.js''**, so it does not appear in ''about:config'' until you type it — issue [[https://github.com/SAP/project-foxhound/issues/260|#260]], still open, asks for it to be renamed and added to the default config. Note also that the guard is main-thread-only: on worker threads escaping stays on.))
-What it costs to ignore: Sabino et al. re-ran their confirmation stage with escaping off and **quintupled the confirmed DOM-XSS flows, from 68 to 347**, while their manual sample found the extra ones "would not be exploitable in any modern browser". Their own reading of the apparent decade-long decline in DOM-XSS prevalence is that this is one of the factors behind it — "modern browsers encode any special characters in URLs" — rather than the web having got safer {[sabino2026_detection]}. Whichever way you set it, say so and say what the other setting would have produced.
+What it costs to ignore: Sabino et al. re-ran their confirmation stage with escaping off and **quintupled the confirmed DOM-XSS flows, from 68 to 347**, while their manual sample found the extra ones "would not be exploitable in any modern browser". Their own reading of the apparent decade-long decline in DOM-XSS prevalence is that "several factors, such as dataset and methodology, may contribute" and that this is the first of them — "modern browsers encode any special characters in URLs". The others they name include genuine improvements, such as better ad blocking and HTTPS/mixed-content enforcement, so treat this as one measurement artefact among real effects rather than as a debunking {[sabino2026_detection]}. Whichever way you set the pref, say so, and say what the other setting would have produced.
 </WRAP>
@@ Line 236: / Line 236: @@
 # Sinks where HTML or JavaScript syntax in the tainted substring is what makes a
-# flow dangerous. For a network sink, syntax characters are irrelevant.
+# flow dangerous. `iframe.srcdoc` is in here because its content is parsed as a
+# whole HTML document, exactly like `document.write`. Navigation sinks
+# (`location.href`, `a.href`, `window.open`) are deliberately NOT here: they are
+# dangerous through the URL scheme (`javascript:`), which this syntax screen does
+# not model, so silence about them is honest rather than reassuring.
 HTML_JS_SINKS = {
     "innerHTML", "outerHTML", "insertAdjacentHTML", "document.write",
     "document.writeln", "eval", "Function.ctor", "script.text",
-    "script.innerHTML", "eventHandler", "setTimeout", "setInterval",
+    "script.innerHTML", "script.textContent", "eventHandler", "setTimeout",
-    "Range.createContextualFragment(fragment)",
+    "setInterval", "Range.createContextualFragment(fragment)", "iframe.srcdoc",
 }
 ENCODING_OPS = {"encodeURI", "encodeURIComponent", "escape"}
@@ Line 253: / Line 257: @@
 # call that reports the flow as an ordinary `function` node, so without this the
 # JIT-blindness metric and the scripthash key both describe your own code.
-# Override with --harness if your harness script is named differently.
+# This is a NAME heuristic and it cuts both ways: a page function called
+# something like `reportTaintSinkStats` would be misclassified as harness code.
+# Set --harness to a pattern that matches your harness and nothing else.
 HARNESS_RE = re.compile(r"taint_reporting|ReportTaintSink|__playwright", re.I)
@@ Line 277: / Line 283: @@
 def script_of(flow: list[dict], harness_re: re.Pattern) -> tuple[str, str]:
-    """(scripthash, filename) of the node nearest the sink that is page code."""
+    """(scripthash, filename) of the node nearest the sink that is page code.
+    A node can carry a filename with no scripthash (inline handlers, and nodes
+    the engine could not attribute to a compiled script). Keep the first such
+    filename so `--unit script` has something to fall back on rather than
+    silently collapsing every unhashed flow into one empty key.
+    """
+    fallback = ""
     for node in flow:
         if is_harness(node, harness_re):
@@ Line 284: / Line 297: @@
         if loc.get("scripthash"):
             return loc["scripthash"], loc.get("filename", "")
-    return "", ""
+        if not fallback and loc.get("filename"):
+            fallback = loc["filename"]
+    return "", fallback
@@ Line 308: / Line 323: @@
             return False
     return False
+def utf16_slice(value: str, begin: int, end: int) -> str:
+    """Slice `value` by UTF-16 code units, which is how the engine counts.
+    Foxhound's `begin`/`end` are offsets into a SpiderMonkey string, and JS
+    strings are UTF-16. Python slices by CODE POINT, so a single character
+    outside the Basic Multilingual Plane anywhere earlier in the string (an
+    emoji, some CJK extensions) shifts every later Python index by one and the
+    slice silently returns the wrong substring. On "\U0001F600\U0001F600<>PADDING"
+    the engine's offsets 4..6 bound "<>", while `value[4:6]` returns "PA" — which
+    would report a dangerous flow as having held no syntax character.
+    """
+    units = value.encode("utf-16-le")
+    return units[2 * begin:2 * end].decode("utf-16-le", errors="replace")
+def utf16_len(value: str) -> int:
+    """Length of `value` in UTF-16 code units, i.e. in the engine's own unit."""
+    return len(value.encode("utf-16-le")) // 2
@@ Line 318: / Line 353: @@
     if sink not in HTML_JS_SINKS:
         return False
-    substring = value[taint_range["begin"]:taint_range["end"]]
+    substring = utf16_slice(value, taint_range["begin"], taint_range["end"])
     return not DANGEROUS.search(substring)
@@ Line 340: / Line 375: @@
                 "scripthash": scripthash,
                 "script": filename,
+                # In UTF-16 code units, the unit the offsets are expressed in.
                 "chars": taint_range["end"] - taint_range["begin"],
+                "chars_of": utf16_len(value),
                 "operations": [n["operation"] for n in flow],
                 "jit_blind": is_jit_blind(flow, harness_re),
@@ Line 526: / Line 563: @@
           "d7069063759edbf2dcf45741802bc405")
     check("harness script still does not clear jit_blind", flows([hh])[0]["jit_blind"], True)
+    # A filename with no scripthash is still a usable key for --unit script.
+    nohash = copy.deepcopy(WIKI_EXAMPLE)
+    for n in nohash["detail"]["str_taint"][0]["flow"]:
+        loc = n.get("location") or {}
+        loc.pop("scripthash", None)
+        loc["filename"] = "https://domgo.at/inline"
+        n["location"] = loc
+    nh = flows([nohash])[0]
+    check("no scripthash -> empty hash", nh["scripthash"], "")
+    check("no scripthash -> filename fallback", nh["script"], "https://domgo.at/inline")
+    check("script unit falls back to filename", UNITS["script"](nh), "https://domgo.at/inline")
+    # UTF-16 offsets: an astral character before the range must not shift it.
+    astral = copy.deepcopy(WIKI_EXAMPLE)
+    astral["detail"]["str"] = "\U0001F600\U0001F600<>PADDING"
+    astral["detail"]["str_taint"][0]["begin"] = 4
+    astral["detail"]["str_taint"][0]["end"] = 6
+    check("utf16_slice finds the real substring", utf16_slice(astral["detail"]["str"], 4, 6), "<>")
+    check("naive python slice would have been wrong", astral["detail"]["str"][4:6], "PA")
+    check("astral shift does not hide a dangerous substring",
+          flows([astral])[0]["no_syntax_chars"], False)
+    check("string length is counted in UTF-16 units", utf16_len(astral["detail"]["str"]), 13)
+    # iframe.srcdoc is parsed as HTML, so it is in the syntax-screen sink set.
+    srcdoc = copy.deepcopy(WIKI_EXAMPLE)
+    srcdoc["detail"]["sink"] = "iframe.srcdoc"
+    check("srcdoc is screened", flows([srcdoc])[0]["no_syntax_chars"], True)
+    nav = copy.deepcopy(WIKI_EXAMPLE)
+    nav["detail"]["sink"] = "location.href"
+    check("navigation sinks are not screened", flows([nav])[0]["no_syntax_chars"], False)
+    # An empty flow must not crash and must not be attributed to anything.
+    empty = copy.deepcopy(WIKI_EXAMPLE)
+    empty["detail"]["str_taint"][0]["flow"] = []
+    er = flows([empty])[0]
+    check("empty flow source", er["source"], "unattributed")
+    check("empty flow is jit_blind", er["jit_blind"], True)
+    check("empty flow not encoded", er["encoded_at_sink"], False)
     # No source-flagged node must not be relabelled.
@@ Line 589: / Line 665: @@
 </file>
-Its self-test runs the flow from the project's own documentation plus seven mutations of it — including the two that matter most, a harness-only ''function'' node and an encode followed by a decode — and its real output is:
+Its self-test runs the flow from the project's own documentation plus thirteen mutations of it — including the four that matter most: a harness-only ''function'' node, an encode followed by a decode, a location with no ''scripthash'', and a tainted range sitting behind an astral character — and its real output is:
 <code>
 $ python3 foxhound_flows.py --selftest
-selftest: 19 checks passed
+selftest: 31 checks passed
 flows: 1   sites: 1   pages: 1   scripts: 1
@@ Line 663: / Line 739: @@
   * The population is a **full-text sweep, not a schema query**: the extraction's ''tools[]'' finds 9 of the 13 papers and misses both citation-only cases, which are the ones the page has to separate out. Every paper's role — used, extended, baseline, citation, homograph — was decided by reading the sentence, and the report fails loudly if the sweep and the hand map ever disagree. No name fold was needed: the three spellings in the corpus (''Foxhound'', ''FoxHound'', ''Project Foxhound'') are case variants of one token with no synonyms, so the residue is empty by construction rather than by luck.
   * The 2025–2026 rows are provisional, and the seven venues exclude EuroS&P (where the browser is described), ACSAC, RAID, AsiaCCS, CHI and SOUPS. Every count here is a lower bound on the tool's real use, and the project's own 14-entry publication list is the easiest demonstration of that.
-  * The nine schema evidence quotes were checked against ''paper.cols.txt'': 5 exact, 4 partial under the dataset's five-word-window test, 0 below threshold. The 13 hand-picked deciding quotes are all exact, two of them only after being shortened to a contiguous fragment because the surrounding sentence is spliced across columns in the repaired rendering. Every per-paper figure and quoted sentence used above — 48 of them — was located verbatim in the source text.
+  * The nine schema evidence quotes were checked against ''paper.cols.txt'': 5 exact, 4 partial under the dataset's five-word-window test, 0 below threshold. The 13 hand-picked deciding quotes are all exact, two of them only after being shortened to a contiguous fragment because the surrounding sentence is spliced across columns in the repaired rendering. Every per-paper figure and quoted sentence used above — 50 of them — was located verbatim in the source text.
   * The contributor-overlap figure counts an exact, diacritic-folded name match against the README's contributor list, which is explicitly not exhaustive. It is a lower bound in both directions.
   * Nothing on this page is a benchmark we ran. The build times are the project's CI on hosted runners, the overhead figures are Calzavara et al.'s on landing pages, and no source gives a per-page crawl rate for Foxhound on a modern population.