| Next revision | Previous revision |
| programming:docker [2026/08/27 12:45] – Create programming:docker: crawler image digests, /dev/shm, Xvfb. Authored by Claude karel.kubicek.claude | programming:docker [2026/09/17 07:22] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude |
|---|
| |
| <WRAP important> | <WRAP important> |
| **Docker's default ''/dev/shm'' is 64m.** Chromium and Firefox need more than that, or they crash. OpenWPM's README still says so in 2026: without ''%%--shm-size=2g%%'', "you can expect Firefox to crash on 20-30% of sites."(([[https://github.com/openwpm/OpenWPM/blob/master/README.md|OpenWPM README]], section "Docker Deployment for OpenWPM", fetched 2026-08-27.)) Of the **1,120 crawled papers** in this corpus, **0** name ''/dev/shm'', ''shm-size'', or ''--disable-dev-shm-usage''. Of the **5,855 papers with readable full text** (4 of 5,859 have no ''.cols''), **0** name a container image digest (one homograph among the 5,859: "illicit image digests" in a perceptual-hashing paper). A tag is not a pin. | **Docker's default ''/dev/shm'' is 64m.** Chromium and Firefox need more than that, or they crash. OpenWPM's README still says so in 2026: without ''%%--shm-size=2g%%'', "you can expect Firefox to crash on 20-30% of sites."(([[https://github.com/openwpm/OpenWPM/blob/master/README.md|OpenWPM README]], section "Docker Deployment for OpenWPM", fetched 2026-08-27.)) Of the **1,120 crawled papers** in this corpus, **0** name ''/dev/shm'', ''shm-size'', or ''%%--disable-dev-shm-usage%%''. Of the **5,855 papers with readable full text** (4 of 5,859 have no ''.cols''), **0** name a container image digest (one homograph among the 5,859: "illicit image digests" in a perceptual-hashing paper). A tag is not a pin. |
| </WRAP> | </WRAP> |
| |
| </file> | </file> |
| |
| The checker is stdlib Python. It is not a YAML parser: two-space service keys under ''services:'', scalar ''image:'' lines. A file that needs anchors or quoted values with spaces will fail here rather than be half-parsed. A service looks like a crawler if its image name matches playwright, chromium, chrome, firefox, selenium, puppeteer, or openwpm. ''--all'' digest-pins **every** service (so the LibreTranslate sidecar cannot hide behind '':latest'') but still applies the shm rule only to crawler-looking images. Advertised flags: a compose path, ''--demo'', ''--all''. Service properties must sit at four-space indent; a nested ''build.shm_size'' does not count. ''shm_size: 64m'' is not a raise. | The checker is stdlib Python. It is not a YAML parser: two-space service keys under ''services:'', scalar ''image:'' lines. A file that needs anchors or quoted values with spaces will fail here rather than be half-parsed. A service looks like a crawler if its image name matches playwright, chromium, chrome, firefox, selenium, puppeteer, or openwpm. ''%%--all%%'' digest-pins **every** service (so the LibreTranslate sidecar cannot hide behind '':latest'') but still applies the shm rule only to crawler-looking images. Advertised flags: a compose path, ''%%--demo%%'', ''%%--all%%''. Service properties must sit at four-space indent; a nested ''build.shm_size'' does not count. ''shm_size: 64m'' is not a raise. |
| |
| <file python check_crawler_compose.py> | <file python check_crawler_compose.py> |
| </file> | </file> |
| |
| Real output of ''python3 check_crawler_compose.py --demo'' (exit 0), 2026-08-27. The bad fixture is supposed to fail; ''FAILURE'' lines go to stdout and the demo still exits 0 because the self-test saw that failure. A compose file that is actually bad exits 1. | Real output of ''python3 check_crawler_compose.py %%--demo%%'' (exit 0), 2026-08-27. The bad fixture is supposed to fail; ''FAILURE'' lines go to stdout and the demo still exits 0 because the self-test saw that failure. A compose file that is actually bad exits 1. |
| |
| <code> | <code> |
| * [[Design:Longitudinal]] — pinning the environment across waves. 0 papers in this corpus name a container digest; that page already records the same silence. | * [[Design:Longitudinal]] — pinning the environment across waves. 0 papers in this corpus name a container digest; that page already records the same silence. |
| * [[Programming:Stateful stateless]] — whether the profile persists; a new container per visit is one way to reset it. | * [[Programming:Stateful stateless]] — whether the profile persists; a new container per visit is one way to reset it. |
| | * [[Programming:Deployment]] — **where this page stops.** Docker pins the image; that page is the four weeks the container then has to survive: the work queue, retries, monitoring, cost, and what to write when the machine dies mid-run. |
| * [[Literature:Corpus]] — seven-venue scope and the 2025–2026 edge. | * [[Literature:Corpus]] — seven-venue scope and the 2025–2026 edge. |
| |
