| Next revision | Previous revision |
| design:user_studies [2026/08/27 14:18] – New page: user studies as the other top-level Design branch (N=1357 humanSubjects). Authored by Claude. karel.kubicek.claude | design:user_studies [2026/08/27 14:26] (current) – Apply generic-review wording: extraction proxy, recovered-crossover bound, ethics pointer, Qualtrics upper bound. Authored by Claude. karel.kubicek.claude |
|---|
| | annotatorType ''crowdworkers'' | crowdworkers as **labelers** | 73 | 1.2% | | | annotatorType ''crowdworkers'' | crowdworkers as **labelers** | 73 | 1.2% | |
| |
| **Every figure below is on ''humanSubjects'' (N=1,357) unless it says otherwise.** | **Every figure below is on ''humanSubjects'' (N=1,357) unless it says otherwise.** That is an extraction signal — ''participants[]'' non-empty — not a hand-adjudicated IRB determination. A paper can have participants the extractor missed, and a ''participants[]'' tuple can be a visitor log or a donated panel rather than a recruited sample (see n≥10,000 below). |
| |
| The type tag and the participants array disagree in both directions: **1,125** papers have both, **232** have ''participants[]'' but no ''user-study'' type, **24** have the type and an empty ''participants[]''. The type-only papers are crawls, telemetry, or crowd //annotation// the extractor filed as a study type. Unioning the two would invent a population that is neither. The item brief's "965 papers" was computed on the old 4,322-paper corpus and is stale; the current count is **1,357**. | The type tag and the participants array disagree in both directions: **1,125** papers have both, **232** have ''participants[]'' but no ''user-study'' type, **24** have the type and an empty ''participants[]''. The type-only papers are crawls, telemetry, or crowd //annotation// the extractor filed as a study type. Unioning the two would invent a population that is neither. The item brief's "965 papers" was computed on the old 4,322-paper corpus and is stale; the current count is **1,357**. |
| ===== How the Field Recruits ===== | ===== How the Field Recruits ===== |
| |
| The schema has a recruitment enum. **903 of 1,357 (66.5%)** state a channel; **454 (33.5%)** produce only a sentinel. Sentinel-only is the finding: one human-subjects paper in three does not name how people were recruited. Sentinels are never counted as a channel. | The schema has a recruitment enum. **903 of 1,357 (66.5%)** state a channel; **454 (33.5%)** produce only a sentinel. Sentinel-only is the finding: in the extraction, one human-subjects paper in three has no named recruitment channel. Sentinels are never counted as a channel. |
| |
| ^ recruitment (schema enum) ^ Papers ^ Share of 1,357 ^ | ^ recruitment (schema enum) ^ Papers ^ Share of 1,357 ^ |
| **Schema is a lower bound. Full-text recovery is an upper bound.** Schema MTurk is 49.8% of the recovered-MTurk set; schema Prolific is 31.4% of recovered-Prolific. Union of recovered MTurk∪Prolific: **409 of 1,357 (30.1%)**; 64 papers hit both. Of the 193 ''other-crowd-platform'' papers, 168 name one of the families above and **25 name nothing recoverable** — listed in full on the provenance page. | **Schema is a lower bound. Full-text recovery is an upper bound.** Schema MTurk is 49.8% of the recovered-MTurk set; schema Prolific is 31.4% of recovered-Prolific. Union of recovered MTurk∪Prolific: **409 of 1,357 (30.1%)**; 64 papers hit both. Of the 193 ''other-crowd-platform'' papers, 168 name one of the families above and **25 name nothing recoverable** — listed in full on the provenance page. |
| |
| Qualtrics full-text hits **124 of 1,357** human-subjects papers. It is the survey **tool**, not a recruitment family, and it is not in the table. | Qualtrics full-text hits **124 of 1,357** human-subjects papers. That is an **upper bound** (the same related-work trap as the crowd-platform recovery). It is the survey **tool**, not a recruitment family, and it is not in the table. |
| |
| ==== Which platform is current ==== | ==== Which platform is current ==== |
| | 2025–2026* | 303 | 7 | 24 | 34 | 70 | | | 2025–2026* | 303 | 7 | 24 | 34 | 70 | |
| |
| Per calendar year, recovered Prolific **overtakes** recovered MTurk in **2024** (39 vs 28), and the gap is wide in the provisional 2025–2026 slice (46 vs 19 in 2025; 24 vs 5 in 2026). Schema-only, the crossover is later: 2025–2026* schema Prolific 34 vs schema MTurk 7. Report both bounds. After 30 September 2026 the MTurk column is historical. | Per calendar year, recovered Prolific **overtakes** recovered MTurk in **2024** (39 vs 28), and the gap is wide in the provisional 2025–2026 slice (46 vs 19 in 2025; 24 vs 5 in 2026). That crossover is on the recovered-full-text series — an **upper bound** on both platforms, because a related-work citation fires. Quote-only recovery already has Prolific ahead overall (102 vs 92). Schema-only, the crossover is later: 2025–2026* schema Prolific 34 vs schema MTurk 7. Report both bounds. After 30 September 2026 the MTurk column is historical. |
| |
| University-pool, professional-network, social-media, snowball, in-person and panel-company are all still live channels. Professional-network is the second schema enum (189) and grew in 2022–2024 (76 of 486). Panel-company is small in the schema (42) and is how the n≥10,000 **survey** giants actually get people — Kantar {[herbert2025_digital]}, Cint {[redmiles2020_comprehensive]}. | University-pool, professional-network, social-media, snowball, in-person and panel-company are all still live channels. Professional-network is the second schema enum (189) and grew in 2022–2024 (76 of 486). Panel-company is small in the schema (42) and is how the n≥10,000 **survey** giants actually get people — Kantar {[herbert2025_digital]}, Cint {[redmiles2020_comprehensive]}. |
| ===== What to Report ===== | ===== What to Report ===== |
| |
| A reviewer reconstructing your sample should not have to guess. Of 1,357 human-subjects papers in these venues, **33.5%** do not name the recruitment channel, **45.0%** do not state compensation, **51.0%** do not state a region. Do not be those papers. | A reviewer reconstructing your sample should not have to guess. Of 1,357 human-subjects papers in these venues, **33.5%** have no named recruitment channel in the extraction, **45.0%** do not state compensation, **51.0%** do not state a region. Do not be those papers. |
| |
| - **Who was recruited, through what channel, on which platform.** "Online" is a sentinel. Name Prolific, the university pool, Kantar, the in-product surface, or the professional list. If they were visitors of a live site rather than recruits, say that — it changes what n means {[utz2019_informed]}. | - **Who was recruited, through what channel, on which platform.** "Online" is a sentinel. Name Prolific, the university pool, Kantar, the in-product surface, or the professional list. If they were visitors of a live site rather than recruits, say that — it changes what n means {[utz2019_informed]}. |
| - **What they were paid, in an amount a reader can convert to an hourly rate.** Prolific's absolute floor on 2026-08-27 is £6 / $8 per hour; recommended is £9 / $12. Kelley et al.'s 25–70 cents {[kelley2012_guess]} is a 2010–2011 fact, not a template. If compensation was optional, say how many took it {[klemmer2025_transparency]}. | - **What they were paid, in an amount a reader can convert to an hourly rate.** Prolific's absolute floor on 2026-08-27 is £6 / $8 per hour; recommended is £9 / $12. Kelley et al.'s 25–70 cents {[kelley2012_guess]} is a 2010–2011 fact, not a template. If compensation was optional, say how many took it {[klemmer2025_transparency]}. |
| - **Where they were, at least at country level.** 62.6% of papers that state a region name the United States. If your sample is WEIRD, Wei et al. {[wei2024_solk]} is the paper a reviewer will cite back at you. If it is not, Herbert et al. {[herbert2025_digital]} is the current existence proof that a 12-country panel is possible. | - **Where they were, at least at country level.** 62.6% of papers that state a region name the United States. If your sample is WEIRD, Wei et al. {[wei2024_solk]} is the paper a reviewer will cite back at you. If it is not, Herbert et al. {[herbert2025_digital]} is the current existence proof that a 12-country panel is possible. |
| - **The ethics determination**, not a vibe. The venues that carry this literature will reject on ethics with the technical review untouched — see [[Practices:Ethics]]. | - **The ethics determination**, not a vibe. What the seven venues require, and that an ethics problem can stop a paper independently of the technical review, is [[Practices:Ethics]] — including the "six of the seven" caveat for TheWebConf. |
| - **Attention checks, and what they actually show.** Pei et al. {[pei2020_attention]} demonstrated they can be passed automatically. Report the check, the fail rate, and that a passed check is not proof of attention. | - **Attention checks, and what they actually show.** Pei et al. {[pei2020_attention]} demonstrated they can be passed automatically. Report the check, the fail rate, and that a passed check is not proof of attention. |
| - **If you also crawled: which claims come from the crawl and which from the people.** Zeber et al. {[zeber2020representativeness]} is the reason the two are not interchangeable. | - **If you also crawled: which claims come from the crawl and which from the people.** Zeber et al. {[zeber2020representativeness]} is the reason the two are not interchangeable. |
| - **Preregistration** if you have hypotheses about people — already normal on this side of the house, almost absent on the crawl side. See [[Statistics:Study preregistration]]. Tests, corrections, and clustered/paired designs: [[Statistics:Hypothesis testing]], [[Statistics:Pvalue corrections]]. Open coding: [[Statistics:Interrater agreement]]. | - **Preregistration** if you have hypotheses about people. Almost all of this corpus's preregistrations sit in participant studies rather than crawls — the counts are on [[Statistics:Study preregistration]], not re-derived here. Tests, corrections, and clustered/paired designs: [[Statistics:Hypothesis testing]], [[Statistics:Pvalue corrections]]. Open coding: [[Statistics:Interrater agreement]]. |
| |
| ===== Live check: the platforms this page named ===== | ===== Live check: the platforms this page named ===== |
| 'permanently close' -> 'r 30, 2026. Amazon Mechanical Turk will permanently close on September 30, 2026. For Workers and Requesters currently using the service, please visit our <a href="/help">FAQ</a> pa' | 'permanently close' -> 'r 30, 2026. Amazon Mechanical Turk will permanently close on September 30, 2026. For Workers and Requesters currently using the service, please visit our <a href="/help">FAQ</a> pa' |
| ok mturk-requester https://requester.mturk.com/ | ok mturk-requester https://requester.mturk.com/ |
| 'no longer accepting new customers' -> 'Amazon Mechanical Turk is no longer accepting new customers. We recommend existing customers migrate to a third-party solution.' | 'no longer accepting new customers' -> '9888;️ Amazon Mechanical Turk is no longer accepting new customers. We recommend existing customers migrate to a third-party solution. More information is available <a href=' |
| ok prolific-pricing https://researcher-help.prolific.com/en/articles/445239-what-is-your-pricing | ok prolific-pricing https://researcher-help.prolific.com/en/articles/445239-what-is-your-pricing |
| '£6.00 / $8.00' -> 'ft"><p><b>Absolute minimum allowed:</b> £6.00 / $8.00 per hour</p></div></li></ul></div><div class="intercom-interblocks-paragraph no-margin intercom-interblocks-align-left"><p>You' | '£6.00 / $8.00' -> 'ft"><p><b>Absolute minimum allowed:</b> £6.00 / $8.00 per hour</p></div></li></ul></div><div class="intercom-interblocks-paragraph no-margin intercom-interblocks-align-left"><p>You' |
| ok prolific-pay https://researcher-help.prolific.com/en/articles/445266-how-much-should-i-pay-participants | ok prolific-pay https://researcher-help.prolific.com/en/articles/445266-how-much-should-i-pay-participants |
| '£6' -> '><p>We enforce a minimum hourly rate of £6 / $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the' | '£6' -> '><p>We enforce a minimum hourly rate of £6 / $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the' |
| '$8' -> 'e enforce a minimum hourly rate of £6 / $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the effort' | '$8' -> 'e enforce a minimum hourly rate of £6 / $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the effo' |
| '£9' -> ' £6 / $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the effort required, the participants you' | '£9' -> ' £6 / $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the effort required, the participants you’' |
| '$12' -> ' $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the effort required, the participants you’re ta' | '$12' -> ' $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the effort required, the participants you’re ta' |
| ok cloudresearch https://www.cloudresearch.com/ | ok cloudresearch https://www.cloudresearch.com/ |
