Source: How people read the web and what copy changes measurably do
Status: source note, first research pass on 29 September 2026 by a research agent. Sources were fetched and read in that pass unless marked [not verified]; some CXL pages were read via Wayback Machine copies. Grades: strong, moderate, weak, anecdotal. The low-traffic testing playbook distilled from this note lives in From product strategy to page copy, Testing playbook for low traffic; this note keeps the sources.
In short
How people read a page: 1. How people read on the web. Homepage, pricing and form evidence: 3. Landing page benchmarks, with caveats: 4.
Testing copy: 6. Testing copy properly; the playbook to follow is Testing playbook for low traffic.
Evidence grades
Strong: controlled experiment, large sample or replicated.
Moderate: well-run study with a smaller sample, or large correlational data.
Weak: single small study, vendor data, or self-reported answers.
Anecdotal: a single case.
CXL's research pages return 403 errors (they now also sit under Speero), so they were read from Wayback Machine copies. One source could not be fetched: the BBC's debunk of the goldfish claim. It is flagged [not verified] where it comes up.
1. How people read on the web (Nielsen Norman Group)
| Source | Finding | Method / sample | Quality |
|---|---|---|---|
| How Users Read on the Web, Jakob Nielsen, NN/g, 1997. https://www.nngroup.com/articles/how-users-read-on-the-web/ | "79 percent of our test users always scanned any new page they came across; only 16 percent read word-by-word." | Small qualitative usability studies (Study 2: 15 of 19 participants scanned first). | Moderate for direction; the percentage comes from a small 1997 sample. |
| Concise, SCANNABLE, and Objective: How to Write for the Web, John Morkes & Jakob Nielsen, 1997. https://www.nngroup.com/articles/concise-scannable-and-objective-how-to-write-for-the-web/ | Measured usability against a promotional control page: concise +58%, scannable +47%, objective (no marketese) +27%, all three combined +124%. Promotional language "imposes a cognitive burden": readers have to filter out hype to get at the facts. | 3 studies, 81 participants in total. The measurement study (Study 3) had 51 experienced users on one travel site about Nebraska. | Moderate. A real controlled comparison, but old, one site and 51 users. The direction has held up in later NN/g work; the exact 124% should not be quoted as a general law. |
| F-Shaped Pattern For Reading Web Content, Jakob Nielsen, 2006. https://www.nngroup.com/articles/f-shaped-pattern-reading-web-content-discovered/ | Eye tracking of 232 users across thousands of pages. "The first two paragraphs must state the most important information." Start subheads, paragraphs and bullets with information-carrying words. | Eye tracking. | Strong (large, and replicated below). |
| F-Shaped Pattern of Reading: Misunderstood, But Still Relevant, Kara Pernice, 2017 (reviewed 2026). https://www.nngroup.com/articles/f-shaped-pattern-reading-web-content/ | The F-pattern still holds on desktop and mobile, and mirrors in right-to-left languages. Other patterns: layer-cake (headings only), spotted, marking, bypassing, commitment (reads everything when motivated). The F-pattern is a failure mode that appears when a page gives no structure, not a goal to design for. | Eye tracking, roughly 45 to 47 participants per test. | Strong. |
| Text Scanning Patterns: Eyetracking Evidence, Kara Pernice, 2019. https://www.nngroup.com/articles/text-scanning-patterns-eyetracking/ | Patterns ranked from least to most effective for the reader: F, spotted, layer-cake, commitment. Layer-cake scanning of headings is "by far the most effective way" to scan a page; only reading almost every word does better. Recommends meaningful subheadings, bolded keywords, bullets, front-loading, no walls of text. | Eye tracking synthesis. | Strong. |
| How People Read Online: New and Old Findings, Kate Moran, 2020. https://www.nngroup.com/articles/how-people-read-online/ | 5 eye-tracking studies (2006 to 2019), 500+ participants, 750+ hours. "People rarely read online," and this has not changed in 23 years. How much people read depends on motivation, task, focus and the individual. Patterns were near-identical in a Beijing study (12 participants). | Eye tracking. | Strong. |
| How Little Do Users Read?, Jakob Nielsen, 2008. https://www.nngroup.com/articles/how-little-do-users-read/ | Users read at most 28% of the words on an average visit, and "20% is more likely". They read half the text only on pages of 111 words or fewer. Time on page is about 25 s plus 4.4 s per extra 100 words, so only about 18% of added words get read. | Reanalysis of instrumented-browser logs from Weinreich, Obendorf, Herder & Mayer (2005 data): 59,573 page views, 45,237 after cleaning. | Moderate. Naturalistic data, but a small user panel collected in 2005. The "20%" is an estimate from a model, not a direct count. |
| How Long Do Users Stay on Web Pages?, Jakob Nielsen, 2011. https://www.nngroup.com/articles/how-long-do-users-stay-on-web-pages/ | "Users often leave Web pages in 10 to 20 seconds," but pages with a clear value proposition hold attention far longer. 99% of pages show "negative aging": once a visitor gets past the first seconds, they tend to stay. You must "communicate your value proposition within 10 seconds". | Based on Liu et al. (Microsoft Research, 2010): 205,873 pages, more than 2 billion dwell times. | Strong for the dwell-time curve. The 10-second guidance is an inference drawn from it. |
| Scrolling and Attention, Therese Fessenden, 2018. https://www.nngroup.com/articles/scrolling-and-attention/ | 57% of viewing time above the fold (down from 80% in 2010), 74% in the first two screenfuls. More than 65% of above-fold time went to the top half of the viewport. There is still a sharp drop in attention at the fold. | Eye tracking, 120 participants, 130,000+ fixations, 1920×1080 screens. | Strong. |
| The Fold Manifesto, Amy Schade, 2015. https://www.nngroup.com/articles/page-fold-manifesto/ | Content 100 px above the fold got 102% more views than content 100 px below (57,453 fixations). Warns against "false floors": sparse hero sections that make users think the page has ended. | Eye tracking. | Strong. |
| Banner Blindness Revisited, Kara Pernice, 2018. https://www.nngroup.com/articles/banner-blindness-old-and-new-findings/ | Users ignore anything that looks like an ad: by location (top banner, right rail), by styling (boxed, bright, animated), and content next to ads gets ignored too. Implication: don't style your key value claim or CTA as a promo banner. | Eye tracking, 26 participants (part of 1997, 2007 and 2018 studies). | Moderate to strong (replicated across decades). |
| Inverted Pyramid, Amy Schade, 2018. https://www.nngroup.com/articles/inverted-pyramid/ | Put the conclusion first, then the supporting detail, then the background. Front-load headlines, first sentences and paragraph openings. | Guidance built on the reading studies above. | Moderate (derived guidance, not tested directly). |
| First 2 Words: A Signal for the Scanning Eye, Jakob Nielsen, 2009. https://www.nngroup.com/articles/first-2-words-a-signal-for-scanning/ | Users often see only about the first 2 words (about 11 characters) of list items and links. "Gift Cards" got 85% correct predictions of where it led; "Introducing…" got 0%. | 80 participants, 20 sites. | Moderate to strong. |
| Writing Hyperlinks, Marieke McCloskey, 2014. https://www.nngroup.com/articles/writing-links/ | Link text should be descriptive, unique and keyword-first. Avoid "click here" and "read more". | Guidance plus eye tracking. | Moderate. |
| Headlines / headings as pick-up lines, Hoa Loranger, 2015. https://www.nngroup.com/articles/headings-pickup-lines/ | Headings must make sense out of context. Be specific ("Data visualization helps you detect and prevent fraud faster"), avoid idioms, trim words, front-load keywords. | Guidance. | Moderate. |
2. Plain language and readability
| Source | Finding | Method / sample | Quality |
|---|---|---|---|
| Consequences of erudite vernacular utilized irrespective of necessity, Daniel M. Oppenheimer, Applied Cognitive Psychology 20:139 to 156, 2006. https://onlinelibrary.wiley.com/doi/10.1002/acp.1178 | 86.4% of 110 polled Stanford undergraduates admitted to making their vocabulary more complex to sound smart. Across experiments, more complex text meant the author was judged less intelligent, whatever the essay's quality. The effect runs through processing fluency. Hard-to-read fonts also lowered judged intelligence (Exp. 4). | 5 experiments with 27 to 85 Stanford undergraduates each (51 in the font experiment, Exp. 4). In Exp. 5 the effect reversed: when the difficulty had an obvious, irrelevant cause (a low-toner printout), readers rated the author more intelligent. | Moderate. Controlled experiments, but small student samples, and the outcome is a judgment of the author, not a conversion. |
| The Public Speaks: An Empirical Study of Legal Communication, Christopher R. Trudeau, Scribes Journal of Legal Writing 14 (2011 to 2012). Summary in the Michigan Bar Journal, Sept 2012: https://www.michbar.org/file/barjournal/article/documents/pdf4article2079.pdf | 376 respondents. They chose the plain version 80% of the time and it won all 11 paired comparisons. Preference rose with education: 77% (less than a bachelor's), 79% (bachelor's), 82% (master's/doctorate), 86% (law degree). 97% preferred "among other things" to "inter alia". 81% preferred "under" to "pursuant to" (69% of lawyers did). As sentences got more complex, the preference for plain versions grew: 86% where a sentence had several problems vs 75% with one. 44% of clients had stopped reading a document out of frustration. | 28-question survey with paired passages. | Moderate. A stated preference, not measured behavior, but it directly refutes "experts want jargon". |
| Plain Language Is for Everyone, Even Experts, Hoa Loranger, NN/g, 2017. https://www.nngroup.com/articles/plain-language-experts/ | Highly educated specialists (IT manager, professor, nurse practitioner with a doctorate) prefer short, scannable, jargon-light content just as much as anyone. | Qualitative usability sessions; sample size not stated. | Weak to moderate (qualitative). |
| Writing for lower-literacy users, Jakob Nielsen, 2005. https://www.nngroup.com/articles/writing-for-lower-literacy-users/ | Lower-literacy readers plow through text word by word and miss anything outside the main flow. In a pharma site rewrite, higher-literacy users also improved: success 68% to 93%, time 14.3 to 5.1 min. Recommends a grade 6 level for key pages and grade 8 elsewhere. | Usability case study. | Moderate (single case, large effect). |
| Legibility, Readability, and Comprehension, Jakob Nielsen, 2015. https://www.nngroup.com/articles/legibility-readability-comprehension/ | Aim for grade 8 for broad consumer audiences. For college-educated or B2B readers, grade 12, "several steps below" their education. In a B2B rewrite case, recall of product characteristics went from 33% to 65%. | Guidance plus a case study. | Moderate. |
| Highlights of the 2023 U.S. PIAAC Results, NCES, Dec 2024. https://nces.ed.gov/surveys/piaac/2023/national_results.asp | 28% of U.S. adults score at or below Level 1 in literacy, up 9 points since 2017. | National probability sample (OECD PIAAC). | Strong. |
| Use clear language, GOV.UK content guidance. https://guidance.publishing.service.gov.uk/writing-to-gov-uk-standards/writing-guidelines/clear-language/ | "Plain English is mandatory for all of GOV.UK." Cites the legal-language research above: 80% prefer clear English, and preference grows with education and specialist knowledge. Guideline: split sentences over 25 words. | Policy plus the research it cites. | Policy; its evidence is Trudeau. (The often-quoted "reading age of 9" line was not on this page when checked.) |
| Writing for Dollars, Writing to Please, Joseph Kimble, Carolina Academic Press (2012; 2nd ed.). https://cap-press.com/books/isbn/9781611631913/Writing-for-Dollars-Writing-to-Please | Summarizes 50 studies: readers prefer plain language, understand it faster, comply more, and organizations save money. | Literature compilation. | Moderate (the review is well regarded; each study varies in quality). |
3. Homepage, landing page, pricing and form evidence
| Source | Finding | Method / sample | Quality |
|---|---|---|---|
| Homepage Design: 5 Fundamental Principles, Huei-Hsin Wang, NN/g, 2024. https://www.nngroup.com/articles/homepage-design-principles/ | Communicate who you are and what you do (a descriptive tagline plus the unique value), show examples of real content rather than generic categories, use descriptive link labels with high "information scent", keep it simple. Also cites Google data: bounce probability rises 32% as load time goes from 1 s to 3 s. | Guidance synthesis. | Moderate. |
| Top 10 Guidelines for Homepage Usability / Tagline Blues, NN/g. https://www.nngroup.com/articles/top-ten-guidelines-for-homepage-usability/ , https://www.nngroup.com/articles/tagline-blues-whats-the-site-about/ | Start with a tagline that says what the company does. Test: list your tagline and three competitors' taglines without names, and see whether outsiders can tell which company does what. "Welcome" copy carries no information. | Guidance. | Moderate. |
| B2B Usability, Jakob Nielsen, 2006. https://www.nngroup.com/articles/b2b-usability/ | Out of 28 information types, users ranked pricing first, scoring 29% higher than the runner-up (availability). Most B2B sites hid it. B2B task success was 58% vs 66% on consumer sites. | 79 participants: 12 focus groups, 55 usability sessions, 7 field studies, 179 sites. | Moderate to strong (old but large). |
| What B2B Designers Can Learn from B2C About Building Trust, Therese Fessenden, 2019. https://www.nngroup.com/articles/b2b-trust-from-b2c/ | Hidden pricing "makes it seem unaffordable" and costs trust. Show starting prices or ranges and comparison tables. Explain why you ask for personal data. Users notice testimonial authors' titles and affiliations. | Qualitative research plus guidance. | Moderate. |
| "Powered By AI" Is Not a Value Proposition, Pavel Samsonov, NN/g, 2025. https://www.nngroup.com/articles/powered-by-ai-is-not-a-value-proposition/ | Lead with the user's outcome, not the technology. Cites research that mentioning AI can lower purchase intent. That research is Cicek, Gursoy & Lu, Journal of Hospitality Marketing & Management, 2024, "Adverse impacts of revealing the presence of 'Artificial Intelligence (AI)' technology in product and service descriptions on purchase intentions", doi:10.1080/19368623.2024.2368040, 34(1), 1 to 23, online 19 June 2024. Experimental surveys with more than 1,000 US adults across eight product and service categories found lower purchase intent, more so for high-risk products (https://www.eurekalert.org/news-releases/1052886, read 29 October 2026; study details in the evidence digest). | Guidance plus online scenario experiments. | Moderate for the principle. The AI study measures stated intent, not real purchases. |
| The Presentation of Your Value Proposition Matters, Ben Labay, CXL Institute, 2016. Original URL https://cxl.com/research-study/value-proposition-study/ (read via the Wayback Machine) | Tested 4 value-prop formats on a fintech SaaS landing page (Personal Capital): 3 bullets, 3 bullets with descriptions, 5 bullets, short paragraph only. The paragraph-only version was noticed slowest and fewest people understood the site's purpose (χ² p = 0.002). Some respondents guessed "electronics" or "office supplies". Versions with more concrete bullets gave better recall of services and advantages. Plain bullets were somewhat preferred. Likert "confidence" scores showed no difference while open-ended comprehension did, so ask people to explain the page back to you instead of asking how confident they feel. | Eye tracking plus survey, about 215 participants across 4 variants. | Moderate (panel participants, one site). |
| Which Types of Social Proof Work Best?, Ben Labay, CXL Institute, 2016. https://cxl.com/research-study/social-proof/ (via the Wayback Machine) | 8 social-proof types vs a control on a hosting page. None drew attention faster (average 8.3 s to first fixation). Testimonials held attention longest. Recall was significantly better for high-profile client logos (p = 0.009 vs low-profile logos) and testimonials with photos (p = 0.0035 vs without). Social-follower counts and low-profile logos were not memorable. | Eye tracking plus recall survey, about 200 participants. | Moderate (recall, not conversion). |
| Pricing page study: highlighted plan, Ben Labay, CXL Institute. https://cxl.com/research-study/pricing-page-study-highlight/ (via the Wayback Machine) | Highlighting a plan drew attention to it faster when plans ran cheap-to-expensive (p = 0.017). More participants chose the PRO plan when plans ran expensive-first and when it was highlighted. The authors' own limit: participants were not real buyers. | Eye tracking plus a hypothetical choice survey. | Weak to moderate. |
| How Does Product Copy Format Affect User Engagement?, CXL Institute, 2016. https://cxl.com/research-study/product-copy-engagement/ (via the Wayback Machine) | For spec-driven products, bullets held attention longer than paragraphs (hard drive p = 0.0199). For "design" products the format mattered less. | Eye tracking, at least 42 people per cell. | Moderate. |
| How people read short articles, Madeleine Sidoff, CXL Institute, 2018. https://cxl.com/research-study/people-read-short-articles-original-research/ (via the Wayback Machine) | On a 300-word article: 97% read the title, 98% glanced at the subtitle, 60% "finished" but by skimming, 91% read the image caption, and the banner ad got under 1.5 s of attention. | Eye tracking, 62 younger plus 33 older participants. | Moderate. |
| Cart abandonment statistics, Baymard Institute, updated Sept 2025. https://baymard.com/lists/cart-abandonment-rate | Average documented abandonment is 70.22% (50 studies, 2006 to 2025). Reasons, excluding people who were "just browsing": extra costs 40%, slow delivery 20%, didn't trust the site with card details 19%, forced account creation 18%, too long or complicated 17%, couldn't see the total cost upfront 12%, returns policy 13%. Better checkout design could lift conversion 35.26% on large sites (a Baymard estimate). | Meta-list plus Baymard's own survey. | Moderate to strong for ecommerce. Applies to SaaS by analogy: price transparency and trust copy. |
| Mobile Form Usability: Never Use Inline Labels, Baymard, 2013. https://baymard.com/blog/mobile-forms-avoid-inline-labels | Placeholder-as-label caused severe problems in every Baymard test, and users deleted their input just to see the label again. Put labels above fields. | Moderated usability testing. | Strong (consistent across their tests). |
| Structuring descriptions by highlights, Baymard. https://baymard.com/blog/structure-descriptions-by-highlights | Long descriptions chunked into highlights (each with an image) engage users, yet 78% of sites don't do it. 50% of sites have spec sheets that are hard to scan. | Usability testing plus benchmark. | Moderate to strong. |
| Inline Validation in Web Forms, Luke Wroblewski with Etre, A List Apart, 2009. https://alistapart.com/article/inline-validation-in-web-forms/ | The best of five inline variants, compared with validating only on submit: +22% success, 22% fewer errors, +31% satisfaction, 42% faster, 47% fewer fixations. Helped on hard fields (username, password) but confused people on easy ones. Every inline version checked username and password while the user typed, after a short delay, and the article recommends that for fields with strict rules. On the easy fields, validating after the user left a field was 7 to 10 seconds faster than the "while" and "before and while" variants. | 22 participants, 6 form versions. | Moderate (small). This is often misattributed to Baymard. |
4. Landing page conversion research (read the caveats)
| Source | Finding | Method / sample | Quality |
|---|---|---|---|
| Conversion Benchmark Report, Unbounce, 2024. https://unbounce.com/conversion-benchmark-report/ and SaaS page https://unbounce.com/conversion-benchmark-report/saas-conversion-rate/ | More than 41,000 landing pages, 464M visitors, 57M conversions. All-industry median 6.6%, SaaS median 3.8%. Pages at a grade 5 to 7 level convert at 11.1%, 56% above grade 8 to 9 and more than double professional-level copy (5.3%). For SaaS the figures are 12.9% vs 2.1%. Word count correlates negatively with conversion (−18.6%), and so do three-syllable words (−24.3%). SaaS pages that did best had 250 to 725 words. | Machine-learning analysis of Unbounce customer pages. | Moderate, correlational only. Unbounce itself notes it measures conversion events, not their value, and that effects vary by industry. Simple offers such as free ebooks likely use both simple copy and easy conversions, which confounds the result. |
| What works in e-commerce: a meta-analysis of 6700 online experiments, Will Browne & Mike Swarbrick Jones, Qubit, 2017 (audited by PwC). https://gwern.net/doc/economics/advertising/2017-browne.pdf | Average revenue-per-visitor lift by treatment: scarcity +2.9%, social proof +2.3%, urgency +1.5%, abandonment recovery +1.1%, recommendations +0.4%. Colour 0.0%, buttons −0.2%, calls to action (reworded to be more suggestive, e.g. "contact us" to "get a quote") −0.3%, landing page −0.3%. 90% of experiments had an effect under 1.2% either way. Detecting a 1% conversion lift needs about 120,000 converters per variant. | Bayesian meta-analysis of roughly 2,600 categorized tests (from 6,700 in total), retail and travel. | Strong, and unusually free of publication bias because it covers all tests, not just published winners. |
| Analysis of 115 A/B Tests, Georgi Georgiev, Analytics-Toolkit, 2018 (updated 2022). https://blog.analytics-toolkit.com/2018/analysis-of-115-a-b-tests-average-lift-statistical-power/ | Median lift 3.92% across the 85 tests left after pruning. Only 27% of the 115 tests were statistically significant positive results. About 70% were underpowered, having been designed to detect lifts of 20 to 40% that rarely happen. A later analysis of 1,001 tests on the author's platform (https://blog.analytics-toolkit.com/2022/what-can-be-learned-from-1001-a-b-tests/, 2022, read 29 October 2026) found a median lift of 0.08%, a mean of 2.08%, and 33.5% significant positive outcomes. | Meta-analysis. | Moderate. |
| p-Hacking and False Discovery in A/B Testing, Berman, Pekelis, Scott & Van den Bulte, SSRN 2018 (later in Information Systems Research). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791 | 2,101 Optimizely experiments. About 73% of experimenters stopped a test as soon as it reached 90% confidence. About 73 to 77% (by their estimate) of effects are truly null. Optional stopping pushes the false discovery rate from 33% to 40%. | Regression discontinuity. | Strong. |
| Optimizely "127,000 experiments" report, as critiqued by Ron Kohavi on LinkedIn (2023). https://www.linkedin.com/posts/ronnyk_lessons-learned-from-running-127000-experiments-activity-7143376795940106240-faUh | Optimizely reported a 12% win rate. Kohavi argues its default α = 0.1 means roughly 38% of the "significant" wins are false positives, and a true success rate near 9%. | Commentary on vendor data. | Moderate. |
| Wynter message testing. https://wynter.com/post/message-testing and https://wynter.com/research-playbook (read 29 October 2026) | Panel of 86,000+ identity-verified B2B professionals, results in 12 to 48 h. Open-ended feedback on clarity, relevance, value, differentiation. Wynter puts qualitative saturation at 12 to 13 responses, cites a review of 23 articles in which 9 to 17 participants sufficed for homogeneous groups, and sets 15 respondents as its default test size. Common failures: vague value props ("drive more revenue"), generic wording, weak differentiation. | Vendor method description. | A method, not evidence of lift; vendor source. |
| Optimizely / VWO published case studies (general caveat) | Published wins are selected and survive: vendors publish winners, losers go unreported, and many "cases" are underpowered or were stopped early. Compare the Qubit, Georgiev and Berman data above: real average effects are about 0 to 4%, far below the lifts that published case studies report. Kohavi et al. (below) invoke Twyman's law: "any figure that looks interesting or different is usually wrong." | n/a | Treat any single case study as anecdotal. |
5. Experimentation culture: most ideas fail
| Source | Finding | Quality |
|---|---|---|
| The Surprising Power of Online Experiments, Ron Kohavi & Stefan Thomke, HBR, Sept to Oct 2017. https://hbr.org/2017/09/the-surprising-power-of-online-experiments | The Bing headline test: in 2012 an idea to change how ad headlines display sat in the backlog for more than 6 months. Per summaries of Kohavi's book, it lengthened the title line by merging it with the first line of ad text. Once tested, it raised revenue 12%, "more than $100 million in the United States alone" per year, without hurting user metrics. It was "the best revenue-generating idea in Bing's history". At Google and Bing, only about 10% to 20% of experiments generate positive results. At Microsoft overall, one-third are positive, one-third neutral, one-third negative. About 80% of Bing changes run as controlled experiments. Most engagement changes move metrics by less than 1%. | Strong (practitioner data from the world's largest experimentation programs). |
| A/B Testing Intuition Busters, Kohavi, Deng & Vermeer, KDD 2022. https://doi.org/10.1145/3534678.3539160 (author PDF https://bit.ly/ABTestingIntuitionBusters) | Success rates: Microsoft 33%, Bing 15%, Booking.com, Google Ads, Netflix about 10%, Airbnb search 8%. With 80% power and p < 0.05, the false positive risk runs from 5.9% (Microsoft) to 26.4% (Airbnb). At 20% power it rises above 50% for success rates of about 10% or less. A/B tests need "at least thousands of active users, preferably tens of thousands." Sample size n ≈ 16σ²/δ² per variant. Their example: a 3.7% baseline and a 10% relative lift need 41,642 users per variant. The case they examine ran on about 80 users per variant, which gave about 3% power. | Strong. |
| Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO, Kohavi, Henne & Sommerfield, KDD 2007. https://exp-platform.com/Documents/GuideControlledExperiments.pdf (read 29 October 2026) | Popularized HiPPO (Highest Paid Person's Opinion), which the paper traces to Avinash Kaushik's "HiPO". Primacy and newness effects: returning users either resist a change or poke at it out of novelty, so some tests need to run for several weeks. Analyze new users separately. | Strong. |
| Trustworthy Online Controlled Experiments, Kohavi, Tang & Xu, Cambridge University Press, 2020. https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B518F59 (read 29 October 2026) | The standard reference: OEC, sample ratio mismatch, novelty effects, Twyman's law. (The book's content was confirmed through its KDD papers; the book itself was not read in this pass.) [not verified] | Strong. |
| Democratizing online controlled experiments at Booking.com, Kaufman, Pitchforth & Vermeer, 2017. https://arxiv.org/abs/1710.08217 | More than 10 years of evidence-based development. Anyone can own an experiment end to end. A central repository keeps failures as well as wins. | Moderate (descriptive). |
6. Testing copy properly
Peeking: Evan Miller, How Not To Run an A/B Test, 2010. https://www.evanmiller.org/how-not-to-run-an-ab-test.html. Checking repeatedly and stopping at the first p < 0.05 inflated a nominal 5% false positive rate to 26.1% in his example. Fix the sample size in advance (n = 16σ²/δ²), or use sequential or Bayesian designs built for optional stopping. Calculator: https://www.evanmiller.org/ab-testing/sample-size.html. Its defaults (a 20% baseline, a 5-point absolute lift, 80% power, 5% significance) need 1,030 per variant (https://www.evanmiller.org/ab-testing/defaults.js, read 29 October 2026). Strong.
What this means for a small SaaS (worked example with the formula above): with a 3% visitor-to-signup rate, detecting a 20% relative lift (3.0% to 3.6%) needs about 12,900 visitors per variant, roughly 26,000 in total. A 10% lift needs about 52,000 per variant. At 1,000 visitors a week, even the 20% test takes about half a year. Qubit's data says real copy effects are usually under 1 to 2%. Conclusion: below roughly tens of thousands of visitors per month, A/B testing homepage copy for conversion is mostly noise.
Novelty and primacy effects: Kohavi 2007 (above). Run full weeks and check new users separately. Strong.
Qualitative sample sizes: Jakob Nielsen, Why You Only Need to Test with 5 Users, 2000. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/. About 85% of usability problems found with 5 users (Nielsen and Landauer model, L ≈ 31%). Better: 3 rounds of 5 with fixes in between, 3 to 4 per distinct user group, and 20 or more for quantitative measures. Strong for finding problems; not for measuring preference.
Five-second tests: Lyssna guide. https://www.lyssna.com/guides/five-second-testing/. Show the page for 5 s, then ask "What was this page about?", "What stood out?", "What was unclear?". 5 to 10 participants surface the major patterns. Limits: it can't tell you why, recall is biased, and there's no task context. Method guidance (vendor).
First-click testing: Jeff Sauro, MeasuringU, 2011. https://measuringu.com/first-click/. Cites Bob Bailey's finding that when the first click is correct, 87% eventually succeed, against 46% when it is wrong. Moderate (secondary citation of Bailey's work).
Message testing panels: Wynter (above). Test comprehension with open-ended questions; the CXL value-prop study shows Likert confidence scores can miss real comprehension failures.
Other qualitative sources that need no traffic: customer interviews, sales-call recordings, support tickets, reviews (yours and competitors') for real customer vocabulary, the tagline blind test (NN/g), preference tests (weak: people choose differently from how they behave, as the CXL pricing study says itself).
7. SEO and AI answer engines
| Source | Finding | Quality |
|---|---|---|
| Creating helpful, reliable, people-first content, Google Search Central, updated 5 Oct 2026 (read 29 October 2026). https://developers.google.com/search/docs/fundamentals/creating-helpful-content | People-first content; self-assessment questions (original information, descriptive non-sensational headings, first-hand expertise). E-E-A-T is not itself a ranking factor; trust matters most. "Who, How, Why": bylines, disclosing automation. Warning signs: writing to a word count ("No, we don't" have a preferred one), mass-produced content, changing dates without real updates. | Authoritative for Google's intent. How the ranking systems work stays opaque. |
| AI features and your website, Google, updated Dec 10, 2025. https://developers.google.com/search/docs/appearance/ai-features | "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." You don't need to create "new machine readable files, AI text files, or markup", nor special schema.org data. Google claims clicks from AI Overview pages are "higher quality" (longer time on site), without published data. | Authoritative for policy. The click-quality claim is unverified vendor data. |
| Google users are less likely to click on links when an AI summary appears, Pew Research Center, July 22, 2025. https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/ | 900 U.S. adults, 68,879 searches (12,593 with an AI summary). Result-link clicks were 8% with a summary vs 15% without. Only 1% clicked links inside the summary. Sessions ended 26% vs 16% of the time. | Strong (a real behavioral panel). |
| Update: AI Overviews Reduce Clicks by 58%, Ryan Law, Ahrefs, 4 Feb 2026. https://ahrefs.com/blog/ai-overviews-reduce-clicks-update/ (read 29 October 2026) | 300,000 keywords, December 2023 vs December 2025. Position-1 CTR was about 58% lower where an AI Overview appeared (0.016 vs 0.037 forecast). Ahrefs' April 2025 run on earlier data found 34.5%. | Moderate (vendor, correlational, before and after). |
| GEO: Generative Engine Optimization, Aggarwal et al., KDD 2024. https://arxiv.org/abs/2311.09735 | 10,000-query benchmark on a simulated generative engine. Adding quotations (+41% on position-adjusted word count; 27.2 vs 19.3), statistics (25.2) and cited sources (24.6), plus improving fluency, raised a source's visibility in answers. Keyword stuffing did worse than no optimization (17.7). Visibility of sites ranked 5th rose 115% with "Cite Sources", while top-ranked sites lost some. Results varied by domain. | Weak to moderate: one paper, a lab set-up with 2023-era models, partly checked on Perplexity. Treat as a hint, not a playbook. |
| How AI is changing search behaviors, Moran, Rosala & Brown, NN/g, Aug 2025. https://www.nngroup.com/articles/ai-changing-search-behaviors/ | 9 participants. People still start with Google and use AI chat as a complement. AI Overviews reduce clicks. | Weak (qualitative, n = 9). |
| How People Read Online report (2nd ed.), NN/g. https://www.nngroup.com/reports/how-people-read-web-eyetracking-evidence/ | Readers glance at AI summaries. Bolded keywords and answers stated up front still matter. NN/g says the same principles that help scanning also help AI comprehension. | Moderate (NN/g's assertion, not tested). |
Bottom line on writing for answer engines: the evidence is thin and moves quickly. What the credible sources agree on: write clear, specific pages that answer the question up front, with concrete facts, figures and attributed sources. Don't keyword-stuff. Expect fewer clicks from informational queries. There is no verified special format.
8. Accessibility and copy (W3C WCAG)
SC 2.4.4 Link Purpose (In Context), Level A: each link's purpose must be clear from its text or its programmatic context. "Click here" fails. Put the ambiguous words at the end of a descriptive sentence. SC 2.4.9 (AAA): purpose clear from the link text alone. https://www.w3.org/WAI/WCAG22/Understanding/link-purpose-in-context.html
SC 2.4.6 Headings and Labels, Level AA: "Headings and labels describe topic or purpose." Headings can be short. It is separate from 1.3.1 (correct markup) and 3.3.2 (form labels). https://www.w3.org/WAI/WCAG22/Understanding/headings-and-labels.html
SC 3.1.5 Reading Level, Level AAA: if text needs more than a lower secondary reading level (7 to 9 years of schooling), provide a simpler version or supplement (a summary, visuals, audio). https://www.w3.org/WAI/WCAG22/Understanding/reading-level.html
SC 1.1.1 Non-text Content / alt decision tree: use empty alt for decorative images or images whose text is already nearby. For a linked image, describe the destination or action. For an informative image, give its meaning briefly. For text in an image, repeat the text. For a complex image, put the data on the page. https://www.w3.org/WAI/tutorials/images/decision-tree/
Evidence-backed rules for website copy
Source synthesis from this note. The organization's rules are in Principles, which wins where the two differ.
Say what it is, who it's for and why it matters within the first screen, in about 10 seconds. Sources: Nielsen 2011 dwell-time analysis of 2B dwell times (visitors often leave within 10 to 20 seconds); NN/g homepage principles 2024; Fessenden 2018 (57% of viewing time above the fold, 74% in the first two screens). Strong for the first screen; moderate for the 10-second figure, Nielsen's inference from the dwell-time curve.
Write for scanning. Expect readers to get through about 20% of a page's words, at most 28%, and less as pages get longer. Sources: Nielsen 2008 (a model of browser logs from 25 users); Moran 2020 (500+ participants, for scanning itself). Strong for scanning; moderate for the percentage.
Front-load everything. The first 2 paragraphs carry the message; headings, bullets and links start with the information-carrying word (users see about 2 words / 11 characters). Sources: Nielsen 2006 (232 users); Nielsen 2009 (80 users). Strong.
Use descriptive headings that make sense out of context, so layer-cake scanners get the whole story from headings alone. Sources: Pernice 2019; Loranger 2015; WCAG 2.4.6. Strong.
Cut marketese; state facts and specifics. Objective copy scored +27% in measured usability, and all three changes combined +124%. Sources: Morkes & Nielsen 1997; Loranger 2015 (specific benefit headlines). Moderate (the direction is solid; the size is from one old study).
Be concise. Concise copy alone scored +58%. Unbounce: word count correlates negatively with conversion, and the best SaaS pages had 250 to 725 words. Sources: Morkes & Nielsen 1997; Unbounce 2024. Moderate (Unbounce is correlational).
Write plainly even for experts. Preference for plain versions rose with education, from 77% to 86% among lawyers. Needless complexity makes you look less intelligent. Sources: Trudeau 2012 (376 respondents); Oppenheimer 2006; Loranger 2017. Moderate.
Target grade 7 to 9 for broad audiences and no higher than about grade 12 for specialists. Grade 5 to 7 pages converted best in Unbounce's data. 28% of U.S. adults are at or below literacy Level 1. Sources: Nielsen 2015; Unbounce 2024 (correlational); NCES PIAAC 2023. Moderate.
Use concrete bullets for the value proposition, not a vague paragraph. The paragraph-only variant was noticed slowest and misunderstood most. Source: CXL value-prop study 2016. Moderate.
Name the outcome, not the technology ("Powered by AI" is not a value proposition). Sources: Samsonov, NN/g 2025; Cicek et al. 2024 (stated intent). Moderate.
Show pricing, or at least starting prices and ranges. B2B users rank pricing first (29% above the next item). Hidden costs and unclear totals drive abandonment (40% and 12%). Sources: Nielsen 2006; Fessenden 2019; Baymard 2025. Moderate to strong.
Social proof: use high-profile logos and testimonials with real photos, names and titles. Skip follower counts and unknown logos. Social proof was among the few treatments with a reliably positive revenue effect (+2.3%). Sources: CXL 2016 (recall); Qubit 2017 (6,700 experiments); Fessenden 2019. Moderate (recall plus ecommerce data).
Don't let key messages look like ads, such as boxed, bright or animated promos in banner or right-rail positions. Source: Pernice 2018 banner blindness. Moderate to strong.
Put important content above the fold, and avoid "false floors" (sparse hero sections that stop scrolling). Sources: Schade 2015 (+102% views just above the fold); Fessenden 2018. Strong.
Use descriptive, unique link and button text; never "click here" or "read more". Sources: WCAG 2.4.4 (Level A); McCloskey 2014; Nielsen 2009. Strong (standard plus usability evidence).
Keep form labels visible above fields and write specific error text. Validate hard fields, such as username and password, inline while the user types, after a short delay. Sources: Baymard 2013; Wroblewski 2009 (best inline variant: +22% success, 22 participants). Moderate to strong.
Don't expect micro-copy tweaks (CTA wording, button colour) to move revenue. Average effects were about 0%, and 90% of experiments landed within ±1.2%. Work on the offer, clarity, proof and price instead. Sources: Qubit 2017; Kohavi & Thomke 2017. Strong.
For search and AI answers, write people-first pages that answer up front, with specific facts, figures and attributed sources. Never keyword-stuff. Sources: Google 2025; GEO 2024 (stuffing scored below baseline); Pew 2025. Moderate for the principle, weak for any AI-specific tactic.
Assume your idea is wrong until tested: only 10 to 33% of ideas win, even at Microsoft, Google and Booking. Hold your copy opinions loosely, HiPPO included. Sources: Kohavi & Thomke 2017; Kohavi et al. 2022. Strong.
Testing playbook for a small SaaS with little traffic
The low-traffic testing playbook is Testing playbook for low traffic in From product strategy to page copy; the evidence behind its steps is in 6. Testing copy properly.
Commonly cited "stats" that are myths, misused or unsourced
The verdicts on commonly cited web statistics (the eight-second attention span, the three-click rule, "users read 20 percent of words", button-colour lifts and the rest) are in Myths, Web reading and testing, each with its source.