6 Executing Demand Experiments

Who answered, who did not, and what they thought you asked

Designing a good demand experiment is necessary—but not sufficient.

Between a well-designed survey and usable demand evidence lies execution. This is where plans encounter reality, and where many otherwise sound demand experiments quietly fail.

Execution introduces risks that design alone cannot eliminate. Who actually sees the survey, who chooses to respond, how questions are interpreted in context, and how carefully responses are given all shape the evidence that results.

These risks are rarely obvious. Bad execution does not always produce noisy or chaotic data. More often, it produces data that looks clean, coherent, and convincing—while pointing in the wrong direction.

This chapter focuses on the most common execution failures in demand experiments. The goal is not to eliminate these risks entirely. It is to recognize them early, interpret evidence cautiously, and know when learning must continue before decisions are made.

Many of the execution failures discussed in this chapter can be reduced—but not eliminated—through careful framing and validation.

Several toolkits are designed to support that work, especially:

This chapter explains why those tools matter.

Sampling: Who Actually Had a Chance to Answer?

Demand experiments do not happen in theory. They happen in the world.

However carefully a survey is designed, the evidence it produces depends on a basic fact: who was actually given a chance to respond.

This question comes before who chose to respond. It comes before analysis. And it often determines what the evidence can and cannot say.

The Population Is Not the Sample

Entrepreneurs usually begin with a target population in mind.

They might say they are studying:

potential first-time buyers,
people who experience a particular problem,
customers who care about a specific attribute,
or a defined segment of a broader market.

This population defines who could plausibly be a customer.

But the population is not the sample.

The sample consists of the people who were actually reached—those who were exposed to the survey and had a real opportunity to respond. This group is sometimes called the sampling frame, even if no formal sampling plan was used.

The distinction matters.

Eligibility Is Not Representativeness

A common mistake is to assume that if all respondents belong to the target population, the sample must be appropriate.

This is not true.

A sample can consist entirely of legitimate customers and still badly misrepresent market demand.

Why? Because market demand depends not just on who is included, but on which parts of the population are represented.

If a sample draws only from a narrow, distinctive subset of the population, the resulting demand evidence will reflect that subset’s preferences—not the population as a whole.

This is especially likely when the sampling frame is defined by:

shared institutions or affiliations,
strong cultural or social coordination,
unusually high engagement or commitment,
or easy accessibility rather than relevance.

In these cases, the sample may be inside the population while failing to span it.

How Convenience Sampling Shapes the Frame

Most early demand experiments rely on convenience sampling.

Surveys are sent to people who are:

easy to reach,
already connected to the entrepreneur,
active in a particular community,
or readily available through existing channels.

Convenience sampling is not inherently wrong. It is often unavoidable.

The risk is not convenience itself, but unexamined convenience.

Convenience sampling determines which parts of the population are given a voice. When access is limited to a narrow or highly coordinated subset, other regions of the population are excluded entirely—before anyone chooses whether to respond.

This exclusion happens quietly. No data is collected from those outside the frame, and their absence can be mistaken for indifference rather than inaccessibility.

Why Composition Matters for Demand

Demand is an aggregate object.

Entrepreneurs are rarely trying to learn how one individual behaves. They are trying to assess whether an initiative can support a viable business at the level of the target market.

That requires understanding the distribution of demand across the population:

how preferences vary,
how willingness to pay differs,
how usage intensity changes across customers.

A sample that captures only one region of this variation—even if that region is valid—can lead to systematic overestimation or underestimation of demand.

The danger is not that the data are wrong.
The danger is that they are conditionally right and interpreted as generally right.

The First Sampling Question to Ask

Before looking at response rates or survey results, ask a simpler question:

Who was actually given a chance to respond?

Answering this requires more than listing where the survey was posted or whom it was sent to. It requires reflecting on how access was determined—and which parts of the population that access favored.

Good sampling discipline begins here:

not with statistics,
not with correction,
but with awareness of coverage.

A Protective Rule of Interpretation

When sampling frames are narrow, conclusions should be narrow as well.

Evidence gathered from a distinctive subset should be interpreted as describing that subset, not the entire market you hope to serve.

This is not a reason to avoid early demand learning. It is a reason to be precise about what the evidence actually supports.

Understanding who had a chance to respond raises an obvious question about how they came to have it.

If you find that your sampling frame is narrow or difficult to justify, pause before proceeding.

This is often a sign that the demand learning problem needs to be reframed. Returning to the problem framing toolkit can help clarify which population truly matters for the decision at hand.

Getting People to Answer

The last section asked who had a chance to respond. The next one asks who took it. Between those two questions sits work that nobody assigns you, and that this book has so far left out.

Your Sampling Frame Is Something You Built

It is easy to read the previous section as describing a frame you can go and inspect: where the survey was posted, who it was sent to, who happened to see it. That reading is too passive. Nobody hands you a frame. You build one, person by person, and what you end up with is the residue of where you looked, who agreed to help, and where you eventually stopped trying.

This is why two people studying the same market can produce evidence that cannot be compared. One posted a link in a general forum and took whoever wandered in. The other spent a week persuading a trade association to forward it to their members. Both have forty responses. Only one of them has forty responses from the people the decision is about.

Response Rate Is a Leading Indicator of Acquisition Cost

Most people treat low response as an obstacle standing between them and the real work.

Do the arithmetic before you accept that. Suppose it took two hundred approaches to collect forty answers to a survey that costs a respondent three minutes and no money whatsoever. You will eventually need people to do something considerably harder than that. They will have to hand you money.

Acquisition runs through the same channel you just used, against much higher friction. What it costs shows up in the profit equation directly: in variable cost if you pay for each customer acquired, in fixed cost if you build a channel and staff it. A channel that cannot deliver free survey responses cheaply is unlikely to deliver paying customers cheaply.

So the recruiting you are finding tedious is not standing between you and your measurement of \(\mathsf{q}\). It is your first measurement of \(\mathsf{c}\) and \(\mathsf{f}\).

Take that literally rather than as consolation. Record how many approaches each response cost you, and record it by channel. Those numbers are evidence, and they are close to the only evidence about acquisition cost available to you before you have something to sell.

Track the Funnel, Not the Count

The number that matters is not how many responses you have. It is where you lost everyone else.

Count people at four points: approached, started, passed the screener, finished. Each gap says something different, and each one is a finding rather than a failure.

Lost before starting. You cannot reach this group through this channel. That is an access finding, and it is often worth more than what the survey would have told you.

Screened out in volume. The segment is rarer in this population than you assumed. Notice what you now have: an empirical estimate of how common your customer actually is, which is precisely the number market sizing asserts and never measures.

Started, then abandoned at the price questions. Something is unclear, in the instrument or in the product description, or the questions ask more than a stranger will give. This is fixable, and worth fixing before you gather anything further.

Finished, with willingness to pay clustered near zero. You have your answer. It arrived cheaply and early, which was the entire point of asking.

Everyone who failed to finish told you something. A raw count throws all of it away.

Where the People Already Are

The common failure is to broadcast and hope the right people identify themselves.

Broadcasting works badly for a structural reason. Your target segment is by construction a minority of any general audience, so a general channel delivers mostly people you will screen out, at full effort per person. You exhaust yourself assembling a sample you then discard.

Somebody has already done the aggregating for you. Your people gather somewhere: a club, a trade association, a subreddit, a Discord, a conference, a waiting room, a shop that serves them, a queue they stand in. Find the gathering rather than the individuals, then ask whoever runs it.

Borrowed credibility is the largest single lever available to you. A link from a stranger converts close to nothing. The same link forwarded by a moderator, an organizer, an association’s staff, or a shop owner converts many times better, because the trust already exists and you are borrowing it instead of building it. A week spent earning one introduction usually beats a week spent posting.

Ask in person, deliver digitally. A face-to-face request followed by I’ll send you the link outperforms a purely digital approach by a wide margin, because the commitment becomes social rather than transactional. This is the move most people avoid and the one that works.

Be specific about why you are asking. I’m a student working on a project requests charity. I’m trying to work out whether this particular thing, for people like you, is worth building, and I’ll send you what I find offers an exchange.

Length is the biggest thing you control. Every question costs completions, and the price questions you actually need are already demanding. Cut to the screener, the willingness-to-pay block, and only those characteristics you will genuinely segment on. Everything else is a tax paid in lost responses.

What to Offer

Incentives are where a demand survey differs from every other kind of survey.

An ordinary survey can pay its respondents without much harm. This one cannot, for a specific reason. You are asking people what they would pay. Paying them to answer makes money salient at exactly the wrong moment, and it recruits people who answer surveys for money, a group defined by a trait that has nothing to do with wanting your product.

Offering the product itself is worse, which is counterintuitive enough to say plainly. Cash selects for people who want cash, roughly unrelated to your question. Your product selects for people who want your product, which is the very thing you are trying to measure. You would be recruiting on your own dependent variable. And giving it away announces that its price is zero, moments before you ask what someone would pay for it. A discount carries the same defect in milder form, and adds one of its own by naming a price before you ask for theirs.

What remains is workable. Early access at the eventual price concedes nothing on price and selects only mildly on interest. An adjacent item, raffled — an accessory, or something else from the same category — is the best incentive most ventures can offer, because it biases toward category interest, which overlaps your screener rather than your outcome. The findings themselves, sent back to everyone who answered, remain the cleanest offer available and cost you nothing but the sending.

One further possibility belongs outside this list, because it is not an incentive at all.

If you are tempted to give people the product in exchange for answering, consider selling it to them instead. A preorder, a deposit, or a pledge at a stated price is not a stated preference. It is a purchase, and it is stronger evidence than any survey can produce. It does not replace the survey, which estimates a whole curve where a preorder gives you a single point. It validates one, at a real price, with real money, which is exactly the one-point anchor test the estimated curve should be held to.

When You Cannot Reach Them At All

Sometimes the honest outcome of recruitment is that it does not work.

You tried several channels, spent real effort, and could not assemble enough of the right people to say anything. The temptation then is to widen the definition until responses arrive, which converts a clean negative result into a misleading positive one.

Record what happened instead. A group you cannot reach is not a market you can serve. Whatever they might have paid, you have no way to put an offer in front of them, and no way to do it repeatedly at a cost you could carry.

That is not a failed experiment. It is an answer to a question the profit equation asks and market sizing never does.¹

Selection Bias: Who Chose to Respond?

Once a sampling frame is established, a second question immediately follows:

Of those who had a chance to respond, who actually did?

This is the problem of selection bias.

Selection bias arises when the people who choose to respond differ systematically from those who do not. In demand experiments, this difference is rarely random—and it rarely cancels out.

Response Is an Action, Not a Coin Flip

Responding to a survey is itself a choice.

That choice reflects motivation, attention, availability, interest, and perceived relevance. People who respond are not interchangeable with people who remain silent.

In demand experiments, respondents are often more likely to be:

interested in the problem or solution,
opinionated about the offering,
motivated to help,
optimistic about the idea,
or personally aligned with the entrepreneur or team.

None of this makes their answers dishonest. It does mean their answers are selective.

Silence Is Not Neutral

A common mistake is to treat non-response as missing data rather than as information.

In reality, silence often carries meaning.

People who do not respond may:

be indifferent,
be too busy,
find the offering irrelevant,
feel unsure how to answer,
or simply not care enough to engage.

In early demand learning, these silent individuals are often closer to the marginal customer—the ones whose decisions determine how steep or fragile demand really is.

Ignoring silence does not remove its effect. It hides it.

Why Selection Bias Inflates Early Demand

Selection bias tends to distort demand evidence in predictable ways.

Because respondents are often more engaged than non-respondents, early demand estimates frequently:

overstate appeal,
overstate willingness to pay,
understate price sensitivity,
and underrepresent hesitation or indifference.

This is why early demand evidence often looks encouraging—and why it so often fails to hold up when exposed to a broader audience.

The problem is not that respondents are lying.
The problem is that they are not typical.

Because selection bias tends to inflate early demand estimates, validation is essential.

The demand evidence validation toolkit provides practical ways to check whether encouraging early results hold up under replication or follow-up testing.

Selection Bias Compounds Sampling Bias

Selection bias does not occur in isolation.

It compounds whatever sampling limitations already exist.

If the sampling frame is narrow, selection bias operates within that narrow group. If the frame overrepresents highly engaged customers, selection bias amplifies that engagement further.

This is why well-designed surveys can still produce misleading confidence if execution effects are ignored.

Good demand learning requires attention to both:

who was allowed to respond, and
who chose to respond.

What Selection Bias Does Not Mean

Recognizing selection bias does not mean:

surveys are useless,
early demand cannot be learned,
or evidence should be ignored entirely.

It means that demand evidence must be interpreted as conditional, not definitive.

Early evidence is strongest when:

it aligns with realistic expectations,
it survives replication or follow-up testing,
and it remains stable across different samples or framings.

A Practical Discipline

Before acting on demand evidence, pause and ask:

Who might have been most eager to respond?
Who might have ignored this entirely?
How would the demand curve look if quieter, less motivated customers were better represented?

You do not need precise answers to these questions. You need to acknowledge that they exist.

Selection bias cannot be eliminated.
It can be respected.

The next section turns to a different execution risk—one that arises even when the right people respond: measurement error.

Measurement Error: What Did They Think You Asked?

Even when the right people are given a chance to respond—and even when they choose to do so—another risk remains.

Measurement error arises when respondents interpret a question differently than the entrepreneur intended.

This is not about carelessness or deception. It is about how humans make sense of questions in context.

Measurement Is an Interaction, Not a Transmission

Survey questions are not transmitted directly from designer to respondent.

They are interpreted.

Each respondent brings their own assumptions about:

what the product really is,
what “one unit” means,
how often the decision occurs,
what constraints apply,
and what situation they should imagine.

If those assumptions differ from the ones embedded in the experiment design, the responses may be internally consistent—and still answer a different question.

Ambiguous Units Create Invisible Error

One of the most common sources of measurement error is an unclear unit.

If respondents are unsure whether “one” means:

one item,
one visit,
one month,
one session,
or one bundle,

their willingness-to-pay and quantity responses become difficult to interpret.

Respondents will often resolve this ambiguity silently, choosing an interpretation that feels reasonable to them. The survey does not flag the mismatch. The data still looks clean.

The error appears only later, when conclusions fail to align with reality.

Time Frames Drift Easily

Time frames are equally fragile.

If a question does not clearly specify when the decision is made and how often it can repeat, respondents may answer using different horizons:

some imagining a one-time purchase,
others imagining occasional use,
others imagining habitual consumption.

When time frames drift, quantities and valuations drift with them.

This is why defining the decision period is not a technical detail. It is a prerequisite for meaningful demand evidence.

Hypothetical Contexts Invite Substitution

Measurement error also arises when respondents quietly substitute a different context for the one intended.

They may imagine:

a different version of the product,
a different competitive environment,
a different budget constraint,
or a different level of urgency.

This substitution is often unconscious. Respondents answer sincerely—just not about the scenario the entrepreneur had in mind.

The more abstract the question, the more room there is for substitution.

Clean Data Can Still Be Wrong

Measurement error is especially dangerous because it does not always create noise.

Responses can be smooth, consistent, and well-behaved—and still misaligned with the intended question.

When this happens, analysis does not correct the problem. It amplifies it.

Models faithfully process what they are given. If the measurement is off, the output will be precise and misleading at the same time.

Measurement error often reflects gaps in how units, time frames, or scenarios were specified.

When this occurs, the right response is not adjustment but redesign.
The problem framing toolkit and the survey design toolkit provide structured ways to revisit these design choices before proceeding to analysis.

What Measurement Discipline Looks Like

Measurement discipline does not require perfect control.

It requires attention to whether respondents are answering the same question you think you are asking.

Signs of measurement trouble include:

unexpected dispersion in responses,
implausible quantities or prices,
confusion in open-ended answers,
or difficulty explaining results in plain language.

When these appear, the right response is not adjustment or correction. It is redesign.

Measurement Error Is a Design–Execution Boundary

Measurement error sits at the boundary between design and execution.

Good design reduces it. Good execution does not eliminate it.

This is why demand learning is iterative. Early experiments reveal not just demand, but also flaws in how demand is being measured.

Recognizing measurement error early is a strength, not a failure.

The final section of this chapter explains why ignoring these risks is especially dangerous—and why bad data is often worse than no data at all.

Why Bad Data Is Worse Than No Data

When demand experiments fail, they rarely fail loudly.

More often, they produce results that look reasonable, quantitative, and actionable—while quietly pointing in the wrong direction.

This is what makes bad data dangerous.

No Data Leaves You Aware of Your Ignorance

When no data exists, entrepreneurs know where they stand.

They may be uncertain, uncomfortable, or hesitant—but they are aware that judgment is being exercised without evidence. Decisions made in this state tend to be cautious, provisional, and revisable.

Ignorance is visible.

That visibility matters. It keeps entrepreneurs alert to risk and open to learning.

Bad Data Creates False Confidence

Bad data does something different.

It replaces visible ignorance with illusory knowledge.

Numbers appear. Charts can be drawn. Estimates can be computed. The language of evidence enters the conversation—even though the evidence itself is fragile or misaligned.

Once this happens, uncertainty does not disappear. It becomes harder to see.

Decisions made on bad data often feel responsible precisely because they are justified quantitatively. This is why they are so difficult to unwind later.

The demand evidence validation toolkit is designed for precisely this situation: to help determine whether evidence is strong enough to support commitment, or whether learning must continue.

Models Amplify, They Do Not Correct

Analytics do not rescue flawed data.

Models assume that inputs mean what we think they mean. They do not question:

whether the right people were sampled,
whether respondents interpreted questions consistently,
whether quantities and prices refer to the same unit and period,
or whether the scenario was cognitively real.

When these assumptions are violated, models do not fail gracefully. They amplify error with precision.

The result is not random noise. It is confident misdirection.

When demand evidence is fragile, inconsistent, or poorly grounded, the most responsible next step is often not further analysis—but validation.

Why Entrepreneurs Are Especially Vulnerable

Entrepreneurs are particularly exposed to this risk for three reasons.

First, early decisions feel urgent. There is pressure to “get something” rather than wait for better evidence.

Second, early demand evidence often comes from surveys or small samples, where design and execution errors have outsized effects.

Third, once commitments are made—pricing decisions, investments, launches—it becomes psychologically and organizationally difficult to revisit the underlying data.

Bad data does not just mislead a single decision. It shapes a path.

A Conservative Rule of Thumb

A useful rule in early demand learning is this:

If the data would make you more confident than you should be, you should not use it yet.

This does not mean you need perfect evidence. It means you need evidence whose limitations you understand.

Sometimes the right response to a weak experiment is not adjustment or correction, but restraint. Choosing not to act on fragile results is itself a disciplined decision.

Poor execution does not just create noise — it creates data that should not be estimated at all.

Before moving on to demand estimation, it is worth pausing to check whether the evidence you have collected actually supports the kind of inference you want to make. The toolkit Preparing Data for Demand Estimation provides a short set of checks designed to catch problems before modeling turns them into false confidence.

What This Chapter Was Really About

This chapter was not about technique.

It was about protecting judgment.

Designing and executing demand experiments responsibly is not about eliminating uncertainty. It is about ensuring that whatever confidence you gain is earned—and that whatever uncertainty remains is visible.

The goal is not to move faster.
It is to move with integrity.

In the next chapter, we turn from execution to interpretation: how demand evidence is transformed into demand curves, and how judgment enters once again—this time in estimation.

Expeditionary Innovation makes this test the explicit gate of its first phase: see Test Your Access. The act is the same and the purpose differs. There you are checking whether sustained contact is possible before committing months to a community. Here you need that contact to produce a demand curve. A failure in either place is the same finding.↩︎