A Survey Result Includes the Question That Produced It

A survey percentage is an answer to a particular question asked of particular people in a particular way. Removing the wording and response options can make the result sound broader than the evidence supports. “Seventy percent support the service” may conceal whether respondents were asked about its existence, its quality, its hours, or one proposed change.

The question is part of the measurement instrument. It should be read alongside the sample, field dates, response options, and method. A large number of responses does not repair a question that asks two things at once or leaves respondents without an accurate answer choice.

Start with an ambiguous question

Imagine a fictional survey about a local recreation center. It asks: “Are the center's hours convenient and its activities affordable?” The choices are yes and no.

A respondent who finds the hours convenient but the prices difficult has no single accurate answer. Another who finds the activities affordable but cannot attend during opening hours faces the same problem. A “no” can therefore mean several different things.

Suppose 60% answer yes. That result cannot establish that 60% separately approve of hours and 60% separately approve of prices under independent questions. The survey measured agreement with a combined statement, with whatever interpretation respondents gave it.

Pew Research Center's question-writing guide discusses this problem with questions containing more than one concept. The solution begins by defining what the survey intends to measure, not by finding a more impressive way to summarize the percentage.

Separate the concepts before interpreting the result

Two distinct questions might ask about the convenience of the center's opening hours and the affordability of the activities the respondent considered. Each still needs an appropriate time frame, audience, and response scale.

The distinction produces a useful interpretation table:

The table is an illustration, not a claim that these four patterns exhaust every real experience. It shows information that a single yes-or-no combined question cannot recover afterward.

Once the original survey has been completed, splitting its wording in the report does not create two measurements. A reader should preserve the original question and acknowledge its limitation rather than assigning separate meanings to the same answer.

A time frame defines the experience being recalled

“Do you use the center?” might include someone who visited once five years ago, someone attending weekly, and someone planning a first visit. “During the past 30 days, how many times did you visit?” asks a more specific behavior question, but it still needs clear boundaries about what counts as a visit.

The two questions are not interchangeable measures. A change from one to the other can change the result even if behavior has not changed.

AAPOR's best-practice guidance emphasizes specific questions and pretesting with people similar to the intended respondents. The point is to learn how respondents interpret the wording, not merely to check whether the form opens on a phone.

For the fictional center, a pretest might reveal that people count an outdoor event as a visit while the survey designer meant only entry to the building. That discovery improves the definition before the percentage becomes a headline.

Answer choices can create or remove ambiguity

Consider a visit-frequency question with choices “0–2,” “2–4,” and “4 or more.” A respondent with exactly two visits fits two categories; someone with four fits two as well. The overlap makes the result depend partly on arbitrary choice.

For nonnegative whole-number visits, a set such as “0,” “1–2,” “3–4,” and “5 or more” separates those counts more clearly. The categories still reflect a design decision about how much detail to preserve.

A missing category can be equally consequential. If only “satisfied” and “dissatisfied” are offered, someone with no experience of the service may be forced into a response that does not describe them. A neutral view, uncertainty, refusal, and lack of applicability are also different states; they should not be casually collapsed into one label after collection.

The appropriate choices depend on the question. A form should not add every imaginable option indiscriminately, but the reporting must make clear which choices were available and how missing answers were handled.

Open answers and prompted choices measure different tasks

“What would most improve the center?” invites respondents to generate an answer. A list containing longer hours, lower prices, more activities, and better transport asks them to recognize and choose among supplied ideas.

Pew's methods guide explains that open and closed formats can produce different response patterns. The options themselves make certain ideas available in the moment. A frequently selected prompted option is not necessarily the issue respondents would most often mention without prompting.

Neither format is universally superior. An open format can reveal unanticipated concerns, while a structured format can support a specified comparison. The result should retain the task: volunteered answer or selection from a stated list.

When open answers are coded into categories, the coding scheme becomes another part of the measurement. Two phrases may be grouped together or separated depending on the rules. Reported category counts should not be treated as though respondents necessarily selected those exact labels themselves.

“Select all” changes the denominator story

Suppose respondents can select every activity they use. One person can appear in several activity counts, so the percentages may legitimately add to more than 100%. That does not necessarily indicate a tabulation error.

In an invented sample of 100 respondents, 60 select swimming, 50 select classes, and 30 select court sports. The total is 140 selections, but there are still 100 people. A statement that “43% of respondents chose swimming” based on 60 divided by 140 would use selections rather than respondents as the denominator.

The two calculations answer different questions: the share of people who selected swimming, or swimming's share of all selections. The report needs to name which one it uses.

Our percentage-points guide explains why a percentage becomes meaningful only with its base. A chart label that says “percent” is not enough when respondents can contribute more than one answer.

Question order supplies context

Imagine a survey that first asks respondents to recall problems with parking, noise, and crowded rooms, then asks for an overall rating of the center. Those earlier questions may make particular experiences more salient than they would be at the start of the questionnaire.

The reverse order can produce a different context. A general rating asked first is not necessarily the same measurement as that rating asked after a detailed list of complaints or benefits.

Both Pew and AAPOR discuss order effects and the importance of comparable context in repeated surveys. Randomizing some items can distribute order effects, but it does not make wording irrelevant or justify randomizing a scale whose order carries meaning.

For a reader, the practical requirement is modest: inspect the questionnaire sequence, especially when a result is surprising or a trend depends on small changes. The preceding questions can be part of the explanation.

An agreement statement is not always a neutral shortcut

“The center provides excellent value” followed by agree/disagree asks respondents to react to the researcher's assertion. A question asking them to rate value on a clearly defined scale poses a different task.

The wording can also assume facts that do not apply. “How much did the new schedule improve your visits?” presumes improvement. A respondent who experienced no change or found the schedule worse needs an accurate route to express that.

These are examples of measurement choices, not evidence that every survey using agreement scales is invalid. The question should fit the concept and population, and its limitations should be considered when interpreting answers.

Avoid replacing a respondent's measured opinion with a more flattering or more critical paraphrase. “Rated value positively under this scale” is a narrower claim than “proved the program is worth its cost.”

Repeated surveys need a stable instrument

Suppose last year's survey asked whether the center was “affordable,” while this year's asks whether its prices were “reasonable for the quality provided.” Those phrases may overlap, but they are not identical questions. A difference in results can reflect changed experience, changed interpretation, or both.

AAPOR recommends keeping wording, framing, and methodology as comparable as possible when measuring change. If a change is needed, a designed comparison of versions can help assess its effect. It should not be assumed that a nicer-sounding rewrite preserves the old measure automatically.

Changes in interview mode can matter too. A response given privately online may differ from one given to an interviewer. The relevant report should explain methodological changes rather than presenting every difference as a shift in the underlying opinion.

The time series is strongest when its measurement history is visible. A broken comparison is not repaired by using the same chart color across years.

A large sample cannot answer a different question

Ten thousand responses to the combined hours-and-affordability question still do not reveal the separate answers. More observations can improve some aspects of precision while leaving the ambiguity intact.

Our margin-of-error guide explains sampling uncertainty. Question wording, coverage, nonresponse, and reporting choices are additional issues. A narrow sampling margin should not be presented as a complete measure of all uncertainty in a survey claim.

Likewise, a survey of current users does not automatically describe everyone who might use the center. Nonusers may face barriers or have preferences that the sample never captured. The audience named in the headline must match the population the study can support.

What a result can support without becoming causal

Suppose respondents who report longer travel times give lower convenience ratings. The association is useful, but it does not by itself show how much ratings would improve if a transport service were added. Other differences between respondents may matter.

Our correlation and causation guide explains the extra comparison needed for a causal claim. A survey can describe experiences, attitudes, and reported behavior while leaving intervention effects unresolved.

The same caution applies to averaging rating scales. Our mean and median guide explains that a summary needs a clear interpretation. The numerical coding of response categories and the distribution of answers belong in the analysis, especially when a mean is used to compare groups.

A denominator change can mimic an opinion change

Suppose an invented survey has 100 respondents: 60 positive answers, 20 negative answers, and 20 people who say the question does not apply. Positive responses are 60% of all respondents. Among the 80 people giving a positive or negative rating, they are 75%. Both fractions can be computed from the same answers.

If one annual report uses all respondents and the next excludes the not-applicable group, a chart can appear to show a rise from 60% to 75% even when the response pattern is unchanged. The apparent fifteen-percentage-point improvement comes entirely from a reporting change.

This example does not imply that either denominator is always preferable. One describes the whole responding group; the other describes those providing a substantive rating under that classification. The report should identify which question it intends to answer and apply the definition consistently.

Missing answers require similar care. An unanswered item is not automatically negative, neutral, or not applicable. Coding it into one of those categories without a stated basis adds a meaning the respondent did not provide. Preserve the category and explain how it enters the calculation.

Keep a short evidence record beside the headline

A reader should be able to find the exact question, response options, eligible respondents, field dates, collection mode, and treatment of missing answers. For repeated surveys, the record should identify changes in wording or method. For a percentage, it should identify the denominator.

Then the headline can stay within the evidence: “Among surveyed recent users, this share selected these responses to this question.” That wording may be less sweeping than “everyone loves the service,” but it is more useful because another reader can tell what was actually measured.

The survey result includes the question that produced it. Keeping that question visible preserves the meaning of the percentage and makes both its value and its limits easier to assess.