Mitigating Investigation Issues

Planning for problems before they happen

A statistical enquiry uses data to investigate a question or hypothesis. Even a sensible plan can encounter difficulties: the intended population may be hard to identify, selected people may not respond, or the results may be unexpected.

A proactive strategy is an action planned before a problem occurs, rather than a response invented afterwards. It can prevent a problem or minimise its effects. For each possible difficulty, consider what could go wrong, how it could affect the conclusions, and what action would reduce that risk.

Identifying exactly who belongs in the population

The target population is the complete set of people, objects or events about which you want to draw conclusions. A sample is the subset from which you actually collect data. These are not necessarily the same: collecting answers from a few students does not mean the population is just those students.

In an investigation of opinions about a school's uniform, the population would be all students at that school, not teachers or other staff. Defining this before collection prevents different data collectors from including different kinds of people.

A sampling frame is a list of population members from which a sample can be selected. For the uniform investigation, a current school student list would be suitable. Check that the list covers the intended population and is up to date. An incomplete list excludes some people before selection even begins; selecting randomly from it cannot repair those omissions.

Sometimes a complete list is difficult to obtain. For example, it is not straightforward to list all the fish in a lake. The collection method must then suit what is actually accessible, and its limitations must be considered when drawing conclusions.

Choosing only convenient people can also leave groups poorly represented. Interviewing the first students arriving at the school gates might capture a group travelling on the same bus. Deliberately including students from each year group improves coverage, although choosing convenient students within each group can still introduce bias. The important question is whether the people reached can reasonably represent the intended population.

Reducing non-response and its effects

Non-response occurs when a selected person does not provide a response, perhaps because they cannot be contacted or do not wish to participate. An incomplete response is different: a questionnaire is returned, but some questions are unanswered.

Both reduce the amount of usable data. They may also introduce bias if people who respond differ from those who do not. For example, a survey sent to businesses on a database may miss businesses that have closed. Struggling or unsuccessful businesses could therefore be under-represented, giving an overly favourable picture.

Plan ways to make participation easier:

• Use short, clear questions and give respondents enough time to answer.
• Plan reminders or another contact attempt for people who do not respond, without pressuring them to participate.
• Consider an anonymous questionnaire for sensitive topics, since people may feel more comfortable answering honestly.
• Consider interviews where questions may need explanation, while recognising their greater time and cost and the possibility of interviewer influence.

There is no method that guarantees complete, honest responses. Interviews allow clarification and tend to obtain more answers from people interviewed, but respondents may give answers they think the interviewer wants. Anonymous questionnaires can be quicker and easier to complete privately, but unanswered or misunderstood questions are harder to resolve.

Choose the method that best addresses the likely difficulty. Monitor which groups are responding: a large number of replies does not by itself show that the responses represent the whole population.

Testing the collection method on a small scale

A pilot survey is a small trial before the main survey. It can reveal confusing wording, missing answer options or questions that people regularly leave blank. The purpose is to improve the method before the same problem affects many responses.

For example, an app-use question offering only ‘once’, ‘twice’ and ‘three to five times per day’ leaves no suitable answer for someone who never uses the app or uses it more than five times. Adding suitable options prevents those respondents from having to leave the question blank or choose an inaccurate answer.

Questions also need clear time periods and non-overlapping answer categories. A small pre-test serves a similar purpose in an experiment: try the procedure on a small sample, identify practical problems, and fix them before full collection.

Preventing missing values and recording errors

Prepare the recording sheet before collecting data. Give each required measurement a clear place and state the units to use. This reduces forgotten measurements and inconsistent recording.

For example, when collecting puppies' ages and weights, separate labelled columns help the collector record both values for each puppy. Checking the sheet during collection makes it easier to recover a missing age while the owner is still available.

Plan to check data before analysis. Look for missing entries, misplaced decimal points and inconsistent units. If a value is missing, try to obtain it from the original respondent or record rather than inventing it. If only part of a record is usable, decide whether it can contribute to the relevant analysis, and consider how keeping or removing it could affect the conclusions.

Handling unexpected values and conclusions fairly

An outlier is unusually large or small compared with the other values. It is a reason to investigate, not automatic proof of an error. A company chief executive's very high salary could be genuine. A sports-club member's recorded age of 500 is likely to be a recording mistake.

Plan to check unusual values against the original records or the circumstances of collection. Correct an error where the correct value can be established; otherwise remove a known incorrect value and explain why. Keep genuine unusual values rather than deleting them merely because they are inconvenient.

An unexpected overall result is not necessarily a collection problem either. The data may fail to support the original hypothesis. Check the method and data for weaknesses, then report the conclusion the evidence supports. Evaluation may suggest an improved investigation, but it should not be used to change sound results to match expectations.