Sources of Experimental Errors - Market Research

After having described the different types of experiments, we now turn to sources of potential errors in experiments.

There are several errors which may distort the accuracy of an experiment. These are briefly described below.

• History: 

History refers to the effect of extraneous variables as a result of an event that is external to an experiment occurring at the same time as the experiment.

For example, consider the design O1X O2 where O1 and O2 represent the sales affected by salesmen in an enterprise in the pre-training period and post-training period, respectively and X represents a sales training programme.

This experiment is expected to indicate the effectiveness of the sales training programme by showing higher sales in the post-training period as compared to sales in the pre-training period.

If the general business conditions have improved during the training period, when the sales could have risen even without the sales training programme.

• Maturation: 

Although maturation is similar to history, it differs from it, as the actual outcome is usually less evident.

Maturation refers to a gradual change in the experimental units arising due to the passage of time.

In our earlier example of training programme, salesmen have become more matured and more experienced due to the passage of time.

As a result, the improvement in sales performance cannot be attributed to the training programme alone.

Another example could be of consumer panels.

The members of such panels forming test units may change their purchase behaviour during the period when an experiment is on.

As the time between O1 and O2 becomes longer, the chance of maturation affects also increases.

• Pre-measurement effect: 

This error is caused on account of the changes in the dependent variable as a result of the effect of the initial measurement.

For example, consider the case of respondents who were given a pretreatment questionnaire. After their exposure to the treatment, they were given another questionnaire, an alternative form of the questionnaire completed earlier.

They may respond differently merely because they are now familiar with the questionnaire. In such a case, respondents’ familiarity with the earlier questionnaire is likely to influence their responses in the subsequent period.

• Interactive testing effect: 

This error arises on account of change in the independent variable as a result of sensitizing effect of the initial measurement. In other words, the first observation affects the reaction to the treatment.

For example, consider the case that respondents have been given a pretreatment questionnaire that asks questions about various brands of hair oil.

The pretreatment questionnaire may sensitise them to the hair oil market and distort the awareness level of new introduction, i.e. the treatment. In such a case, the measurement effect cannot be generalised to non-sensitised persons.

• Instrumentation: 

Instrumentation refers to the changes in the measuring instrument over time.

For example, consider the case when the interviewer uses a different format of a questionnaire in O2 as compared to that used in O1 .

This would case an instrumentation effect.

A similar example could be of an interviewer who in his enthusiasm and interest in the survey in O1 , explained to the respondents whenever there was any difficulty.

But the same interviewer gradually loses his interest in the survey and does not explain properly to the respondents in the post-measurement period-O2 .

Yet another example could be when sales aremeasured in terms of revenue and the company has increased the prices of its products in the intervening period.

• Selection bias: 

Selection bias refers to assigning of experimental units in such a way that the groups differ on the dependent variable even before the treatment.

Such a situation arises when test units may choose their own groups or when the researcher assigns them to groups on the basis of his judgment.

To overcome this bias, it is necessary that test units be assigned to treatment groups on a random basis.

• Statistical regression: 

Statistical regression effect occurs when test units have been selected for exposure to the treatment on the basis of an extreme pretreatment measure.

For example, a training programme may be devised only for salesmen whose performance have been very poor.

Sales increases in the post-treatment period may then be attributed to the regression effect.

This is because random occurrences such as weather, health or luck may contribute to the better performance of salesmen in the subsequent period.

Thus the effect of training programme may get distorted on account of this factor.

• Mortality: 

Mortality refers to the loss of one of more test units while the experiment is in progress.

It may be emphasised that mortality leads to the differential loss of respondents from the various groups.

This means that respondents, who left, say group A are different from those who left group B, thus making the groups incomparable.

In case the experiment pertains to only the group, mortality effect occurs when responsiveness of the respondents who have remained in the experiment differs from responsiveness of those who have ceased to be in the experiment.