PPA 696 RESEARCH METHODS

EXPERIMENTAL DESIGNS  FOR RESEARCH

CAUSALITY

To establish whether two variables are causally related, that is, whether a change in the independent variable X results in a change in the dependent variable Y, you must establish:

1) time order--The cause must have occurred before the effect;

2) co-variation (statistical association)-- Changes in the value of the independent variable must be accompanied by changes in the value of the dependent variable;

3) rationale-- There must be a logical and compelling explanation for why these two variables are related;

4) non-spuriousness-- It must be established that the independent variable X, and only X, was the cause of changes in the dependent variable Y; rival explanations must be ruled out.

EXPERIMENTAL DESIGNS

True experimental designs include:

Pre-test/Post-test control group design

Solomon Four-Group design

Post-test only control group design

Pre-test/Post-test control group design
This is also called the classic controlled experimental design, and the randomized pre-test/post-test design because it:

1) Controls the assignment of subjects to experimental (treatment) and control groups through the use of a table of random numbers.

This procedure guarantees that all subjects have the same change of being in the experimental or control group.  Because of strict random assignment of subjects, it is assumed that the two groups are equivalent on all important dimensions and that there are no systematic differences between the two groups.  Researchers may substitute matching for random assignment.  Subjects in the two groups are matched on a list of characteristics that might affect the outcome of the research (e.g., sex, race, income).  This may be cheaper but matching on more than 3 or 4 characteristics is very difficult.  And if the researcher does not know which characteristics to match on, this compromises internal validity.

2) Controls the timing of the independent variable (treatment) and which group is exposed to it.

 Both group experience the same conditions, with the exception of the experimental group, which receives the influence of the independent variable (treatment) in addition to the shared conditions of the two groups.

3) Controls all other conditions under which the experiment takes place.

Nothing but the intervention of the independent (treatment) variable is assumed to produce the observed changes in the values of the dependent variable.

The steps in the classic controlled experiment are:

1) randomly assign subjects to treatment or control groups;

2) administer the pre-test to all subjects in both groups;

3) ensure that both groups experience the same conditions except that in addition the experimental group experiences the treatment;

4) administer the post-test to all subjects in both groups;

5) assess the amount of change on the value of the dependent variable from the pre-test to the post-test for each group separately.

    R     O₁     X     O₂
    R     O₁             O₂

This diagram can be expanded upon as in the following table:
 

Scientific Random Assignment of Subjects to:	1st observation (measurement) of the dependent variable  O1 = Pre-test	Exposure to the Treatment (X) (independent variable)	2nd observation (measurement) of the dependent variable O2 = Post-test
Experimental Group	Experimental Group's average score on the dependent variable	X	Experimental Group's average score on the dependent variable
Control Group	Control Group's average score on the dependent variable		Control Group's average score on the dependent variable

The difference in the control group's score from the pre-test to the post-test indicates the change in the value of the dependent variable that could be expected to occur without exposure to the treatment (independent) variable X.

Control group    -    control group     =    control group difference
pre-test score        post-test score            on the dependent variable

The difference in the experimental group's score from the pre-test to the post-test indicates the change in the value of the dependent variable that could be expected to occur with exposure to the treatment (independent) variable X.

Experimental group    -    experimental group     =    experimental group difference
pre-test score                post-test score                    on the dependent variable

The difference between the change in the experimental group and the change in the control group is the amount of change in the value of the dependent variable that can be attributed solely to the influence of the independent (treatment) variable X.

Control group difference    -    experimental group difference    =    difference attributable to X

This can be illustrated by the following experiment to see whether participation in small group discussions would improve medical students' ability to respond to emotional needs of patients:
 

Scientific Random Assignment of Medical Students to:	How many times did students use emotional words to describe patients	Exposure to the Treatment (X) (independent variable)	How many times did students use emotional words to describe patients
Small group discussions (experimental group)	Average of .68 times per student in 3 case studies	Attended small group discussions plus regular course work	Average of 2.02 times per student in 3 case studies
Control Group	Average of .89 times per student in 3 case studies	Regular course work only	Average of 1.13 times per student in 3 case studies

The control group used emotional words an average of .89 times per student (in three case studies) on the pre-test and an average of 1.13 times per student (in three case studies) on the post-test.  The difference in the control group's score from the pre-test to the post-test is +.24 times per student.  This indicates the change in using emotional words that could be expected to occur with regular course work only.

The experimental group used emotional words an average of .68 times per student (in three case studies) on the pre-test and an average of 2.02 times per student (in three case studies) on the post-test.  The difference in the experimental group's score from the pre-test to the post-test is +1.34 times per student.  The experimental group's score from the pre-test to the post-test indicates the change in using emotional words that could be expected to occur with regular course work plus the small group discussions.

The difference between the change in the experimental group (+1.34) and the change in the control group (+.24) is +1.10.  This is the amount of change in using emotional words that can be attributed solely to the influence of the small group discussions.

The controlled or true experimental design allows the researcher to control for threats to the internal and external validity of the study.  Threats to internal validity compromise the researcher's ability to say whether a relationships exists between the independent and dependent variables.  Threats to external validity compromise the researcher's ability to say whether this study's findings are applicable to any other groups.

Controlling for Threats to Internal Validity

2) Maturation:  were changes in the dependent variable due to normal developmental processes?  No, because both groups experienced the same developmental processes.

3) Statistical Regression:  did subjects come from low or high performing groups?  Differences between the two groups that could influence the dependent variable would be controlled for as subjects were generally equivalent at the beginning of the research.

4) Selection:  were the subjects self-selected into experimental and control groups, which could affect the dependent variable?  No, the subjects were assigned by strict random selection and all had equal chance of getting the treatment or control condition.

5) Experimental Mortality:  did some subjects drop out?  did this affect the results?  About the same number of students made it through the entire study in both the experimental and control groups, so there appears to be no bias.

6) Testing:  Did the pre-test affect the scores on the post-test?  Both groups got a pre-test; but a pre-test may have made the experimental group more sensitive to the treatment.

7) Instrumentation:  Did the measurement method change during the research?  The measurement method and instruments did not change.

8) Design contamination:  did the control group find out about the experimental treatment?  did either group have a reason to want to make the research succeed or fail?  The researcher must do some qualitative investigation to find out if there was design contamination.
 

Controlling for Threats to External Validity

2)  Effects of Selection:  Probably applicable to other medical students.

3) Effects of Setting:  Medical schools have their own cultures; doubtful if this would be applicable to other types of students.

4) Effects of History:  No information given

5) Effects of Testing:  No information given

6)  Reactive effects of experimental arrangements:  It would be best to replicate the results in other medical schools.
 

Post-Test Only Control Group Design

For small groups, however, a pre-test is necessary.  Also, a pre-test is necessary if the researcher wants to determine the exact amount of change attributable to the independent variable alone.
Public administrators would like to be able to use experimental designs for policy and program evaluation.  Did a regional economic development policy bring more business to the economically depressed region?  Did the Women-Infants-and-Children (WIC) program lower the rate of malnutrition in young children?
 

Unfortunately, it is difficult for public administrators to meet the requirements of the classic controlled experimental design.

 -It is difficult to conduct program evaluations in a laboratory, where other influences can be controlled.

 -It is difficult to achieve random assignment, due to political and ethical concerns.

 -Policies may not be specific as to what changes they intend to produce.

 -Often funds or other resources for a large-scale research project are lacking.

 -Decision makers often operate on short time frames and cannot wait for an experimental study to run its course.

 -Research is not always begun before a program is implemented, and therefore cannot take measurements of the "before" condition.

 -Programs are not always implemented all at once, but rather gradually, over time, which may diminish their effects.

 -Programs may not be implemented as originally intended, or may not produce the effects that researchers are looking for.

 -Small treatments with modest goals are more amenable to a controlled experimental design than large scale social programs with ambitious goals.