The purpose of this chapter is to help you meet the following objectives:

1. Define external validity.

(Review question 1)

(Textbook emphasis pp. 337-338)

2. Define and give examples of each of the threats to external validity.

(Review questions 2-4)

(Textbook emphasis pp. 338-342)

3. Describe how each of these threats operates to weaken the external validity of an experiment.

(Review question 5)

(Textbook emphasis pp. 338-342)

4. Describe strategies for overcoming these threats to external validity.

(Review questions 6-9)

(Textbook emphasis pp. 343-350)

5. Describe and identify examples of correctly written operational definitions.

(Review questions 10-11)

(Textbook emphasis pp. 345-348)

6. Given a research variable, write an operational definition of it.

(Review questions 12-13)

(Textbook emphasis pp. 345-348)

The information contained in the programmed unit of Chapter 4 in the Workbook would be a valuable review for objectives 5 and 6 of this chapter.

REVIEW QUIZ

1. Which of the following provides the best description of the term external validity?

a. It deals with the question of whether the treatment really caused the outcome observed at the end of an experiment.

b. It deals with the question of whether the measurement strategies in an experiment actually measured what they claimed they were measuring.

c. It deals with the question of how far the results of an experiment can be generalized.

d. It deals with the question of the actual practical importance of the results of an experimental study.

e. It deals with the relationship of the results of an experimental study to the overall status of the information which could be gained from an extensive review of the literature on the topic.

2. Mrs. Watkins has developed a new strategy for reducing fear of public speaking. She has time to try it on only five students in the course of this semester. She therefore asks for volunteers from among her sophomore students. Ten students volunteer and she randomly assigns five of these to the experimental treatment, while using the other five as a control group. At the end of the treatment, she discovers that the experimental students have reduced their fear of public speaking substantially more than those in the control group, who received a traditional treatment. Mrs. Watkins therefore concludes that her new approach is effective and recommends its use throughout the school. Based on this information, which of the following is the most obviously serious threat to the external validity of Mrs. Watkins' study?

a. An interaction of the treatment with persons.

b. An interaction of the treatment with the setting.

c. An interaction of the treatment with time.

d. None of the above appears to be a serious threat to the external validity of this study.

3. Mr. Grey decided that since there had been major publicity in the newspaper about drug-related arrests, it would be a good idea to introduce a unit to his eighth graders about drug abuse. He wanted to find out whether or not the unit would be effective. Since he taught four separate social studies classes, Mr. Grey introduced the unit to one group first and used the other three as a control group. Through this procedure he discovered that the scores of the first group went up immediately when the unit was introduced; and the scores of the other groups went up later when it was introduced to them. He concluded on this basis that the new drug abuse unit was effective and he decided to continue offering it in subsequent years because of this proven success. Based on the information given, what is the most obvious problem with the external validity of Mr. Grey's study?

a. The interaction of the treatment with persons.

b. The interaction of the treatment with the setting.

c. The interaction of the treatment with time.

d. None of the above poses a particularly serious threat to the external validity of the study.

4. Mrs. Weller has developed a new strategy for teaching language skills to her preschool children. She supervises two preschool classes. At the beginning of the year she obtains permission from thirty parents to involve their children in the experimental program. She randomly assigns fifteen of these children to the experimental group, which meets in the morning, and the other fifteen to the control group, which meets in the afternoon. She gives the experimental treatment to the morning group, and she notifies the parents of this. She likewise notifies the parents of the afternoon group that they will initially be a control group, but that during the second half of the year, they also will receive the treatment. Mrs. Weller herself teaches both of the classes. She finds at the end of her study that based on her observation of the language patterns of the children, the experimental group far outperforms the control group. Based on the information given, what is the most serious threat to the external validity of Mrs. Weller's study?

a. The interaction of the treatment with persons.

b. The interaction of the treatment with the setting.

c. The interaction of the treatment with time.

d. None of the above poses a particularly serious threat to the external validity of this study.

5. When we say that the interaction of the treatment with time is a threat to the external validity of an experiment, what does this mean?

a. The fact that the treatment was not centrally located between the pretest and the posttest could help account for the findings of the experiment.

b. It could have been some historical event that occurred simultaneously with the experiment, rather than the experiment itself, which caused the outcome of the experiment to occur.

c. It could have been a combination of some historical event and the treatment (rather than the treatment alone) which caused the observed outcome.

d. Results which occur at one time of the day are not necessarily likely to be replicated if the study were conducted at a different time of the day.

e. None of the above is an adequate explanation of this statement.

Examine the following four statements and determine which of the listed strategies would be directly helpful in improving the external validity of the study.

6. Mr. Rostankovsky has developed a procedure for increasing the critical thinking ability of high school students during junior year social studies. He has conducted an experimental study with his third-period students to demonstrate with good internal validity that the procedure actually works. He is concerned, however, that his third-period students may not be typical of the other students to whom he would like to be able to generalize this procedure. What is the first thing that Mr. Rostankovsky should do to increase the external validity of his study?

a. Replicate the experiment with a new group of subjects the next semester.

b. Repeat the study in a more lifelike situation.

c. Determine the extent to which his third period students are in fact similar to the other students to whom he wishes to generalize.

d. Replicate the study on a larger group of subjects.

e. Operationally define his treatment more concretely.

7. Mrs. Gordon has developed a counseling technique to reduce test anxiety among seventh and eighth grade students. She has proven it to be effective on students who volunteered to take part in the treatment, but she wants to be able to apply the technique even to those students who do not volunteer for such a treatment. Which of the following steps would most effectively help her to make this generalization?

a. Operationally define her treatment more clearly.

b. Replicate the experiment with some nonvolunteers.

c. Define and determine the ways in which the volunteers differ from nonvolunteers.

d. A combination of a and b would be most effective.

e. A combination of b and c would be most effective.

8. Mrs. Wallace administered a test to her elementary school students to determine their attitude toward policemen. Then she had Officer Friendly visit the class and discuss ideas about policemen with the children. Afterwards, she retested them with a similar test and found that the experimental group who met with Officer Friendly far exceeded the control group in the performance on the posttest. The study possessed good internal validity, but Mrs. Wallace was concerned that it might have been a combination of the pretest and the treatment (rather than the treatment alone) which caused the improved attitudes. Which of the following steps would most effectively help Mrs. Wallace overcome this threat to the external validity of her study?

a. Replicate the study again on a population of children as different as possible from the first one.

b. Replicate the study again with a different Officer Friendly.

c. Replicate the study with a similar group of people approximately a year later.

d. Use a less obtrusive pretest.

e. A combination of a and b would be the most effective.

9. Since the local public television was going to broadcast a full-length version of the novel Moby Dick, Mr. Twain decided this would be a good time to alter his approach to teaching that novel in his class. Therefore, he developed a new unit and introduced this unit to two of his classes while having the other two use the old approach. The study possessed good internal validity, and the results indicated that the new approach actually worked. If Mr. Twain wanted to generalize his results to students in subsequent years, which of the following steps would be most helpful in extending the external validity of his study?

a. Replicate the experiment with a less obtrusive pretest and posttest.

b. Replicate the experiment on a similar group of students.

c. Replicate the experiment on a dissimilar group of students.

d. Replicate the study among a group of students who had not watched the television series.

e. None of the above would be effective in increasing the external validity of his generalizations.

10. Which of the following is the most important characteristic of an operational definition?

a. It must be stated in conceptual terms which are familiar to the reader or listener.

b. The definition should conceptually distinguish the variable from other variables with which it might be confused.

c. The operational definition should state observable behavior, events, or characteristics which we are willing to accept as evidence that the variable exists.

d. The operational definition should focus on the outcome, rather than on the treatment.

e. The operational definition should focus on the treatment, rather than on the outcome.

11. A researcher has stated that she has proven that computer simulations lead to increased ability in scientific problem solving. If we wanted to know her operational definition of computer simulations, what would we ask her?

a. How did she refute the possibility that something else rather than the computer simulations might have caused the improvement?

b. What type of test did she use to measure problem solving ability?

c. What kind of computer simulations did she use and how did she introduce them?

d. Why did she think this outcome occurred?

e. How does she know the same outcome would occur under different circumstances?

12. A researcher reported the results of an experimental study that showed that "listening to popular music enables children to learn spelling words more effectively." Which of the following would be the most useful operational definition of "listening to popular music"?

a. Popular music is the music which is preferred by the majority of the students in the class.

b. Popular music is the music of the current peer group.

c. Listening to popular music causes relaxation and reduces anxiety,

and therefore leads to an atmosphere conducive to learning.

d. The children listened to music played from a prerecorded tape (consisting of the top 40 records during the week prior to the study session) while they studied their spelling words quietly.

e. None of the above provides an operational definition of "listening to popular music."

13. A researcher reported that "allowing the students to take part in the decision-making process reduces antagonism towards the teacher." Which of the following is the best operational definition of "antagonism toward the teacher"?

a. Antagonism is a feeling of hostility toward the teacher.

b. Antagonism is a feeling of hostility based on irrational reactions to the teacher's behavior.

c. Antagonism is defined as making a contemptuous sound or remark to the teacher.

d. Antagonism is the opposite of friendliness.

POSSIBLE PROBLEMS AND SOLUTIONS

1. Confusion of internal with external validity. External validity is external in the sense that we are concerned with applying the results outside the original experimental setting. Internal validity is internal in the sense that we are concerned with the degree to which the circumstances within the original experimental setting warrant the conclusions about the relationship between the independent and dependent variables. A close examination of Figure 15.1 in the textbook should clarify this distinction.
   
   
2. Confusion of history as a threat to internal vs. external validity. If the unique events occur for just to one of the groups in the experiment (for example, if something unusual happens in the presence of the experimental group but not to the control group), these events pose a threat to internal validity. If the unique events occur equally for all of the groups in the experiment, then this is a threat to external validity. If there is only one group (that is, no control group), then the events pose a threat to both internal and external validity.
   
   
3. Confusion of expectancy as a threat to internal vs. external validity. If the expectancy leads the expector (for example, the experimenter or the students) to influence events during the experiment in such a way as to cause one group to perform better than the other group, this is a threat to the internal validity of the experiment. If circumstances are controlled so that expectancy does not cause one group to be treated differently than the others, it would still be possible that expectations about the overall experimental setting could have created an environment that made that setting different from other settings. In this case, there would still be a threat to external validity. For example, a teacher might have positive expectancies about an experimental treatment, and the experimenter might foster similar positive expectations in another teacher with regard to a control condition. If the experimental group performed better on a posttest than the control group, expectancy would not be a threat to internal validity (since both teachers had positive expectations). However, it may still be improper to generalize results to other groups for which the teachers do not have similar positive expectations.
   
   In actual practice, expectancy is likely to be a threat to both internal and external validity almost every time it occurs. Both internal and external validity can be strengthened by conducting experiments in such a way as to minimize artificial expectations among the researchers or subjects in experimental settings.
   
   
4. Confusion of pretesting as a threat to internal vs. external validity. If the pretest is given only to one of the groups (for example, only to the experimental group), then the pretest is a threat to the internal validity of the experiment. If all groups receive the pretest, then there is still a threat to external validity, if the results are to be generalized to persons who have not taken a pretest. If there is no control group, pretesting poses a threat to both internal and external validity.
   
   
5. Confusion regarding the operational definition of independent variables. Readers often find it easy to write or recognize operational definitions of dependent (outcome) variables. This is partly because most educators are already familiar with behavioral objectives, which are specific examples of operational definitions of outcome variables. Readers often mistakenly think that all operational definitions have to look like behavioral objectives. This is not the case.
   
   An operational definition of an independent (treatment) variable states what the experimenter did to implement the experimental treatment (or what the experimenter observed to verify that the experimental conditions existed). In most research articles, the full operational definition of the independent variable is given in the methods section of the report, where the researcher describes what happened to the subjects during the experimental treatment. The following paragraph gives an example.
   
   In the report in Appendix C of the textbook, the authors found that "using computer simulations led to improved thinking skills performance." The detailed operational definition of "using computer simulations" occurs on pages 460-461 of the textbook. A more concise statement of that operational definition would be "using the specified set of programs under the circumstances described in the treatment section of the report." The value of knowing this operational definition is that readers and other researchers can make judgments regarding the likelihood that what these authors say about "using computer simulations" can be applied to other settings. For example, part of the "circumstances described in the treatment section" is that the experimental students took notes and integrated information in a laboratory guide while performing the simulations. It is not obvious that the results would generalize to settings in which a teacher's operational definition of "using computer simulations" consisted of having students run similar simulations without taking integrated notes.
   
   Sometimes operational definitions of independent variables sound weak or even silly. For example, I once showed my students a report that concluded that "students who employ metacognitive thinking skills will display stronger reading comprehension than those who do not do so." I told them that the operational definition of "employing metacognitive thinking skills" was "receiving instructions at the beginning of a reading passage to make predictions and to verify these predictions during subsequent reading." Some students objected. Some pointed out that just telling students to make and to verify predictions does not mean that they will actually do so. Others pointed out that metacognitive skills surely consisted of more than merely making and verifying predictions. My students were right, but so was I. This was the author's operational definition of "employing metacognitive skills." My students were merely pointing out that there were weaknesses in this operational definition.
   
   
6. Knowing what the researcher did to implement the treatment gives us the operational definition of the independent variable. Once we have this operational definition, then we can make comments about its quality and judgments regarding what generalizations can be made from the research.

CROSS-REFERENCES TO OTHER CHAPTERS

Chapter 15 makes reference to the following concepts that are defined and discussed in other chapters. These are listed in the order in which they occurred in Chapter 15.

Internal validity (which is referred to in numerous places throughout this chapter) is the main topic of Chapter 10.

Selection bias (which is a threat to internal validity that is comparable to the external validity threat of the interaction of persons with the treatment) is discussed on pages 223-225.

The characteristics of volunteers (which often create a problem of interaction with persons) are discussed on pages 184 and 185.

Expectancy as a threat to internal validity is discussed on pages 235-237.

History as a threat to internal validity is discussed on pages 221-223.

Experimental design (which is mentioned in this chapter as having relatively little effect on external validity) is discussed in Chapters 11 and 12.

Meta-analysis (which can make an important contribution to external validity) is the main topic of Chapter 16.

Qualitative research (which can make an important contribution to external validity) is the main topic of Chapter 9.

Random selection (which can contribute to external validity, but rarely does so) is discussed on pages 172-173.

Random assignment (which makes a major contribution to internal but not to external validity) is discussed on pages 253-268.

Moderator variables (which can help researchers and readers delineate the extent to which the results of research studies can be generalized) are discussed on pages 24-26.

Interactions (which can help researchers and readers delineate the extent to which the results of research studies can be generalized) are discussed on pages 376-381.

Control variables (which can help researchers and readers determine the limits to be imposed on the extent to which the results of research studies can be generalized) are discussed on pages 26-28.

Basic research (which is often criticized for lacking external validity with regard to applied, practical settings) is discussed on pages 381-384.

Operational definitions (which are introduced on page 345 as important tool for delineating the extent to which the results of research studies can be generalized) are the main topic of Chapter 4.

The concept of multiple operational definitions (or triangulation) is introduced on pages 70-73.

EXAMPLES OF IMPORTANT CONCEPTS IN THIS CHAPTER

Examples of important concepts are given on the following pages:

Interaction of treatment with persons - p. 340.

Interaction of treatment with the setting- p. 340.

Interaction of treatment with historical context - p. 342.

Operational definitions of research variables - p. 346-348.

DEFINING KEY TERMS

The following matching exercises focus on the key terms in this chapter. Instead of using them as matching exercises, you may find it effective to try to define each of the terms. The correct answers can be found by checking the answers to the matching exercise.

MATCHING EXERCISE - KEY TERMS

Listed below are several terms that are employed in discussions of external validity. Match each term with one of the definitions given below.

a. setting.

b. historical context.

c. persons.

d. replication.

e. operational definition.

1.   _____ The observable evidence we are willing to accept as an indication that an outcome, condition, or circumstance exists.
   
   
2.  _____ Redoing a study under different circumstances or with different subjects than was previously the case.
   
   
3.  _____ The subjects to whom a treatment was administered or who were observed in a study or to whom the results of a study will be generalized.
   
   
4.  _____ Events that occur simultaneously with a study or with the setting to which the results of a study will be generalized.
   
   
5. _____ The physical and psychological circumstances that surround a study.

SUPPLEMENTARY ACTIVITIES 

1. Find an example of a study reported in the professional or popular literature which appears to have good internal validity but weak external validity.
   
   
2. Find an example in the popular literature or professional literature in which there is an interaction of the treatment with the persons involved in the study.
   
   
3. Find an example in the popular or professional literature in which there is an example of an interaction of the treatment with the study.
   
   
4. Find an example of a study in the popular or professional literature in which there is an interaction of the treatment with the time of the study.
   
   
5. In each of the above examples, indicate ways in which the experimenter could replicate the experiment in such a way as to improve the external validity of the study.
   
   
6. Find an example of a study in the professional literature in which the researcher has operationally defined the variables in such a way as to enable you to determine the limits of your generalizations which you can draw from the study.
   
   
7. Find an example of a research study in the professional or popular literature in which the generalizations you can make are limited because of the inadequacy of the operational definitions employed by the researcher.
   
   
8. Identify a study from your own professional work where your results have been limited in their generalizability because of weaknesses in the external validity of your design.

Answers to Quiz and Exercises

Review Quiz

1. (c) This is a good summary of external validity. Statement (a) describes internal validity. Statement (b) describes test validity.
   
   
2. (a) Since she plans to generalize the results to non-volunteers, Mrs. Watkins needs to be concerned that the volunteers may be different from the people to whom she plans to generalize her results. While she has demonstrated that her treatment caused results, she is not certain that it will have similar results with non-volunteers.
   
   
3. (c) The major publicity in the newspaper about drug-related arrests that led Mr. Grey to introduce a unit to his eighth graders about drug abuse represents a series of historical events that may not be present when the program is continued in subsequent years. This possibility of historical events interacting with the treatment makes it uncertain that Mr. Grey can generalize his results to other eighth graders.
   
   
4. (b) Since Mrs. Weller developed the program, she probably has a vested interest in seeing it succeed. She is likely to (perhaps unconsciously) have an expectancy that it will be successful. Expectancy is considered to be part of the experimental setting, and so we cannot rule out the possibility that the program's demonstrated success will occur in settings in which Mrs. Weller herself is not present. (Note that this assessment does not mean that Mrs. Weller's efforts were wasted. To the extent that she can demonstrate that she was objective in her implementation of the new methodology, she can have confidence that it will be effective elsewhere. However, it would be wise to replicate the study with someone other than Mrs. Weller implementing the treatment and collecting the data.)
   
   
5. (c) This statement is a good summary of the interaction of the treatment with time as a threat to the external validity of an experiment. Statement (b) is description of history as a threat to internal validity.
   
   
6. (c) Mr. Rostankovsky is concerned with the interaction of the treatment with the unique characteristics of his subjects. Strategies (a) and (c) would both be useful strategies to deal with this threat to external validity. Since the question asks what he should do first, the best answer is (c), since he can carry out that strategy right now. Information he would gain from that analysis would help him both with his immediate interpretation of the results and with his plans for carrying out strategy (a) the next semester. Response (d) would also be a good idea, but with regard to external validity, there is a more urgent need to determine the degree to which the subjects are typical of those to whom the results will be generalized than to increase the number of subjects without concern for the quality of the sample. Responses (b) and (e) deal with a different problem (the interaction of the treatment with the situation); and the anecdote does not say that Mr. Rostankovsky is concerned about this problem.
   
   
7. (e) Responses (b) and (c) both deal with the problem of the interaction of the treatment with the volunteers. A combination of these strategies would be highly desirable. Response (a) deals with the interaction of the treatment with the setting; and the anecdote does not say that Mrs. Gordon is concerned with this problem.
   
   
8. (d) Mrs. Wallace is concerned about the interaction of the pretest with the treatment. Therefore, her solution must deal with this interaction. Solution (d) is the only one that deals with this problem. Solution (a) deals with the interaction of the treatment with the subjects. Solution (b) deals with the interaction of the treatment with the specific Officer Friendly. Solution (c) would help solve the interaction of the treatment with time.
   
   
9. (d) It may seem that Mr. Twain's concern is that his treatment may be interacting with time, and none of these solutions deal specifically with time. However, the television series is the specific aspect of history with which Mr. Twain is concerned. Therefore, replicating the study with students who had not watched the television series would be an effective way to extend the external validity of his study.
   
   
10. (c) Unless it states the observable behavior, events, or characteristics which we are willing to accept as evidence that the variable exists, it would not even be an operational definition.
    
    
11. (c) The description of the kind of computer simulations she used and how did she introduced them are the observable events that enable us to determine what the researcher means by computer simulations. If we plan to use computer simulations ourselves and want to know whether her study is relevant, we can compare her operational definition to our own. Or if we want to replicate her study, we can use this operational definition as the basis of our replication.
    
    
12. (d) "Listening to popular music is the independent variable in an experimental study. Therefore, the operational definition should state the observable activities that the researcher implemented and in which the subjects engaged when the researcher claimed they were "listening to popular music." Response (d) does this.
    
    
13. (c) "Antagonism toward the teacher" is the outcome variable in this study. Therefore, the operational definition should state the observable evidence that the researcher is willing to accept as evidence that the students are feeling antagonism. Response (c) is the best answer. If you feel that "contemptuous" could be further operationally defined, you are right; but response (c) is still the best of the answers listed here. A complete operational definition should include a decision rule describing how the observer should determine whether a sound or remark is contemptuous.

Matching Test

1. e

2. d

3. c

4. b

5. a