The purpose of Chapter 12 is to help you meet the following objectives:

1. Identify the major weakness of the "nondesigns."

(Review question 1)

(Textbook emphasis pp. 270-273)

2. Define and give examples of quasi-experiments.

(Review question 2)

(Textbook emphasis pp. 273-286)

3. Identify examples of each of the following quasi-experimental designs and describe the logic behind each.

(a) Untreated Control Group Design with Pretest and Posttest (Textbook emphasis pp. 273-279)

(b) Nonequivalent Dependent Variables Design (Textbook emphasis pp. 279-280)

(c) Repeated Treatment Design (Textbook emphasis pp. 280-281)

(d) Interrupted Time Series Design (Textbook emphasis pp. 282-284)

(e) Cohort Design (Textbook emphasis pp. 284-285)

(Review questions 3-7)

(Research Design Exercise)

4. Identify the major strengths and weaknesses of each of the above designs.

(Review questions 8-15)

(Textbook emphasis given above.)

5. Apply the above quasi-experimental designs to real life settings.

(Textbook emphasis given above.)

6. Describe the differences among true-experimental, quasi-experimental, and nondesigns with regard to their ability to control the threats to internal validity.

(Textbook emphasis p. 287-288)

7. Describe strategies for combining several quasi-experimental designs to evaluate a single treatment.

(Textbook emphasis pp. 285-286)

(Quasi-Experimental Design Exercise)

REVIEW QUIZ

1. Professor Sanchez gives his students a pretest on their knowledge of contemporary literature. He then provides an eight week unit on the topic. At the end of the eight weeks, Professor Sanchez retests the students and finds that their knowledge of contemporary literature has increased. He concludes that his unit caused this improvement. Which of the following statements is the best evaluation of Professor Sanchez's study?

a. Professor Sanchez has provided absolutely no control over any of the major threats to internal validity.

b. Professor Sanchez has provided some control but he cannot use this study as the basis for establishing cause-and-effect relationships between his unit and student learning.

c. Professor Sanchez can draw cause-and-effect conclusions from his study, but only if he is able to rule out the major threats to internal validity through a nondesign reasoning process.

d. Professor Sanchez's experimental design has successfully ruled out most of the major threats to the internal validity of his study.

2. Which of the following is a major purpose of quasi-experimental design?

a. To enable nonresearchers to perform useful research.

b. To provide as much control as possible over the threats to internal validity in situations when true experiments are not possible.

c. To eliminate the necessity for complex statistical analyses.

d. To facilitate subsequent statistical analyses.

e. To permit statements about cause-and-effect relationships in situations in which the manipulation of subjects is not possible.

Items 3 through 7 each state the reasoning behind one of the major quasi-experimental designs. Examine each statement and determine which experimental design is being described by that statement.

3. "The student improved with regard to the outcome variable I was trying to change, but she remained the same on the other variables I examined. This suggests that my intervention was successful." This is the logic behind the -

a. Untreated control group design with pretest and posttest.

b. Nonequivalent dependent variables design.

c. Repeated treatment design.

d. Interrupted time series design.

e. Cohort design.

4. "The students who completed the course this year are performing better than those who completed the course last year. The students were quite similar in all respects, except that this year's students received a new curriculum. This suggests that the new curriculum causes better performance." This is the logic behind the -

a. Untreated control group design with pretest and posttest.

b. Nonequivalent dependent variables design.

c. Repeated treatment design.

d. Interrupted time series design.

e. Cohort design.

5. "The two groups were similar at the beginning of the semester, but at the end of the semester the group which had received the experimental program performed significantly better. Therefore the experimental program seems to have caused the improvement." This is the logic behind the -

a. Untreated control group design with pretest and posttest.

b. Nonequivalent dependent variables design.

c. Repeated treatment design.

d. Interrupted time series design.

e. Cohort design.

6. "The students have been performing at pretty much the same level on the several previous occasions when they had been measured. However, when the treatment was introduced, their performance improved and stayed at the higher level on subsequent testing occasions. This suggests that the treatment had an impact." This is the logic behind the -

a. Untreated control group design with pretest and posttest.

b. Nonequivalent dependent variables design.

c. Repeated treatment design.

d. Interrupted time series design.

e. Cohort design.

7. "Whenever I introduce the treatment, performance improves. Whenever I remove the treatment, performance gets worse. This strongly suggests that the treatment is causing the improvement." This is the logic behind the -

a. Untreated control group design with pretest and posttest.

b. Nonequivalent dependent variables design.

c. Repeated treatment design.

d. Interrupted time series design.

e. Cohort design.

8. Mr. Webb is an art teacher who is trying to teach his students to recognize certain basic principles of perspective in landscape paintings. He uses a programmed set of materials, but he thinks that a slight modification in the materials will make them more effective. He inserts this change into half the programmed units and then distributes the materials at random to his 100 students. At the end of the unit he discovers that the students who had the modified programmed materials did better on the perspective test than those with the original programmed materials. Mr. Webb concludes that the modification caused the improvements. What research design did Mr. Webb use to evaluate his programmed materials?

a. Posttest-only control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Cohort design.

9. Which of the following is the most serious threat to the internal validity of Mr. Webb's study?

a. Maturation.

b. Statistical regression.

c. Instrumentation.

d. Selection.

e. Expectancy.

10. Miss Waters is teaching basic grammar skills to her fifth graders. She decides that programmed-learning packages are the best way to teach the basic skills. She would prefer to give the packages to individual students who need them rather than lecturing to the whole class on each topic. Her principal is skeptical, however, and insists that she use the old way until she can prove that the new way works. During the second half of the school year, she gives the presentations to the whole class as usual, but in addition, she provides the programmed units on periods to those students who misuse periods in their writing. At the end of the year the students scores are about typical on all skills except the use of periods. The children use periods almost perfectly correctly. She concludes that the new materials really do work, and the principal agrees to let her use the programmed units for all the basic grammar skills. What research design did Miss Waters use?

a. Posttest-only control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Cohort design.

11. Which of the following is the most serious threat to the internal validity of Miss Waters' study?

a. Maturation.

b. Statistical regression.

c. Instrumentation.

d. Selection.

e. Expectancy.

12. Mrs. Curtis gives workshops on child care. The Human Resources Department of a large city hires her to give a workshop to the residents of a low-rent "project." The records for the past several years indicate that in each year there had been a high incidence of child abuse referrals from the project. After Mrs. Curtis gives her workshop, however, the frequency of child abuse drops abruptly. For the next two years, the child abuse rate remains at less than 30% of the previous high rate. The Human Resources Department considers the Curtis workshops to be quite successful. What research design was used to evaluate the Curtis workshops?

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Repeated treatment design.

13. Which of the following is the most serious threat to the internal validity of Mrs. Curtis' study?

a. History.

b. Maturation.

c. Statistical regression.

d. Instrumentation.

e. Pretesting.

14. Mr. Lee is a biology teacher. He wants his students to learn to use the lab equipment without damaging it. He finds that if he puts large posters around the lab and reminds the students of the posters at the beginning of each week, the amount of damage to lab equipment drops almost to zero. On the other hand, whenever Mr. Lee removes the posters, damage to the lab equipment increases again. Mr. Lee decides that since damage goes away whenever the posters are up and occurs whenever they are down, the posters are causing the students to be more careful; and so he decides to leave them up permanently. What research design did Mr. Lee use to evaluate the posters?

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Repeated treatment design.

e. Cohort design.

15. Which of the following is the most serious threat to the internal validity of Mr. Lee's study?

a. History.

b. Statistical regression.

c. Pretesting.

d. Selection.

e. Expectancy.

POSSIBLE PROBLEMS AND SOLUTIONS

1. Confusing the one-group pretest-posttest non-design with the criterion group design (discussed in Chapter 13 on page 293 of the textbook).

The similarity is that in neither case are the subjects randomly assigned to experimental and control conditions.

The difference is that with the criterion group design the researcher groups subjects on the basis of existing characteristics (such as whether students have dropped out of school, post-graduation success on a job, or overcoming an apparent disability). Then the researcher compares these groups on the basis of some possible variables to see if these other variables might have led to being classified ini that group.There was no intention to conduct research until after the subjects dropped out of school, succeeded on a job, or overcame a disability.)

With the one-group pretest-posttest non-design, on the other hand, a decision is made to administer a treatment to a group of subjects that already has been formed or that cannot be compared to another group. For example, the a single class of fifth graders might be pretested at the beginning of the year, then exposed to a treatment, and then posttested at the end of the year. (If this group were compared to another group of fifth graders who received comparable pretests and posttests, then this would be an example of the untreated control group design with pretest and posttest. If the one group of fifth graders were randomly split into two subgroups, with one subgroup receiving the treatment and the other not receiving it, this would be an example of pretest-posttest true experimental design.)

2. Confusing the one-group posttest-only non-design with the criterion group design (discussed in Chapter 13 on page 293 of the textbook).

The main similarity is that there is no pretest.

The difference is that in the non-design a treatment is administered to an identifiable group of subjects (such as a fifth grade class or all the middle-school students in a city).

With the criterion group design the subjects were not even an identifiable group until after the treatment has been administered. With the criterion group design the researcher groups subjects on the basis of existing characteristics (such as whether students have dropped out of school, post-graduation success on a job, or overcoming an apparent disability). Then the researcher compares these groups on the basis of some possible variables to see if these other variables might have led to being classified ini that group.There was no intention to conduct research until after the subjects dropped out of school, succeeded on a job, or overcame a disability.)

3. Confusing the untreated control group design with pretest and posttest with the pretest-posttest true experimental design (discussed on page 263).

The only difference between these two designs is that in the case of the quasi-experimental design (untreated control group design with pretest and posttest) there is no random assignment of subjects to experimental and control conditions. Because of this absence of random assignment, the pretest is essential in the case of the quasi-experimental design; but the pretest is optional in the case of the pretest-posttest true experimental design. That is, the removal of the pretest from the quasi-experimental design leaves us with no reasonable basis of comparison; but the removal of the pretest from the true experimental design leaves us with the posttest-only true experimental design, which still provides a reasonable basis for comparison (because of the random assignment).

4. Confusing the nonequivalent dependent variables design with the untreated control group design.

The similarity is that the diagrams look alike.

The difference is that the diagrams stand for different things. With the untreated control group design, the diagrams represent scores of two groups of subjects on the same dependent variables (e.g., pretest and posttest scores of an experimental group compared to a control group). On the other hand, with the nonequivalent dependent variables design there is only one group of subjects (an experimental group) with scores on two different dependent variables (e.g, pretest and posttest scores for one group on both mathematics and reading tests).

5. Confusing the repeated treatment design with the interrupted time series design.

The main similarity is in the labels given to the designs - the labels could refer to similar designs, but they do not.

In both cases there is only one group of subjects - the experimental group, without a control group.

With the repeated treatment design, a baseline is established by pretesting the subjects before the treatment occurs, then the treatment is administered to all the subjects and the subjects are measured again, then the treatment is withdrawn and the subjects are measured again (to establish a new baseline), and finally the treatment is introduced again (that is, the treatment is repeated) and the subjects are measured again. With the interrupted time series design, a baseline is established by testing the subjects several times, then the treatment is administered (and this interrupts the time series), and the subjects are measured several times again.

6. Confusing the cohort design with the untreated control group design with pretest and posttest.

The similarity is that subjects who received the treatment are compared to subjects who did not receive the treatment.

The difference is that with the cohort design the subjects who received the treatment are different from those to whom they are compared, and the two groups are not even tested at the same time. (For example, fifth graders this year may be compared to last year's fifth graders - an entirely different group of subjects.) With the untreated control group design the performance of the experimental subjects is compared to their own performance on the posttest, as well as to the pretest and posttest performance of another group of subjects, who were pretested and posttested at about the same time, but did not receive the treatment.

RESEARCH DESIGN EXERCISE

1. Mr. Walker is concerned that the school patrol guards have not been carrying out their responsibilities effectively. Each day there are twenty guards on duty. Mr. Walker institutes a program of each day making sure that somebody compliments the guard who is on duty at any designated location. After doing this for one week, Mr. Walker discovers that the guards are performing their duties much more correctly. He continues it for a second week and finds that their performance continues to be good. The next two weeks, however, he discontinues his program and finds that their performance worsens again. Finally, he reinstates his program and finds that the performance improves again. He concludes that the program worked successfully. What research design did Mr. Walker use to evaluate the effectiveness of his program.

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Repeated treatment design.

2. Mr. Long trains pilots for the armed forces. He has developed a new technique he feels will train the pilots to a desired level of effectiveness in a shorter period of time. In the past, it has always taken approximately 16 hours of flight time to train a pilot to perform one of the important skills of flying. With Mr. Long's new method it takes only 14 hours on the average to teach the skill. Since the trainees were similar in most other respects, Mr. Long concludes that his new training procedure is more effective than the old one. What research design did Mr. Long use?

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Cohort design.

3. Miss Philips is teaching a home economics class which includes a unit on sewing. She has developed a programmed unit of instruction to supplement her lecture and the regular practice at the sewing machine. She has two different sections of sewing students. One meets in the morning and the other in the afternoon. Both of them perform about equally on a pretest of sewing ability. She gives her programmed material to the morning group but continues her traditional methods with the afternoon group. At the end of the semester, she gives a test on sewing abilities and discovers that the students who had the programmed materials performed much better. She concludes that her program was successful in teaching the appropriate skills. What research design did Miss Philips use?

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Cohort design.

4. Mr. Cline has been teaching driver education for the past ten years. During this time, an average of about 70% of his students have managed to pass the state driver license examination at the completion of the course. He tries a new strategy which consists of having the students ride with other students and perform critiques of the other students' driving abilities according to a prespecified plan. He discovers that the year he introduces this new method, the number of students passing the state driver license examination increases to 80%. He therefore views his new technique as successful. What research design did Mr. Cline use?

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Repeated treatment design.

e. Cohort design.

5. Each semester Professor Gilbert teaches a course on sports rules. As part of this course, there is a unit on football rules. Part of this unit consists of watching video tapes in which instances of various rules are exemplified. But Professor Gilbert wonders whether these video tapes would be more effective with professional football players or with amateur football players. Therefore, he develops one set of tapes focusing on the rules using amateur players and one set using professional players. He then uses these tapes in alternate semesters. He finds that each semester when he uses the amateur tapes, the students learn the rules more effectively than when they watched the professional tape. He therefore concludes that the amateur tapes caused the students to learn the football rules more effectively than the professional tapes. What research design did Professor Gilbert use?

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Cohort design.

6. Miss Dickinson thinks it would be a good idea for her English students to be able to recognize the authors of many of the famous lines of American poetry. She thinks this can be done most effectively if they listen to recordings of the lines while they read them rather than if they simply read them alone. Her colleague, Mr. Whitman, on the other hand, believes that reading them without listening to the additional tape would enable them to recognize the lines more easily in the future. Since they each teach a class in sophomore English, they decide each of them should try it his or her own way and see which works more effectively. Although the students were approximately the same at the beginning of the semester in their ability to recognize the authorship of a line of poetry, at the end of the semester, Mr. Whitman's students performed much better. Miss Dickinson admitted that he appeared to be right and discontinued her method. Which research design did Miss Dickinson and Mr. Whitman use in his study?

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Cohort design.

7. Mr. Fisher is taking a graduate course in education. As part of the course it is important for him to recognize certain terms so that he can use them in his reading and class discussions. He finds that a good way to do this is to use the matching exercise in the textbook as a drill exercise to provide himself feedback on whether or not he has mastered the terms. He can use this exercise either before he reads the chapter or after he reads the chapter and he wants to know which is the most effective time at which to use this exercise. There is an exercise for terms in the workbook accompanying each chapter. During the second, fourth and sixth weeks Mr. Fisher uses the exercise before reading the chapter. During the third, fifth and seventh weeks, he uses the exercise after reading the chapter. During the eighth week he takes an exam and he discovers that on the exam he misses more of the terms that were studied during the second, fourth, and sixth weeks. He therefore concludes that reading the exercise after reading the chapter is more effective than using it before reading the chapter. What research design did Mr. Fisher use in evaluating these possible study habits?

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Repeated treatment design.

e. Nonequivalent dependent variables design.

8. Mrs. Bruner is concerned that Wanda rarely pays attention in class. She mentions this to the school counselor who tells her that she can discuss this with her at a future interview, but she asks that Mrs. Bruner record Wanda's behavior for two weeks before the conference. During the first week of observation, the teacher discovers that Wanda pays attention only about 10% of the time. During the second week she finds her paying attention only about 12% of the time. During the third week she has a conference with the counselor and develops a plan for helping Wanda to pay attention better. During the third week, again before the treatment was initiated, Wanda pays attention only 9% of the time. At the beginning of the fourth week, Mrs. Bruner implements the new program and finds that Wanda pays attention 25% of the time. During the fourth week, while the procedure is continued, Wanda pays attention nearly 50% of the time. Mrs. Bruner concludes that the new technique is effective. What research design did Mrs. Bruner use?

a. Pretest-posttest control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Repeated treatment design.

9. The goal of the freshman composition class at Gotham State University is to teach the students to organize their ideas effectively. There are two professors of English composition and between them they teach all of the students. One of the professors has a philosophy of very formal education in English composition, and the other has a philosophy of very informal education in English composition. The chairman of the English Department is interested in finding out which of these two philosophies is the most effective. One semester, therefore, he takes all 200 students who are registering for freshman composition and assigns them at random to one of the two professors. The professors are not even informed that such an experiment is going on. At the end of the semester, the students take their regular final examination on English composition. The chairman then compares the performances of the students in the two courses and discovers that the professor who used the more formal approach had students who scored far superior to those who used the informal approach. The chairman therefore concluded that the formal approach was more effective than the informal approach and he communicated these results to the two professors involved. What research design did the chairman use in evaluating the effectiveness of these two philosophies of teaching composition?

a. Posttest-only control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Cohort design.

10. Mrs. Copeland is a school music teacher whose goal is to encourage her high school students to have an appreciation for music. She measures appreciation of music by determining how many students attend an opera presented near the end of the school year. There are 200 students in her various music classes this year. She assigns them at random to the various classes that she teaches and gives each of the classes as much encouragement as possible to become appreciative of opera. At the end of the year, she discovers that nearly 40% voluntarily attended the opera. This is much higher than the 10% which had attended the opera in previous years. She therefore concludes that her present method of encouraging students to appreciate the opera is more successful than the method that she had been using in the previous years. What research design did Mrs. Copeland use?

a. Posttest-only control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Interrupted time series design.

d. Nonequivalent dependent variables design.

e. Cohort design.

11. Professor Smith teaches a graduate education course in which he requires students to read and report on professional journals from the library. Professor Jones maintains that this is a waste of time. He argues that students learn no more and are no more likely to read the journals after they complete the course by being required to read journals as part of a course. Since Professors Smith and Jones each teach parallel sessions of the same graduate education course, they decide to put their theories to a test by randomly assigning the 60 students registered for the course during the spring semester to one or the other of their sessions. At the end of the course, it becomes obvious that the students who took the Jones section actually performed better on the factual exam which was given to both sections than those who took the Smith section of the same course. In a follow-up survey conducted a year later, it was shown that the Jones students were twice as likely to be reading professional journals than were those in the Smith version of the course. Smith reluctantly agreed that Jones had been right and stopped requiring his students to read journals as part of the course. What research design was used in this study?

a. Posttest-only control-group true-experimental design.

b. Untreated control group pretest-posttest design.

c. Nonequivalent dependent variables design.

d. Repeated treatment design.

e. Cohort design.

QUASI-EXPERIMENTAL DESIGN EXERCISE:
INTEGRATING SEVERAL DESIGNS

A teacher implemented a new Science Curriculum, starting on September 1, 2000. The new curriculum was administered to all the seventh graders in the school. There was another school that was similar in most respects, but which did not receive the new curriculum and therefore could serve as a control group. Therefore, we have these two groups:

School I is the Experimental Group.

School II is the Control Group.

The following tests were given in May of 1999, 2000, 2001, and 2002. The tests were given in both schools.

6th grade Science test, Math test

7th grade Science test, Math test

8th grade Science test, Math test

The above tests are directly comparable from year to year. That is, a student's performance one year can be compared to that same student's performance on the same test in successive years to determine whether or not that student improved with regard to that skill. The results of the tests are summarized in Table 12.1.

Using only the information available, design the best possible set of research designs to determine whether or not the new curriculum was actually effective in teaching science. After you have worked out your design or set of designs, compare yours to the answer given in this chapter. There are several possible answers, but it will be useful for you to see if your reasoning matches that incorporated into the book's answer.

Answer

This is actually a combination of several quasi-experimental designs. For example, focusing only on School I and only on the science scores, the study would represent a Cohort Design, which could be diagrammed as shown in Figure 12.2:

 

 

By itself, the information dispolayed in Figure 12.2 is not all that persusavie. However, this Cohort Design can be combined with either the Interrupted Time Series Design or with the Nonequivalent Dependent Variables Design to provide an even better analysis.

Figure 12.3 uses the data from Table 12.1 to demonstrate the results of the Cohort Design combined with the Interrupted Time Series Design. These results show that the scores followed exactly the pattern which would be expected if the Science Curriculum were producing the desired impact.

 

 

Figure 12.4 uses the data from Table 12.1 to demonstrate the results of the Cohort Design combined with the Nonequivalent Dependent Variables Design to provide an even better analysis. These results show that the scores followed exactly the pattern which would be expected if the Science Curriculum were producing the desired impact.

 

Note that Figures 12.2 through 12.4 use only the data from School 1. It is also possible to analyze the data from School 1 and School 2 as a combination of an Interrupted Time Series Design and an Untreated Control Group Design. This combination can be diagrammed as shown in Figure 12.5.

 

 

In addition, it would be possible to combine the Interrupted Time Series Design with both an Untreated Control Group Design and a Nonequivalent Dependent Variable Design. This combination could be diagrammed in Figure 12.6.

As Figure 12.6 shows in this most detailed of the quasi-experiimental analyses, the performance of the students in School 1 and School 2 is parallel as time progresses with regard to mathematics. This is exactly what we would expect, since there was no change in the mathematics curriculum. Likewise, in School 2, progress in mathematics is parallel to progress in science. This also is to be expected, since School II had a new program in neither math or science. However, the science scores of School 1 advance more rapidly than those of School 2 after the initiation of the new program. This is exactly what we would expect if the program produced the desired impact. Likewise, in School 1, performance in science advances more rapidly than performance in math. Again, this is exactly what we would expect if the program produced the desired impact.

Taken alone, the data from any of these separate designs would provide a relatively weak argument. However, when the three quasi-experimental designs are used jointly, as shown in Figure 12.6, they provide very compelling evidence that the new Science Curriculum was effective.

CROSS-REFERENCES TO OTHER CHAPTERS

Chapter 12 makes reference to the following concepts that are defined and discussed in other chapters. These are listed in the order in which they occurred in Chapter 12.

Internal validity and the threats to internal validity (discussed in several places throughout the chapter) are the main topic of Chapter 10.

True experiments (discussed throughout the chapter) are the main topic of Chapter 11.

Selection bias (which is mentioned on pages 270, 274, and elsewhere in the chapter) is discussed in detail on page 223.

History (which is mentioned on pages 271, 275, and elsewhere in the chapter) is discussed in detail on page 221.

Maturation (which is mentioned on page 274) is discussed in detail on page 225.

Instrumentation (which is mentioned on page 275) is discussed in detail on page 226.

Statistical regression (which is discussed on pages 275-276) is discussed in on page 228.

Other threats to internal validity introduced throughout the chapter are discussed in Chapter 10.

Replication (mentioned on page 271) is discussed on pages 344-345.

External validity (mentioned on page 286) is the main topic of Chapter 15.

The pretest-posttest true experimental design (mentioned on page 274) was introduced on page 263.

The interactive effects that threaten internal validity (discussed on pages 274 and 275) were introduced on page 238.

Random assignment (which page 275 describes as an effective way to control the interaction of selection bias with other factors) was discussed in detail in Chapter 10, beginning on page 253.

The characteristics of volunteer subjects (mentioned on page 275) were described in detail on page 184.

The basic logic of experimental design (mentioned on page 277) was discussed in detail on pages 252-254.

Dependent variables (which are at the heart of the research design described on pages 279-280) were discussed on page 23.

EXAMPLES OF IMPORTANT CONCEPTS IN THIS CHAPTER

Sometimes readers want to go directly to examples of topics. Anecdotes or examples of each of these concepts can be found on the following pages of the textbook:

Interaction of selection bias with other factors - pp. 274-278

Nonequivalent dependent variables design - p. 279

Repeated treatment design - p. 280

Interrupted time series design - p. 282-283

Cohort design - p. 284

Combined quasi-experimental designs - p. 287

DEFINING KEY TERMS

The following matching exercise focuses on the key research designs described in this and the previous chapter. Instead of using it as a matching exercise, you may find it effective to try to describe each of the designs. The correct answers can be found by checking the answers to the matching exercise.

MATCHING EXERCISE

Listed below are several labels for experimental and quasi-experimental designs. Match each design with the appropriate description given below.

a. Pretest-posttest control-group true-experimental design.

b. Posttest-only control-group true-experimental design.

c. Untreated control group pretest-posttest design.

d. Interrupted time series design.

e. Repeated treatment design.

f. Nonequivalent dependent variables design.

g. Cohort design.

1. _____ A design in which subjects are randomly assigned to experimental and control groups and then tested after members of the experimental group have received treatment which has been withheld from the control group.
   
   
2.  _____ A design in which the researcher measures the performance of a single group of subjects both with regard to a first outcome variable which is expected to change as the result of a treatment and also with regard to a second outcome variable which is not theoretically expected to change as a result of the treatment.
   
   
3. _____ A design in which the researcher evaluates the effectiveness of a treatment by measuring the performance of separate groups of subjects who come through an institution or program on successive occasions when the treatment is either absent or present.
4.  _____ A design in which subjects are randomly assigned to an experimental and a control group. Both groups receive an initial test, then the experimental group receives a treatment which is withheld from the control group, and finally both groups are again tested after the treatment.
   
   
5.  _____ A design in which a single group of subjects is measured on repeated occasions to see if a pattern of behavior changes abruptly at the point when a treatment is introduced.
   
   
6.  _____ A design in which a nonrandomly-selected control group is compared with an experimental group to see if they differ after a treatment. The groups are not randomly assigned but are determined to be equal on the basis of a test which is administered to both groups prior to the treatment.
   
   
7.  _____ A design in which the effectiveness of a treatment is measured by alternately introducing and removing it and observing if the outcome variable follows a predicted pattern as the treatment is introduced and withdrawn.

SUPPLEMENTARY ACTIVITIES

1. Identify situations in your own life where you have used (or could have used) an untreated control group pretest-posttest design to determine the effectiveness of some policy or program. Determine how effective your use of this design was in this case.
   
   
2. Identify situations in your own professional work where you have used (or could have used) an interrupted time series design to determine the effectiveness of some policy or program. Indicate how effectively you used this design.
   
   
3. Identify situations in your own professional life where you have used (or could have used) a nonequivalent dependent variables design to evaluate the effectiveness of a policy or program. Evaluate how effectively you used this design.
   
   
4. Identify a situation in your own professional life where you have used (or could have used) a repeated treatment design to evaluate the effectiveness of some policy or program. Indicate how effectively you used this design.
   
   
5. Identify examples of situations in which each of the above quasi-experimental designs have been used or could be used in settings which are presented through the popular media.
   
   
6. Identify a situation in which one of the above designs was used and indicate how the internal validity of this experiment could have been enhanced by using this design in combination with at least one other of the above designs.
   
   
7. Make a specific plan regarding how you could use one of the above designs in evaluating the effectiveness of some educational strategy you plan to employ within the next year.
   
   
8. Formulate an intelligent guess about how you think the police department or highway department (or some similar agency) within your community uses quasi-experimental research to evaluate the effectiveness of a policy or program. Interview someone to see how accurate you are in your guess about their use of this design.
   
   
9. Identify a situation in your own career or presented in the media where one of these designs has been misapplied to the evaluation of a policy or program.

Answers to Quiz and Exercise

Review Quiz:

1. (c) Professor Sanchez's job would be much easier if he had used a research design. However, he has at least ruled out selection (by giving the pretest). In addition, he may be able to rule out other threats through logical reasoning. For example, he may conclude that students at this age are unlikely to change as the result of the mere passage of time, and so he may rule out maturation as a threat.
   
   
2. (b) This is a close paraphrase of the purpose of quasi-experimental research design.
   
   
3. (b) This is a close paraphrase of the logic behind the nonequivalent dependent variables research design.
   
   
4. (e) This is a close paraphrase of the logic behind the cohort research design.
   
   
5. (a) This is a close paraphrase of the logic behind the untreated control group design with pretest and posttest.
   
   
6. (d) This is a close paraphrase of the logic behind the interrupted time series research design.
   
   
7. (c) This is a close paraphrase of the logic behind the repeated treatment research design.
   
   
8. (a) By distributing the materials at random, Dr. Webb set up a true experimental design. Even though they continued to be in the same classrooms, the individual students were randomly assigned to treatment and control groups.
   
   
9. (e) Since Dr. Webb developed the innovation himself, it is reasonable to assume that he might have a vested interest in finding it to be successful. He should take some steps to rule out even unintentional bias. For example, if he could distribute the materials in such a way that he himself did not know which students were using his materials and which were using the control materials, then the threat of expectancy would be greatly reduced. The description does not say he took such a precaution.
   
   
10. (d) Miss Waters has measured two outcome (dependent) variables: (1) correct use of periods and (2) correct use of any other punctuation. Her treatment was designed to promote better use of periods - it was not designed to have an impact on other types of punctuation. Since her students improved with regard to the first dependent variable (use of periods) but not the other dependent variable, she can reasonably conclude that her treatment had the desired impact. This is a good example of the use of the nonequivalent dependent variables design.
    
    
11. (e) As in the previous example, Miss Waters has a vested interest in seeing the program succeed. She should take steps to rule out this bias. Perhaps she could have someone else who is skeptical about the use of the new technique use the materials in an experimental setting. If the skeptical teacher would find the same results, this would reduce the likelihood that expectancy was responsible for the improved performance of the students.
    
    
12. (c) This is actually a combination of the interrupted time series design and the cohort design. Mrs. Curtis has used several cohorts over a period of several years to establish a baseline pattern and then to see if the pattern changed when the treatment was introduced.
    
    
13. (a) Of the possibilities listed here, history is the most likely threat. Since each of the cohorts experienced a different history, it would be possible that an event that happened to occur around the time of the introduction of the experimental treatment might have actually been responsible for the changed performance. No historical event is specified in the description, but the likelihood that the other threats listed would cause a problem is much more remote. Another serious threat (not included in the list) would be selection - the students receiving the experimental program might simply be different from those who had received the previous treatment.
    
    
14. (d) Mr. Lee is repeating the treatment. Whenever he provides the treatment, performance improves. Whenever he removes it, performance becomes worse again. This is good evidence that the treatment is having the desired impact.
    
    
15. (e) Since Mr. Lee expects the posters to have a beneficial impact, it is possible that his awareness of the posters and his desire for improvement could have an unintentional impact on the performance of the students. Mr. Lee's problem is not as serious as that of in the questions 8 and 10; however, expectancy is a more serious threat than the others listed. A good way to minimize this threat would be to put the posters in the classroom of another teacher who has no similar expectancies (perhaps a teacher who considers them to be nice decorations) and see if the improvements occur there as well.

Research Design Exercise:

1. (e) Mr. Walker is repeating his treatment. Whenever he employs the treatment, performance improves. Whenever he withdraws it, performance deteriorates. This is a good example of the repeated treatment quasi-experimental design.
   
   
2. (e) Mr. Long's subjects have not been assigned to groups; but rather they are persons who follow other groups of similar persons through the program. This is a good example of the cohort quasi-experimental design.
   
   
3. (b) Miss Phillips did not assign her students at random to experimental and control conditions. Rather, she selected intact groups and assigned the treatment to these groups. It was important that she used a pretest to demonstrate that they were initially similar. This is a good example of the untreated control group pretest-posttest quasi-experimental design.
   
   
4. (e) Mr. Cline has not assigned his students to experimental and control conditions. The students are simply those who naturally follow one another through the program. This is a good example of the cohort quasi-experimental design.
   
   
5. (e) Professor Gilbert has not assigned his students to experimental and control conditions. The students are simply those who naturally follow one another through the program during subsequent semesters. This is a good example of the cohort quasi-experimental design.
   
   
6. (b) Miss Dickinson and Mr. Whitman did not assign their students at random to experimental and control conditions. Rather, they had intact groups and assigned the treatment to these groups. It was important that they used a pretest to demonstrate that they were initially similar. This is a good example of the untreated control group pretest-posttest quasi-experimental design.
   
   
7. (d) Mr. Fisher is repeating the treatment. Whenever he provides the treatment, performance improves. Whenever he removes it, performance becomes worse again. This is good evidence that the treatment is having the desired impact. This is a good example of the repeated treatment quasi-experimental design.
   
   
8. (c) Mrs. Bruner has collected data on several occasions to establish a baseline and then implemented a treatment. The pattern established before the treatment was broken when the treatment was initiated. This is a good example of the interrupted time series quasi-experimental design.
   
   
9. (a) The random assignment to treatments makes this a true experimental design. There was no pretest, and so this is an example of the posttest-only control-group true experimental design.
   
   
10. (e) This is tricky, because the description says that Mrs. Copeland assigned the students at random. Actually, she has put all the current students in the experimental group - the random distribution just decided which of the actual classes each student would be assigned to. (As far as research design was concerned, this random distribution of students was of no significance. If she would have assigned the students at random to the classes and then given some of them the old treatment and some the new treatment, this would have been a true experiment. In other words, if the students had been randomly assigned to experimental and control conditions, this would have been a true experimental design. However, in this example, all of the current students were "randomly assigned" to different classes within the experimental condition.) Since she is comparing current students to previous students in the same program, this is a good example of the cohort design.
    
    
11. (a) Since the students were randomly assigned to experimental and control conditions, this was a true experimental design. There was no pretest, and so this is an example of the posttest-only control-group true experimental design.

Matching Exercise

1. b

2. f

3. g

4. a

5. d

6. c

7. e