Operationally Defining and Measuring Research Variables
The purpose of Chapter 4 is to help you meet the objectives listed below. Examine the objectives, and see if you feel you can adequately meet them. The information in parentheses indicates the review questions, the frames from the programmed unit, and the pages in the textbook which deal most specifically with each objective.
1. Identify examples of well-written operational definitions of educational research variables.(Review questions 1-3)(Frames 1-20 of the Semiprogrammed Review Unit)
(Text emphasis pp. 66-71)
2. Write operational definitions of a given research variable.
(Review questions 1-3)(Frames 1-20 of the Semiprogrammed Review Unit)
(Text emphasis pp. 66-71)
3. Identify strategies for making the evidence acquired through your operational definition as solid or compelling as possible.
(Review question 4)(Frames 21-26 of the Semiprogrammed Review Unit)
(Text emphasis pp. 71-74)
4. Write a correctly stated research prediction.
(Review questions 5-13)(Frames 28-37 of the Semiprogrammed Review Unit)
(Text emphasis pp. 77-80)
1. Miss Wilson is concerned that her testing procedures cause anxiety. She doesn't want this to happen. Which of the following statements is (are) an operational definition of "anxiety"?
a. Anxiety is defined as a painful uneasiness of mind over an impending or an anticipated unpleasant event.b. Anxiety is an undifferentiated fear.
c. Anxiety will be measured by attaching electrodes to the subject and measuring alterations in galvanic skin response.
d. Students will be considered anxious to the extent that they make statements which an impartial observer considers to show anxiety.
e. Student anxiety will be assessed by having them agree or disagree on a five-point scale with a series of statements about their level of anxiety.
2. Doctor Good believes that multimedia technology has the capacity to stimulate interaction with scientific materials in such a way as to enhance motivation and knowledge of scientific processes. Which of the following is the most clearly stated operational definition of multimedia technology?
a. Multimedia technology is the combination of several different media, coordinated by a computer interface, to present a unit of instruction.b. Multimedia technology is defined as the set of strategies that evoke coordinated integration of audio soundtracks and visual images to move students through a unit of instruction.
c. The multimedia technology in this study consisted of the laserdiscs from the BIOSCI II series, delivered via Macintosh LC computers using the 1991 version of the BIOSCI software, with students working in groups of three or four at each workstation.
d. Multimedia technology is the use of several media combined with modern technology.
3. In the preceding scenario, which of the following would be the clearest operational definition of knowledge of scientific processes?
a. The student will demonstrate knowledge of scientific processes by completing an independent research project that integrates at least ten of the principles listed in the course syllabus in a way that meets the standards stated in that syllabus.b. Knowledge of scientific processes is defined as a clear perception of the processes underlying scientific principles rather than a rote memorization of the statement of those principles.
c. The student will demonstrate knowledge of scientific processes by showing that he or she understands the principles behind these processes.
d. Knowledge of scientific processes refers to the cognitive and metacognitive knowledge that enables a student to comprehend the underlying logic of demonstrations in the science course.
4. Which of the following statements describe(s) a recommended strategy for making the evidence obtained through an operational definition as solid as possible?.
a. Use more than one operational definition of a given outcome variable.b. Use both theoretical concepts and concrete terms in your operational definition.
c. Eliminate as many alternate explanations as possible.
d. Use a single operational definition to measure more than one outcome variable.
5. Listed below is a research question. State a research hypothesis based on this question, and then state a research prediction based on the hypothesis.
Question: Does reciprocal teaching lead to a higher level of reading comprehension among elementary school children with reading problems?Operational definitions:
Reciprocal teaching: The strategy described by Palinscar and Brown and demonstrated to the teachers during a inservice workshop and two weeks of training.Reading comprehension: Appropriate answers to factual and inferential questions about reading passages that are appropriate for the children.
Elementary school children: Third and fourth graders in School X.
Reading Problems: Designation as eligible for a Chapter 1 reading program and scoring at least one grade level below expectation on the XX State Reading Achievement Test.
Write the hypothesis here:
Write the research prediction here:
Use the following information for questions 6 through 13:
Miss Barton believes that college students will do just as well in college chemistry if they have not had a high school chemistry class as if they have had such a class. She believes that this is true of both advanced placement and non-advanced placement. She believes that performance in college chemistry courses depends more on an understanding of scientific processes than on initial knowledge of facts about chemistry.
6. What is (are) Miss Barton's independent variable(s)?
_____ college students
_____ performance in a college chemistry class
_____ the teacher of the college chemistry course
_____ having had a high school chemistry course or not having had the course
_____ the sex of the students
_____ understanding of scientific processes
_____ advanced placement vs. non-advanced placement students
_____ (None given)
7. What is (are) Miss Barton's dependent variable(s)?
_____ college students
_____ performance in a college chemistry class
_____ the teacher of the college chemistry course
_____ having had a high school chemistry course or not having had the course
_____ the sex of the students
_____ understanding of scientific processes
_____ advanced placement vs. non-advanced placement students
_____ (None given)
8. What is (are) Miss Barton's moderator variable(s)?
_____ college students
_____ performance in a college chemistry class
_____ the teacher of the college chemistry course
_____ having had a high school chemistry course or not having had the course
_____ the sex of the students
_____ understanding of scientific processes
_____ advanced placement vs. non-advanced placement students
_____ (None given)
9. What is (are) Miss Barton's control variable(s)?
_____ college students
_____ performance in a college chemistry class
_____ the teacher of the college chemistry course
_____ having had a high school chemistry course or not having had the course
_____ the sex of the students
_____ understanding of scientific processes
_____ advanced placement vs. non-advanced placement students
_____ (None given)
10. What is (are) Miss Barton's intervening variable(s)?
_____ college students
_____ performance in a college chemistry class
_____ the teacher of the college chemistry course
_____ having had a high school chemistry course or not having had the course
_____ the sex of the students
_____ understanding of scientific processes
_____ advanced placement vs. non-advanced placement students
_____ (None given)
11. Write a research hypothesis for Miss Barton's research question:
12. Write an operational definition for each of the following research variables:
Performance in a college chemistry course:
High school chemistry course:
College students:
13. Write a research prediction for Miss Barton's research hypothesis, using your operational definitions:
POSSIBLE PROBLEMS AND SOLUTIONS
1. Students often think that an operational definition must be a good operational definition before it can be considered an operational definition. While it is important to develop good operational definitions, in reality many operational definitions are inherently weak. It is important to recognize the operational definitions that form the foundation for an argument or for a research report. It is a separate problem to decide whether these operational definitions (once they have been identified) are good ones or bad ones. The question of quality (validity) of operational definitions is part of the topic of Chapter 5 of the textbook.
2. Students often think that the operational definition has to be stated by the researcher. This is untrue. It is useful for researchers to be aware of and to state clearly the operational definitions of their research variables; but this does not always happen. The de facto operational definition is whatever the researcher does to state a belief that a variable exists or is occurring.
For example, if someone criticizes a teacher for "unprofessional behavior," the critic's operational definition of unprofessional behavior consists of the specific behaviors that the critic cites as examples to support this accusation.If a supervisor accuses an employee of "disloyalty" for stating an objective criticism in a private meeting, then objective criticisms of that type represent the leader's operational definition of disloyalty.
Operational definitions enable us to determine what a person is talking about - even if that person doesn't clearly understand what he or she is talking about!
SEMIPROGRAMMED UNIT ON OPERATIONAL DEFINITIONS
1. A research variable is often invisible (or at least partially invisible) to the researcher trying to observe it. Learning and emotions, for example, occur within a person's brain or neurological system; and educators or researchers cannot directly observe such learning and emotions. Rather, these internal outcomes have to be inferred from observable evidence. An operational definition is the observable evidence which the researcher is willing to accept as an indication that a variable exists or is occurring.
2. The essential characteristic of an operational definition is that it must focus on observable behavior. There are other sorts of definitions which are useful in their own right, but which do not focus on observable behaviors. These other definitions - no matter how excellent and useful they might be - are not operational definitions.
3. Unless you have an operational definition of a research variable, it is impossible to collect data on the extent to which it exists or is occurring. In other words, you cannot do even Level I research without operational definitions. And since Level I research is the basis of all other empirical research, it is difficult to do most forms of empirical research without operational definitions.
4. When instructors lecture to students in Educational Research courses, they try to get them to understand important concepts. In the previous sentence, the instructor's outcome variable is "understanding important concepts." Instructors cannot directly look into their students' minds and ascertain whether or not they understand what the instructors have been saying. Therefore, they need an operational definition, if they wish to determine their students degree of comprehension. Examine each of the following definitions of "understanding important concepts," and determine which ones are operationally stated.
a. Students understand the concept if they grasp its basic meaning. (Go to 5)b. Students understand the concept if they can integrate it with their previous knowledge in related areas. (Go to 5)
c. Students understand the concept if they can give an example of its application when asked to do so. (Go to 6)
d. Students understand the concept if they say they understand it when the instructor asks them whether or not they understood it. (Go to 6)
e. Students understand the concept if they can paraphrase it to the satisfaction of the instructor. (Go to 6)
f. Students understand the concept if they experience no inner dissonance when they think about the concept. (Go to 5)
g. Students understand the concept if they show no external signs of confusion while the instructor is lecturing on the topic. (Go to 6)
5. Wrong. It is not possible for an external observer to determine what goes on within another person's mind. Try again.
6. Right. This is an observable behavior on which an external observer can focus attention. There are four correct operational definitions in 4. When you have found all four, go to 7.
7. Note that even though answers (a), (b), and (f) are not operational definitions, they are still good definitions. In fact, they are probably more important than the operational definitions. By this I mean that the instructor would much rather have a student (a) grasp the basic meaning of what has been said than (g) refrain from looking confused during the lecture. However, even though (a) is more important to the instructor, the fact remains that the instructor can at least observe (g), whereas (a) is something that the instructor has to infer.
8. Definitions (a), (b), and (f) in Frame 4 are examples of conceptual definitions. Such definitions are useful for carrying on conversations with colleagues and for theoretical speculations. They can also be an important part of the research process. But to be useful in empirical research, a conceptual definition must always be backed up by at least one operational definition. Usually, when researchers think of a research variable, they first think of it in terms of a conceptual definition and then derive useful operational definitions based on this initial conceptual definition. Only after the operational definition has been derived can the researcher measure the extent to which the conceptual outcome is present or absent.
9. The fact that (c), (d), (e), and (g) in Frame 4 are operationally stated definitions does not necessarily mean that they are good operational definitions of "understanding important concepts." Whether they are good or not depends on how accurate the inference will be if the researcher uses this operational definition to collect evidence about the conceptual research variable. It is quite possible (but hardly desirable) to have a well-stated operational definition which is logically unrelated to the conceptual variable. Guidelines for deciding whether an operational definition is a good one are part of the discussion of validity in Chapter 5 of the textbook.
10. Mrs. Jones wants to help Johnny become more popular. Which of the following is (are) an operational definition of "popularity"?
a. Johnny will spend less time standing alone at recess. (Go to 11)b. Johnny will considered to be a friend by a greater number of his classmates. (Go to 12)
c. Johnny will be selected by more of his classmates on a sociogram on which they are asked to select their friends. (Go to 11)
d. Johnny's classmates will like him better. (Go to 12)
11. Right. This is a behavior which can be directly observed by the teacher. There are two correct operational definitions. When you have found them both, go to 13.
12. Wrong. How Johnny's friends feel toward him cannot be directly observed by Mrs. Jones. Therefore, this is not an operational definition. (Note: It is possible that you might have read "considered to be a friend" to refer to "being added to a list of friends," which would be an observable behavior. If this was how you read the statement, you have a valid insight. However, it would be better to say "being added to a list..." if that's what you mean.) Review 2 and 3 and then try 10 again.
13. Mr. Dangerfield wants his friends to show him more respect. Which of the following are operational definitions of "respect"?
a. His friends will offer to buy him a drink when he comes into the room. (Go to 14)b. His friends will call him by name when they greet him. (Go to 14)
c. His friends will hold him in high esteem. (Go to 15)
d. His friends will foster a feeling of admiration toward him. (Go to 15)
14. Right. This is a behavior which an impartial observer could easily observe. When you have found both operational definitions, go to 16.
15. Wrong. The feelings of Mr. Dangerfield's friends cannot be directly observed. They can only be inferred from other behaviors. Review 2 and 3, and then try 13 again.
16. Mr. Brown wants his students to be able to apply the principles of mathematics to solving practical problems. Which of the following state(s) an operational definition of Mr. Brown's goal?
a. The students will see the relationship between theoretical concepts and practical applications. (Go to 17)b. Given an applied problem which can be solved by theoretical principles discussed in class, the students will solve the problem. (Go to 18)
c. The students will understand how to look for the common elements in a problem setting which indicate the correct mathematical strategy to employ. (Go to 17)
d. The students will correctly state which mathematical formula is applicable to a given problem setting. (Go to 18)
17. Wrong. This definition describes an internal state of affairs which can only be inferred from other behaviors which are observable. Review 2 and 3, and then try 16 again.
18. Right. This definition is based on a directly observable behavior. When you have found both correct operational definitions, go to 19.
19. Examine the following list, and determine which are operationally stated definitions and which are not.
a. A smart person is one who states that she agrees with my opinions.b. An artist is a person who sees the relationship between the way things are and the way they should be.
c. A creative person is a person who can think of new ways to use old ideas.
d. A creative person is a person who writes down uses for an object which have not previously been stated by other persons for that same object.
e. Happiness is a state of well-being and pleasurable satisfaction.
f. Happiness is getting a gold star on my homework.
Correct answers: a, d, and f. These three definitions state observable behaviors or events which the researcher can use as a basis for determining whether or not the outcome is occurring. The other three are conceptual definitions. These others cannot be directly observed; but rather they have to be inferred from the behaviors of a person.
20. If you got these answers in 19 right, you probably understand the concept of operational definitions. Successful performance on 19 is an operational definition of "understanding operational definitions." If you are satisfied that this is an accurate operational definition in your case, then proceed with the rest of this programmed unit and eventually to the next chapter. If you are still uncertain, seek additional help from the recommended readings in the textbook, from a classmate, or from your instructor.
21. Obviously, many of the definitions stated in 4, 10, 13, and 16 which qualify as operational definitions are not very good definitions. Some concepts are very hard to operationally define. In other words, it is hard to find good evidence about some outcomes. There are two important strategies to help you make your evidence better - to make your operational definitions more solid:
a. Rule out alternate explanations which could be responsible for an observed event or behavior; andb. Use multiple operational definitions.
22. In 6 we said that one of my operational definitions of "understanding important concepts" might be that a student would say "yes" when the instructor asked her if she understood a concept. One of the weaknesses of this operational definition is that "understanding the concept" is only one of several reasons why a person might respond affirmatively to that question. Four plausible explanations for an affirmative answer are listed below:
a. The student might be afraid that the instructor would react with irrational anger or lower her grade (or assign her to do a report) if she says "no."b. The student might be eager to go home, and she might be under the impression that if she says "yes" she will be able to leave sooner.
c. The student herself may be free from the feeling in (b), but she might be aware that her classmates want to leave; and she might want to avoid antagonizing them.
d. The student might actually understand the concept.
The goal of a teacher/researcher is to find logical evidence that (d) is the actual reason for the behavior. the instructor can provide such evidence by ruling out the plausibility of (a), (b), and (c).
23. If the instructor knows from previous experience with the class (and with specific students) that the instructor have good rapport with them, then the instructor have some evidence to rule out (a) as a plausible explanation in 22. Likewise, if the instructor has good reason to believe that the students are not eager to leave, then (b) and (c) can be ruled out as an explanation. For example, if the students know that the class always lasts about the same amount of time regardless of how many questions are asked, then explanations (b) and (c) are unlikely to be responsible for "yes" answers. Even better, if the students have in the past frequently answered "no" to this question under similar circumstances, then this would be rather persuasive evidence that (d), rather than any of the other possibilities, is the correct explanation for affirmative answers. By ruling out the alternate explanations, the instructor can increase the likelihood that the operational definition actually provides evidence related to the conceptual definition.
24. Another good strategy would be to use multiple operational definitions. If the students give examples of the concept and say they understand and don't look confused while the instructor is talking to them, then the evidence gets pretty convincing that they actually do understand the concept. Several diverse pieces of evidence buttress our inferences much more effectively than isolated pieces of evidence.
25. Notice that operational definitions can be extremely useful at both the purely practical and the highly theoretical levels of research. For example, the instructor may have no intention of ever doing formal research in the educational research course regarding whether or not the students understand specific concepts the instructor is teaching. Nevertheless, by clarifying the operational definitions, the instructor can become more adept at ascertaining whether the students have mastered the concepts; and the instructor can change teaching strategies based on observations of this outcome variable. Likewise, at a highly theoretical level, a researcher may be trying to draw conclusions about the relationship between frustration and aggression. By understanding the researcher's operational definitions of these terms, we can make much better use of the results as we think about this issue or apply the conclusions to our own problems.
26. The preceding discussion has focused on the operational definition of outcome (dependent) variables. It is also possible (and important) to operationally define any of the research variables.
27. In a research study, the operational definition of the independent variable is whatever the research did to implement the treatment. For example, is it true or false that "Computer assisted instruction leads to improved performance in thinking skills"? That probably depends on what we mean by computer assisted instruction. Any of the following (and many more) could be operational definitions of computer assisted instruction:
a. Computer assisted instruction is the use of computers to provide drill and practice on activities students have previously learned in class.b. Computer assisted instruction is the use of the computer by the students in any way that enables them to pursue their academic activities.
c. Computer assisted instruction consisted of letting the students individually run the NUMBER MUNCHERS program for fifteen minutes a day while other students pursued other activities.
d. Computer assisted instruction consisted of using the Science 2000 materials in place of the textbook unit of instruction to teach the unit on genetics.
Readers who are familiar with computer assisted instruction might look at some of these and say, "That's not computer assisted instruction, that's...." For the present discussion, discrepancies of that kind don't matter. What does matter is that each of the preceding statements could be an operational definition of computer assisted instruction, if that's what the researcher did to implement the treatment. As was the case with outcome variables, knowing a researcher's operational definition of treatment variables can help us evaluate that research and discuss it intelligently.
28. In the preceding frame, (c) and (d) were better operational definitions than (a) and (b). This is because they state exactly what it was that the researcher was doing to make him/her think that this was an example of computer assisted instruction.
29. Examine the following statements and determine which are operational definitions of cooperative learning.
a. Cooperative learning consisted of having the children work in groups of four, with specific responsibilities (listed on a separate page) assigned to each child. (Go to 30.)b. Cooperative learning refers to a situation in which students are freed from the pressures of competitive learning and are able to work toward common goals with the support of fellow students. (Go to 31.)
c. Cooperative learning consists of students experiencing feelings of group interdependence while still maintaining individual responsibility for their work. (Go to 31.)
d. Cooperative learning was defined as having students work on their term papers in groups of three or four. A single grade was assigned to each project, and each member of the group received that grade. Other than assigning the students to the groups, the teacher gave no further directions that differed from the ordinary way term papers had previously been completed. (Go to 30.)
30. Right. These sentences describe what the teacher or researcher was doing to provide the basis for the claim that he/she was implementing cooperative learning. There are two right answers. When you have found them both, go to 32.
31. Wrong. In a legitimate. conceptual sense this may be what cooperative learning is all about, but these sentences do not tell what the teacher or researcher was doing to provide the basis for the claim that he/she was implementing cooperative learning. Reread 27 and 28 and try 29 again.
32. Note that not all operational definitions of independent (treatment) variables are good operational definitions. For example, when you looked at 29d, you might have said, "That's a stupid way to implement cooperative learning. It's necessary to assign roles to each of the students or at least to find a way to ensure the active participation of all the members."
33. Also note that even the correct answers in Frame 29 could have been more detailed. They probably left you wondering, "But what does this really mean." In a research study, a major part of the methods section describes the treatment. This information comprises a detailed operational definition of the independent variable.
34. Control variables can and should also be operationally defined. Sometimes these operational definitions are obvious. For example, it's easy to understand what's meant by boys and girls. However, if a study was conducted "among low-income, at-risk, learning-disabled fourth graders," it would be useful to operationally define low-income, at-risk, and learning-disabled. To give an operational definition of each of these terms, we would simply state the decision rule by which we classified persons as belonging to that category. (The operational definition of fourth graders seems pretty obvious. If there would be any reason to believe this was ambiguous, then it should be operationally defined.) When you try to write operational definitions of some potential control variables, you will discover that it is easy to be ambiguous.
35. Since moderator and control variables are conceptually similar, everything said in Frame 34 about control variables also applies to moderator variables.
36. Intervening variables are not operationally defined. By their very nature, intervening variables are abstract (conceptual).
37. The research prediction inserts the operational definitions into the statement of the hypothesis. For example, consider the following hypothesis:
Among low-income children, peer tutoring during computer-assisted instruction will lead to improved reading performance.The following are the research variables with their operational definitions:
Low income children: third graders in a specified elementary school in a neighborhood where 90% of the children are eligible for free lunches according to state guidelines.Peer tutoring during computer-assisted instruction: Students will run a specified set of computer programs in pairs. One child in each pair will have received previous instruction regarding error detection and correction and will employ that strategy whenever either member of the pair is informed by the computer that an error has occurred. (The full report would describe this process more completely.)
Reading comprehension: Scores on a standardized, statewide competency test administered to all third graders in May of each year.
Here is the research prediction:
Among third graders in a specified elementary school in a neighborhood where 90% of the children are eligible for free lunches according to state guidelines, students who run a specified set of computer programs in pairs in which one member will employ a specified error detection and correction strategy whenever either member is informed by the computer that an error has occurred will have higher reading scores on the standardized reading comprehension test than similar students who do not take part in this peer tutoring program.Once this research prediction has been stated, it is extremely obvious what needs to be done to test the hypothesis and how far the results can be generalized. In this example, the researcher needs to administer the operationally defined treatment to a group of third graders in a specified elementary school in a neighborhood where 90% of the children are eligible for free lunches according to state guidelines and then see if their scores improve on the statewide reading comprehension test compared to a comparable group of students who do not receive this treatment.
CROSS-REFERENCES TO OTHER CHAPTERS
Chapter 4 makes reference to the following concepts that are defined and discussed in other chapters. These are listed in the order in which they occurred in Chapter 4.Research variables (discussed throughout the chapter) are the main topic of Chapter 2.
Research hypotheses (which are the basis of research predictions, discussed on page 77) are discussed in Chapter 2.
Unobtrusive measurement (introduced on page 70) is further discussed on page 142.
Outcome variables (introduced on page 70) are covered greater detail on pages 101-108.
Experimental research (mentioned on page 77) is covered in Chapter 11.
The methods section of a research report (mentioned on pages 78 and 79) is discussed on pages 436-437.
Research design (introduced on page 79) is one of the main topics of Chapters 11 and 12.
Statistical inference (mentioned on page 79) is covered in Chapter 14.
External validity (mentioned on page 79) is the main topic of Chapter 15.
EXAMPLES OF IMPORTANT CONCEPTS IN THIS CHAPTER
Sometimes readers want to go directly to examples of topics. Anecdotes
or examples of each of these concepts can be found on the following
pages of the textbook:
Dependent variables - pp. 68, 77, 81Operational definitions - pp. 71, 72, 73, 76, 80
Multiple operational definitions - pp. 72, 73, 80
Hypotheses - pp. 77, 78, 80, 81
Research predictions - pp. 77, 78, 80, 81
Independent variables - pp. 77, 81
Moderator variables - pp. 78, 81
Control variables - pp. 78, 81
Intervening variables - p. 81
The following matching exercise focuses on the key terms in this chapter. Instead of using this as a matching exercise, you may find it effective to try to define each of the terms. The correct answers can be found by checking the answers to the matching exercise.
MATCHING EXERCISE - KEY TERMS
Match each of the descriptions below with the corresponding type of research.
a. Conceptual definitionb. Operational definition
c. Research prediction