Choosing the Right Quantitative Research Design

Welcome to the tutorial on quantitative research design. This is Dr. Zapkew, and we're going to spend the next 20 to 30 minutes talking about quantitative research designs. Just like any great builder starts out with a blueprint to guide and direct the construction of a building, a researcher needs to start out with a research design to really guide decisions that need to be made about conducting the research, about making plans. A research design does three things. It says when and how often data is collected, what data needs to be gathered and from whom, and how to analyze the data. In this tutorial we're going to talk about several different quantitative research designs. We're going to discuss when they're used and why they should be chosen. And ultimately the purpose of this tutorial is to help you identify the most appropriate research design for your research study and to help you justify why it's the best choice. Now, before we move on and start Talking about research designs, I want to make a note. Some of you, for your research plans, will actually choose multiple research designs, and this is called a mixed design study. For example, you may choose a causal comparative and a correlational design. This is different than a mixed method approach. A mixed method approach is a full quantitative and full qualitative design. It really requires that you conduct a full-blown quantitative study and a full-blown qualitative study. In essence, you're committing to conduct two separate studies. But here we're only going to talk about quantitative research design with the assumption that you've chosen a quantitative design for your study. Now, there are numerous quantitative research designs, too numerous to really cover here. So we're going to focus our attention primarily on the major designs proposed by Campbell and Stanley in 1963 and Krall in 1993. They say that there are three primary categories of studies. There's descriptive studies, and descriptive studies are done when a researcher wants to know what is. Let's take Charlie, for example. Charlie is a graduate student, and he's interested in college students who have anxiety disorders, because let's say that his cousin has an anxiety disorder. If he simply wants to know how many students who enter college in, let's say, 2012. have an anxiety disorder, a clinically diagnosed anxiety disorder, he may do a descriptive study. He simply wants to know what is, what exists. The next category are correlational studies, and these can be relationship-focused studies or predictive-focused studies. In a correlational study, you really want to know, is there a relationship between two different variables, or does a group of variables or one variable predict another variable? In our example of Charlie, Charlie may want to know, is there a relationship between achievement in a freshman English class and the presence of an anxiety disorder? He wants to know simply, is there a relationship, not if one's causing the other, but is there a relationship between the two? He may also ask, does the presence of an anxiety disorder predict achievement in a college freshman English class? The final category is group comparison, and this usually involves manipulation or treatment of some type. And there are three types. There's a pre-experimental, quasi-experimental, and true experimental. And we'll define those in a few minutes. But really here, let's go back to Charlie. If he wants to do a group comparison study, he's interested in is there a difference in two different groups, or does a specific treatment affect something else? So... for example let's say he finds out that there is a relationship between the presence of an anxiety disorder and academic achievement in a college english class and he finds that the presence of a disorder actually decreases academic achievement so he wants to do an intervention uh... maybe he uh... astor volunteers on who have generalized anxiety disorder he provides one group with a special treatment that helps them lower their anxiety during the English class and he doesn't provide the other group with the treatment and then he compares the two at the end to determine does the treatment ultimately affect academic achievement in the college English class. He's doing a group comparison. We're going to take a more in-depth look at each of these specific research designs but before we do I want to identify or talk about a Few additional designs that some of you may choose to do. In addition to the designs that I just overviewed, some of you may choose to do an instrument development, a content or document analysis. a meta-analysis or a single subject design. Let me talk a little bit about three of those. First of all, some of you may choose to do an instrument development. Oftentimes, when you go to conduct quantitative research, you find that the instrument that you were hoping to find doesn't exist and an instrument needs to be developed. And so you decide to develop an instrument and conduct statistical analysis to determine its validity and reliability. This is a very, very rigorous and intense process. intense type of design. Some of you may choose to do some type of document or content analysis. That is, analyzing some type of communication. And so something that I see often is that students will analyze a discussion forum to determine the presence of, let's say, community in a discussion forum in an online class. Another example of this. maybe looking at lesson plans, analyzing lesson plans for a specific variable. The last is a single subject design. For those of you that are really interested in a specific counseling group or a special education group, and you have a small sample size, you may want to consider this design. It's a really valid research design. And it's similar to a time series design, which we'll talk about, in which each participant really serves as their own control and they're observed repeatedly. Usually what happens is, let's say, for example, we're looking at children with autism and we want to teach them a basic skill such as hand washing. What we would do is you would baseline them. So you would establish a baseline for each participant. How many steps? of the hand washing procedure can they do. You would then, at different times, implement a treatment that maybe teaches them hand washing, and then you would observe the behavior after the treatment. And this is called a single subject design. If you're really interested in doing this design, Wasson 2003 has a great book that I highly, highly recommend. But for the purpose of this tutorial, we're going to go back and focus on the main designs that Campbell and Stanley talk about. So let's go ahead and focus on those. The first type of study we're going to look at is a descriptive study. And descriptive studies focus on what is. They really aim to understand what is happening, what is in a specific situation with an identified population. And they're usually used to gain knowledge to identify a problem. problem or identify what is so that further research can be conducted. It's important to note here that oftentimes descriptive studies are not very rigorous in nature, so a lot of universities don't allow basic descriptive studies for things such as thesis and dissertations, and that's the case at Liberty. We really don't allow them without special permission. But it is important that you know and you understand what a descriptive study is. If I was a researcher and I wanted to do a descriptive study, let's say I response to intervention is fairly new term. It's a new type of intervention in the school systems. And I simply want to know what is the attitude of school counselors about the use of response to intervention. I would do a descriptive study. Another descriptive study may be what is the what responsibilities do school counselors even have in RTI as it's implemented in their school. So just general I want to know what are school counselors doing to help to help implement RTI in the school I would do a descriptive study now most research text identify two different types of descriptive studies a survey study or and an observational study a survey study uses some type of written document or online questionnaire or interview on to gather information about what is and there's usually two basic types of survey studies, and that's longitudinal designs and cross-sectional designs. Longitudinal designs study a population over a period of time. They oftentimes are referred to as trend studies or cohort studies or panel studies. So maybe, going back to our example of Charlie, maybe he wants to study the academic achievement of students identified with anxiety disorder in their freshman year of college. their junior year of college and their senior year of college. He's studying them over a period of time. A cross-sectional study then just looks at individuals at one point in time. So again, going back to our example of Charlie, Charlie simply wants to know what is the academic achievement of freshman college students with an anxiety disorder. So he just was looking at one specific point in time about what is. Oftentimes, surveys collect data on attitudes and beliefs. Then, like I said, the other type of design is the observational design. And this is the process of observing something that's happening and making a conclusion about it. So for example, I may go into a school system and observe teachers or observe school counselors. Again, just as a reminder, descriptive studies oftentimes are not rigorous enough for dissertations and theses. And that's the case at Liberty. We oftentimes don't allow descriptive designs for dissertation purposes. However, descriptive studies can be very important as you build a case for a thesis or a dissertation. Thank you. because they're describing what the problem is. Now let's look at another type of study. The next design we're going to talk about is one that I didn't mention above, but it's similar to a group comparison study, and that's a causal comparative study. The aim of a causal comparative study is really to examine the possible cause and effect relationship between variables that exist. It really, you're studying a phenomenon after the fact. That is, it's already occurred either naturally or it's been manipulated. Let's consider some examples. If you want to examine gender or ethnicity as an independent variable, you have a causal comparative study. It's something that you as the researcher have not manipulated. Gender occurs naturally. Ethnicity occurs naturally. If you as a researcher wanted to examine individuals that smoked versus individuals that didn't, didn't smoke. That has already happened. They've already chosen that behavior. Another example would be an intervention that it's naturally occurring within a school system and is not occurring in another school system. You want to study something that's already been implemented. Oftentimes, causal comparative studies are used in educational research because it's too difficult, too unethical, or impossible to manipulate the independent variable. So, for example, it's unethical for me to tell one group of individuals, let's say one group of students, to smoke and another group not to smoke. Maybe there's a specific intervention, a school-wide intervention happening at a school. And you can't ask the whole school to implement this new intervention just for your research study. That would be too difficult. And since it's naturally occurring, you can go ahead and study it. Now, it's important to note that a causal comparative study is almost identical to a pre-experimental design. And we'll talk about in a little bit that that's not a credible design. But the difference is that oftentimes a causal comparative study has a control group. Actually, it always has a control group. So what you're looking at with a causal comparative design is you're looking at a phenomenon or something that's happened, and you're comparing it with something else. So, for example, a school has this dynamic program that they're implementing, and you're going to compare that school's... student's achievement level to another school that's not using that specific treatment. When we talk about threats to validity, we'll talk about all the controls that need to be in place, because again, there's lots of threats to validity, but this is a valid type of study. I like to give you an example of a... question that may be used with a causal comparative study. For example, I may say, is there a difference in male and female university students' social presence while participating in an eight-week online class? I simply want to know, is there a difference in their social presence between males and females? And again, I can't manipulate it. It's unethical and impossible for me to manipulate gender. And so I would choose a causal comparative study. As we're talking about causal comparative studies, there's a few additional notes that I want to make about this specific design. And especially take note of this if you're planning to use this type of design. Note that like a descriptive study, a causal comparative design does not require the researcher to exert control or manipulate the phenomenon. However, unlike a descriptive study, you are looking at the possible cause and effect relationship between variables, and therefore you do. You have an independent variable with at least two levels. at least two groups that you're comparing, and you have a dependent variable. You're looking at the possible effect that that independent variable has on the dependent variable. And note that I said possible cause and effect. This is really, really important language if you choose a causal comparative design. You cannot prove, you cannot state that you have a cause and effect relationship if you have a causal comparative design, because there are too many threats to validity. So make sure. If you choose a causal comparative design, you're using correct terminology. And just as a reminder, when you choose a causal comparative design, you do so because it's too difficult, too unethical, or impossible for you as the researcher to manipulate the independent variable. Also note that oftentimes causal comparative designs are chosen because you want to explore a phenomenon that's happening. If a lot of causal comparative research has been done in a specific area, and you're finding a lot of causal comparative literature research in that specific area you're studying, it probably means that we need to start moving more toward more rigorous research, such as an experimental design, which we're going to talk about next. So that's the causal comparative design. Actually, before we talk about experimental research, we're going to talk about the correlational research designs. A correlational design, like a causal comparative study, is used for exploratory or beginning research to determine if there is truly more rigorous research that's warranted. The purpose of a correlational study is simply to examine if a relationship exists between variables. Is there a relationship between variables? like we said in the case of Charlie, is there a relationship between the presence of an anxiety disorder, so whether or not somebody has an anxiety disorder, and their academic achievement or their grade in a freshman college class. Other examples of correlational studies may be, is there a relationship between high school GPA and college board SAT scores? We know in the research that some research says there is and some research says there isn't. If you choose to do a correlational design, it's important that you recognize that you cannot speak in terms of cause and effect. You simply have two variables of interest. One does not cause the other because you don't know the direction of the cause and effect. You can't guarantee that one variable caused another. All you know is that they are varying together. Either, let's say we're looking at two scores such as GPA and SAT. We just want to know, do they increase and do they decrease together? Also, you may be looking at predictive. I remember I said at the beginning there are two basic types of correlational designs. There's ones that look at relationship, and thus far everything I've described has been in terms of relationship. There's also predictive studies. If you want to know do SAT scores predict college SAT scores, that you can look at what does one variable predict another variable now it's important to note and specially in talking about prediction that you have a solid theoretical or conceptual framework in which you're basing that study oftentimes with with predictive studies there's been multiple studies that has shown that there's relationship between certain variables or multiple variables and you wanna add to that conceptual or that model or that theoretical model. Let me give you an example of what I mean by that. We know that there's been a lot of research on high school GPA and SAT scores, performance in college, and we've seen that not only does high school GPA predict SAT scores, but also it predicts the performance of high school students. gender and ethnicity and sometimes there's an interaction between gender and ethnicity and social economic status and high school GPA and the results of an SAT score. So let's say in the literature I find a strong conceptual base or a model that says gender and ethnicity and social economic status and GPA predict SAT scores. Let's say that that model predicts SAT scores predicts, let's say, 60% of the SAT score. That means there's still 40% of the model that's unexplained. And maybe I have another framework or another theory that says parents' parental or parental attendance at college is another important piece of the puzzle. So then I'm going to test that. So it's important in correlational designs that you really have a strong theoretical or conceptual rationale for every variable that you plan to study, not just ones that you think, but there needs to be research and theory that backs it up. Again, I want to reiterate, if you choose a correlational study, if you want to look at the relationship between two variables or multiple variables, note that you're you can only make statements about relationship, and you use this design because there's very little research in the area, and it's exploratory. You're only beginning research in the area, or the models that have been presented, specifically predictive models that have been presented, are not complete. There's still parts of it that are unexplained. Now we're going to focus our attention on experimental designs. Campbell and Stanley in 1963 purported that there are three... primary experimental designs. And that's the pre-experimental design, the true experimental design, and the quasi-experimental design. And the distinguishing factor or distinguishing characteristic of all experimental designs is that the researcher manipulates the independent variable. That is, the researcher actually conducts the treatment, does something to the participants. So let's look at these three types of designs. As we discuss these three types of designs, there are commonly used symbols in research text. And actually Campbell and Stanley in 1963 are the ones that proposed these symbols. But anyway, there are these symbols in research text that help us visually see the designs that we're talking about. And so we're going to use those symbols as we talk about these different designs over the next few slides. And the X. Simply means that exposure to treatment. That means there's been a treatment done. O represents observation or the measurement, the dependent variable, really, what you're measuring. Multiple roles reflect multiple groups. And R reflects randomization. And I'll define these more in a little bit. We're going to start by discussing pre-experimental designs. And the primary characteristic of a pre-experimental design is that the... Researcher manipulated the independent variable. Oftentimes, pre-experimental designs are used as preliminary research or pilot studies to determine the effectiveness of a treatment to see if there's more research that's warranted. Campbell and Stanley really say that pre-experimental designs are weak and have little to no value. It's important to note that pre-experimental designs are distinguished as separate from quasi-experimental designs in texts such as Campbell and Stanley, but not so much in Borg, Borg and Gall. So some research texts don't even acknowledge pre-experimental designs or they wrap them into quasi-experimental designs or the quasi-experimental design discussion. I bring up pre-experimental designs here because I want you to be able to recognize them and I want you to recognize that they are not rigorous studies. They, like Campbell and Stanley said, they have no value. So for a dissertation or a thesis they really shouldn't be done. They are, however, often used in program evaluation for school systems and can be very effective for that purpose. An example of a question that may be asked for pre-experimental design is simply, Does parent scores on a parent skill assessment increase after participation in a successful parenting program? Let's look at what a design, that would be probably a one-shot case study. Let's look at what that looks like in the next slide. Here are commonly used pre-experimental designs. As I just mentioned, the question I just proposed, you know, do parents increase in their skills or do parents have better parenting skills after a parenting intervention. That's an example of a one-shot case study. Parents are given a parenting skill intervention, and then their parenting skills are measured. Now, you probably already see the weakness in this study. I'll give you a moment to think about it. Do you have it? The problem with this is I can't say it was the intervention that caused the parenting skills. Perhaps parents already had the skills. Perhaps they were watching a... a parenting program and that caused the score they got in the parent on the parenting skill let's say survey or measurement that i used um so i really can't say it was the intervention that caused caused the effect on the pair or caused the parenting skills additional pre-experimental designs are um the one group pre-test post-test again um here i give parents Here, going back to our parenting example, I would give parents a pretest to measure their parenting skills. I'd do the intervention, and then I'd give them the post-test. Now, this may be a stronger design, because I can see if there is an increase or decrease in parenting skills. But it might have been the pretest that caused the increase on the post-test. So again, I can't say it was the parenting skill class that actually caused. the intervention or caused the increase or decrease in the parenting skills. Same, again, similar weakness in the post-test only non-equivalent group design. If I have two groups and I give one parenting skills and don't give it to the other and then measure them at the end, it may be simple group differences that were the cause, the differences in the parenting skills and not actually the intervention. So, these are pre-experimental designs. Let me give you one more example and see if you can identify the design. Let's go back to our case of Charlie. Let's say Charlie has a group of students, freshman college students, who have an anxiety disorder. And he gives them an anxiety intervention and it helps them reduce their anxiety about their academics or in their classes. And then he post-tests to see if there is a decrease in their anxiety. Again, the problem is if he has one group, he does the intervention, he measures them, it may not be the treatment that actually caused the effect, the lowering, let's say, of the anxiety skills. So again, pre-experimental designs, it's important to know what they are. They can be very effective in program evaluation, sort of exploring, is it even worth going on and... examining a treatment further. However, for a dissertation, they are not acceptable designs. As you can see, there's major threats to validity, and you can't talk about cause and effect relationships. So let's look at two experimental designs that are often used and are some of the strongest designs that can be used for a dissertation. Let's first talk about quasi-experimental designs. Now, the quasi-experimental design is... often used in educational research because it's more convenient and less disruptive than the stronger true experimental design. The purpose of a quasi-experimental design is to test the effectiveness of an intervention or treatment with a target population. It really allows the researcher to control the treatments and look to see if the intervention had an effect on the identified dependent variable. Again, this is the most rigorous design aside from the true experimental design and often chosen in educational research because it's more convenient and less disruptive. Let's look at an example of that. Let's say that you wanted to look at problem-based pedagogy for teaching math. And so you ask the question, what effect does participation in a math lesson developed using a problem-based pedagogy have on teaching math? problem-based pedagogy have on second grade students' math achievement scores when compared to participation in a math lesson that was developed using traditional pedagogy. And you have Mrs. Smith's class and Mrs. Jones's class. And Mrs. Smith, you train to use the problem-based pedagogy approach. And Mrs. Jones, she continues using the traditional approach. And your specific target population is second grade students. Now, it would be really nice, we're going to talk about random assignment in a moment, it would be really nice to randomly assign students to either Mrs. Smith's class or Mrs. Jones's class. However, that's probably not possible in an educational situation. You can't just go randomly assigning students to classes. Therefore, you use Mrs. Smith's class that's already been put together and Mrs. Jones's class. One gets the treatment, one doesn't, and you compare their math achievement scores at the end. You used already composed classes, and so that's a quasi-experimental design. Let's look at, more specifically, let's look at specific types of quasi-experimental designs. If you choose to conduct a quasi-experimental design, you may use one of these specific designs. Probably most prominent is the one that I just explained. And, That's the non-equivalent pre-test, post-test control group design. And that's the middle one here, represented with our little X's and O's. What you do is you pre-test. So you give both, let's say you gave Mrs. Smith's class and Mrs. Jones' class a pre-test. You want to find out what their math achievement was prior to the intervention. You then give Mrs. Smith's class the intervention. You don't give it to Mrs. Jones' class. And then you post-test them. This way, you can actually see if the groups were equivalent in their math achievement prior to the intervention. So you know if it was the intervention or not that caused the... post-test scores. An interrupted time series design is very similar to that where you you create a baseline so you would actually give let's say multiple math achievement tests prior to the intervention you give one group the intervention and then you do multiple post-tests. Another group another design is the counterbalance design and this one can be very effective at looking at when you want to do multiple treatments and you give each group a different regimen of the treatment and then compare them so you can see what the effect is. Now this is a great design one of the weaknesses as you can see would be you you're adding multiple or you're adding more groups and again sometimes in educational research it can be difficult when you have multiple groups because each group needs to be fairly large. When we talk about different When we talk about statistical convention and power, when we talk about an analysis, you'll see that you probably need anywhere between 30 and 60 individuals per group. Again, just to recap, a quasi-experimental design is often used in educational research. It includes both, it has two primary characteristics, and that's manipulation. You as the researcher actually cause the treatment. You manipulate the independent variable. And there's always a control group, a comparison group. Now, talking about control comparison group, let me stop here for a moment and talk about that. There are different types of control groups, and you need to decide what type of control group you are going to use. This Kazdin 2003 book has a great discussion on different types of control groups you have. Most of the time in... In educational research, you're going to use a no treatment control group of some sort or a nonspecific treatment group in which one group gets an intervention and the other group is the same in every single aspect, in every single way, except for that intervention. And let me also say this. When you're thinking about that control group, it's important that The element that you're studying is the only thing different. So, for example, I can't compare Mrs. Jones' math class to Mrs. Smith's English class, or compare a sixth grade math class to a fifth grade math class. Everything about the intervention needs to be the same. So the content of the material, everything needs to be very, very similar except for the intervention. But anyway, when you're talking about control groups or a group that's there as a comparison group, it's important to identify that there are different types of groups. And again, I highly, highly recommend Kasdan as you talk about this. And he talks about factors that you really need to consider when choosing a control, and that's the intent of your research, previous research that's been done, as well as ethical and practical considerations. Let me go back and just remind you, quasi-experimental design has two characteristics, manipulation and control. It is often used in educational research because it's too impossible or not ethical to do the more rigorous true experimental design, which we're going to talk about next. And really the aim of a quasi-experimental design is to determine the effectiveness of some type of intervention. If you're thinking about a causal comparative study and you have the opportunity to do a quasi-experimental study, I highly, highly recommend you consider a quasi-experimental study. It's a much more rigorous, it's a better research design. There's less threats to internal validity. The last design we're going to talk about is the true experimental design. this design is truly the most rigorous design and ultimately the design that you want to choose if you have the opportunity to do so. The purpose of this design is to make sure that you is to examine the cause and effect relationship between variables. You really investigate the possible cause and effect relationship by exposing one or more experimental groups to some type of treatment that you as the researcher do, and you compare those results to one or more control groups not receiving the treatment. Again, this is considered the most rigorous design. Examples of questions that you may ask if you're doing a true experimental design is, is there a difference in students' sense of community based on the type of technology they're using in a class? So, for example, let's say I have two classes. One class does all of their discussion via Blackboard discussion. Another class does all of their discussion via Skype. And I want to compare, does community differ among the students? So everything in the classes are exactly the same except the technology that's used for the discussion. And obviously there's theoretical and conceptual reasons for why you would compare those two things. And let's say there are three characteristics of a true experimental design. So I have already talked about manipulation. So I, as the researcher, am manipulating the independent variable. And I just talked about that. I'm giving one group a discussion using discussion board. And I'm giving another group a discussion. I'm going to do it via Skype or via some type of online conferencing system. There's a control group. So there's a comparison group. And the last thing that I... that characterizes or makes a true experimental design unique is the ability to randomize. So in this specific scenario, I would ask for a group of volunteers, and then I would randomly assign them to either the treatment group or the control group. Now, the important thing to recognize about randomization is this. Campbell and Stanley in 1963 say, that if you randomize, you can assume group equivalence. You can assume a group equivalence. And so you can assume the groups are starting out the same, and so it's actually the treatment that caused the effect. As we've done with the other categories or other types of designs, let's look at specific types of true experimental designs. One of the most rigorous designs is the post-test. only design. In this case, you randomize your groups. You do random assignment. Notice I said random assignment, not random selection. Random assignment, so I have my group of volunteers. In the example I gave, I used college students, let's say. I have a group of them, and then I randomly assign one to treatment, one to non-treatment. I do the treatment, and then I post-test both of them. There's also a pre-test, post-test equivalent group design in which is exactly what I just explained. However, you randomize them, you do a pre-test, you give one group the experiment or the treatment and then you, not the other group, and then you post-test them. Now Campbell and Stanley say that this design introduces a threat to validity and that's the testing threat to validity, so it's preferable to do the post-test only design. There are other textbooks and researchers that disagree with that. An alternative is to combine those two and do a design called a Solomon 4 design. And here you can see two groups get the treatment, two groups get the pretest, and then you compare. And so you can see if the pretest caused any effect or introduced the testing threat to validity. A Solomon 4 design is probably the strongest. The problem with a Solomon 4 design is sample size. You need a very, very large sample size to do a Solomon 4 design. Again, remember, 30 to 60 individuals per group probably. So oftentimes it's just not feasible. It's too costly. You don't have enough individuals to do a Solomon 4 design. Now, we've just discussed the primary designs, and we've talked about the primary designs. But we've talked about the primary designs. discuss types of designs and specific, then specific designs within each of those types. So true experimental design, you have post-test only design, pre-test, post-test equivalent group design, Solomon 4. When you are identifying your specific research design, you can identify the type, but then be as specific as possible. We've talked about the different types of designs and identifiers. identify the different types of designs. We've also talked about reasons that you would choose the designs. And so you also need to talk about why you chose that specific design. Here with the true experimental design, it's the most rigorous research design when I want to look at a cause and effect relationship between variables. And therefore, that's why I would use a true experimental design. Let's talk a little bit more about choosing your design. Here's a list of questions that can be helpful in identifying which design you're using for your research. And remember, I said you can use mixed designs. So let's say, for example, one of my independent variables is gender, which I'm not going to manipulate. But one of my independent variables is a treatment that I, as a researcher, am going to manipulate. Obviously, since I'm not manipulating gender, I have a causal comparative design. But if I... and manipulating some type of treatment depending on whether or not he's randomization I have a quasi experimental or true experimental design so just recognize that time you may have multiple design so for every variable specifically independent variables are on variables of interest you need to go through and ask these questions so let's go through and ask these questions and I want you to think about your research and I want you to think about on answer yes or no to these first of all Or answer the question. First of all, are you concerned with relationship or difference between variable? If you're concerned with the relationship between variables, then you have a correlational design. If you're just looking at the relationship between two variables, which most research is more complex than that. If you're looking at the relationship just between two variables, we may need to talk. Your design may not be rigorous enough. But if you're just looking at the relationship between two variables, you have a basic correlational design. If you're looking at multiple variables, the relationship of multiple variables to one, or examining a model, you have some type of predictive design. So we've already determined your design if you're looking at relationship. Now, if you're looking at the difference between variables, or are two groups different, you then have some type of group comparison. So the next question you're going to ask is, will I manipulate the independent variable? If your answer is no, it's already occurring within the environment, then you have a causal comparative design, which is a type of expose factor design. So you have causal comparative. If your answer to that question is yes, I am going to manipulate the independent variable, then you need to keep asking yourself these questions. The next question you need to ask yourself is, will I use a control group? If your answer to that is no, then you have a pre-experimental design and you need to rethink your design. If your answer to that is yes, that means you have some type of quasi or true experimental design. So the next question you're going to ask yourself is, will you randomly assign your participants? So will you gather volunteer participants and randomly assign them? If you are randomly assigning them, then you... have a true, probably some type of true experimental design. If you're not randomly assigning them, then you're going to have to have a pre-test and you have some type of quasi experimental design. So I want you to walk through for each of your questions or each of your ideas and I want you to identify what design you are choosing. You then need to ask yourself, is your design feasible and is it the best design, the strongest design that you can do and And does it have enough controls for internal validity? I also want you to ask yourself, what other research has been done in this area? And is the design that I'm choosing warranted based on what's already been done? Now, there's two ways to think about this. Maybe a lot of the research in the area that you're looking at has been correlational and causal comparative. And so you're going to do an experimental design. That's a great rationale. Is everything to this date has been exploratory? When I did my dissertation, that was the case. Everything's been exploratory, and now we need more rigorous research. Another way to justify it is most of the research in the area that you're doing has been quasi-experimental because it's too impossible and too difficult to do a true experimental in an educational setting. And so you justify saying other research has... Similar research has used this design and this design is thus warranted. So, again, go through these questions and ask yourself what design, help them guide and identify what design you're doing. Once you have identified the design, you need to ask yourself a few other questions and begin justifying that design. So, using the information that we've just gone over, using the information in your research text as well as the handout I've provided, Identify your design and justify that design. Justify that design using the literature in research text as well as empirical literature as I just explained. The handout that I provided has some examples of identifying and justifying your research design. Here are a few additional examples that you may want to look at. Again, identify what you're examining, what you're doing. identify the design that you're using, how you're implementing that design, and then justify why you're using that design. Why is it the most appropriate design based both, again, on research text as well as other research that has been done in the field. So take some time, work through, like I said, work through the handout. There are some great questions at the end of the handout and write up a section that you could include in a dissertation thesis or your research plan.

It says when and how often data is collected, what data needs to be gathered and from whom, and how to analyze the data. In this tutorial we're going to talk about several different quantitative research designs. We're going to discuss when they're used and why they should be chosen.

And ultimately the purpose of this tutorial is to help you identify the most appropriate research design for your research study and to help you justify why it's the best choice. Now, before we move on and start Talking about research designs, I want to make a note. Some of you, for your research plans, will actually choose multiple research designs, and this is called a mixed design study.

For example, you may choose a causal comparative and a correlational design. This is different than a mixed method approach. A mixed method approach is a full quantitative and full qualitative design.

It really requires that you conduct a full-blown quantitative study and a full-blown qualitative study. In essence, you're committing to conduct two separate studies. But here we're only going to talk about quantitative research design with the assumption that you've chosen a quantitative design for your study. Now, there are numerous quantitative research designs, too numerous to really cover here. So we're going to focus our attention primarily on the major designs proposed by Campbell and Stanley in 1963 and Krall in 1993. They say that there are three primary categories of studies.

There's descriptive studies, and descriptive studies are done when a researcher wants to know what is. Let's take Charlie, for example. Charlie is a graduate student, and he's interested in college students who have anxiety disorders, because let's say that his cousin has an anxiety disorder.

If he simply wants to know how many students who enter college in, let's say, 2012. have an anxiety disorder, a clinically diagnosed anxiety disorder, he may do a descriptive study. He simply wants to know what is, what exists. The next category are correlational studies, and these can be relationship-focused studies or predictive-focused studies.

In a correlational study, you really want to know, is there a relationship between two different variables, or does a group of variables or one variable predict another variable? In our example of Charlie, Charlie may want to know, is there a relationship between achievement in a freshman English class and the presence of an anxiety disorder? He wants to know simply, is there a relationship, not if one's causing the other, but is there a relationship between the two? He may also ask, does the presence of an anxiety disorder predict achievement in a college freshman English class? The final category is group comparison, and this usually involves manipulation or treatment of some type.

And there are three types. There's a pre-experimental, quasi-experimental, and true experimental. And we'll define those in a few minutes. But really here, let's go back to Charlie. If he wants to do a group comparison study, he's interested in is there a difference in two different groups, or does a specific treatment affect something else?

So... for example let's say he finds out that there is a relationship between the presence of an anxiety disorder and academic achievement in a college english class and he finds that the presence of a disorder actually decreases academic achievement so he wants to do an intervention uh... maybe he uh... astor volunteers on who have generalized anxiety disorder he provides one group with a special treatment that helps them lower their anxiety during the English class and he doesn't provide the other group with the treatment and then he compares the two at the end to determine does the treatment ultimately affect academic achievement in the college English class. He's doing a group comparison.

We're going to take a more in-depth look at each of these specific research designs but before we do I want to identify or talk about a Few additional designs that some of you may choose to do. In addition to the designs that I just overviewed, some of you may choose to do an instrument development, a content or document analysis. a meta-analysis or a single subject design. Let me talk a little bit about three of those. First of all, some of you may choose to do an instrument development.

Oftentimes, when you go to conduct quantitative research, you find that the instrument that you were hoping to find doesn't exist and an instrument needs to be developed. And so you decide to develop an instrument and conduct statistical analysis to determine its validity and reliability. This is a very, very rigorous and intense process. intense type of design.

Some of you may choose to do some type of document or content analysis. That is, analyzing some type of communication. And so something that I see often is that students will analyze a discussion forum to determine the presence of, let's say, community in a discussion forum in an online class. Another example of this. maybe looking at lesson plans, analyzing lesson plans for a specific variable.

The last is a single subject design. For those of you that are really interested in a specific counseling group or a special education group, and you have a small sample size, you may want to consider this design. It's a really valid research design.

And it's similar to a time series design, which we'll talk about, in which each participant really serves as their own control and they're observed repeatedly. Usually what happens is, let's say, for example, we're looking at children with autism and we want to teach them a basic skill such as hand washing. What we would do is you would baseline them.

So you would establish a baseline for each participant. How many steps? of the hand washing procedure can they do. You would then, at different times, implement a treatment that maybe teaches them hand washing, and then you would observe the behavior after the treatment.

And this is called a single subject design. If you're really interested in doing this design, Wasson 2003 has a great book that I highly, highly recommend. But for the purpose of this tutorial, we're going to go back and focus on the main designs that Campbell and Stanley talk about. So let's go ahead and focus on those.

The first type of study we're going to look at is a descriptive study. And descriptive studies focus on what is. They really aim to understand what is happening, what is in a specific situation with an identified population. And they're usually used to gain knowledge to identify a problem.

problem or identify what is so that further research can be conducted. It's important to note here that oftentimes descriptive studies are not very rigorous in nature, so a lot of universities don't allow basic descriptive studies for things such as thesis and dissertations, and that's the case at Liberty. We really don't allow them without special permission. But it is important that you know and you understand what a descriptive study is.

If I was a researcher and I wanted to do a descriptive study, let's say I response to intervention is fairly new term. It's a new type of intervention in the school systems. And I simply want to know what is the attitude of school counselors about the use of response to intervention.

I would do a descriptive study. Another descriptive study may be what is the what responsibilities do school counselors even have in RTI as it's implemented in their school. So just general I want to know what are school counselors doing to help to help implement RTI in the school I would do a descriptive study now most research text identify two different types of descriptive studies a survey study or and an observational study a survey study uses some type of written document or online questionnaire or interview on to gather information about what is and there's usually two basic types of survey studies, and that's longitudinal designs and cross-sectional designs. Longitudinal designs study a population over a period of time. They oftentimes are referred to as trend studies or cohort studies or panel studies.

So maybe, going back to our example of Charlie, maybe he wants to study the academic achievement of students identified with anxiety disorder in their freshman year of college. their junior year of college and their senior year of college. He's studying them over a period of time.

A cross-sectional study then just looks at individuals at one point in time. So again, going back to our example of Charlie, Charlie simply wants to know what is the academic achievement of freshman college students with an anxiety disorder. So he just was looking at one specific point in time about what is.

Oftentimes, surveys collect data on attitudes and beliefs. Then, like I said, the other type of design is the observational design. And this is the process of observing something that's happening and making a conclusion about it.

So for example, I may go into a school system and observe teachers or observe school counselors. Again, just as a reminder, descriptive studies oftentimes are not rigorous enough for dissertations and theses. And that's the case at Liberty.

We oftentimes don't allow descriptive designs for dissertation purposes. However, descriptive studies can be very important as you build a case for a thesis or a dissertation. Thank you.

because they're describing what the problem is. Now let's look at another type of study. The next design we're going to talk about is one that I didn't mention above, but it's similar to a group comparison study, and that's a causal comparative study. The aim of a causal comparative study is really to examine the possible cause and effect relationship between variables that exist. It really, you're studying a phenomenon after the fact.

That is, it's already occurred either naturally or it's been manipulated. Let's consider some examples. If you want to examine gender or ethnicity as an independent variable, you have a causal comparative study.

It's something that you as the researcher have not manipulated. Gender occurs naturally. Ethnicity occurs naturally. If you as a researcher wanted to examine individuals that smoked versus individuals that didn't, didn't smoke.

That has already happened. They've already chosen that behavior. Another example would be an intervention that it's naturally occurring within a school system and is not occurring in another school system. You want to study something that's already been implemented.

Oftentimes, causal comparative studies are used in educational research because it's too difficult, too unethical, or impossible to manipulate the independent variable. So, for example, it's unethical for me to tell one group of individuals, let's say one group of students, to smoke and another group not to smoke. Maybe there's a specific intervention, a school-wide intervention happening at a school. And you can't ask the whole school to implement this new intervention just for your research study. That would be too difficult.

And since it's naturally occurring, you can go ahead and study it. Now, it's important to note that a causal comparative study is almost identical to a pre-experimental design. And we'll talk about in a little bit that that's not a credible design. But the difference is that oftentimes a causal comparative study has a control group.

Actually, it always has a control group. So what you're looking at with a causal comparative design is you're looking at a phenomenon or something that's happened, and you're comparing it with something else. So, for example, a school has this dynamic program that they're implementing, and you're going to compare that school's... student's achievement level to another school that's not using that specific treatment.

When we talk about threats to validity, we'll talk about all the controls that need to be in place, because again, there's lots of threats to validity, but this is a valid type of study. I like to give you an example of a... question that may be used with a causal comparative study.

For example, I may say, is there a difference in male and female university students' social presence while participating in an eight-week online class? I simply want to know, is there a difference in their social presence between males and females? And again, I can't manipulate it. It's unethical and impossible for me to manipulate gender. And so I would choose a causal comparative study.

As we're talking about causal comparative studies, there's a few additional notes that I want to make about this specific design. And especially take note of this if you're planning to use this type of design. Note that like a descriptive study, a causal comparative design does not require the researcher to exert control or manipulate the phenomenon.

However, unlike a descriptive study, you are looking at the possible cause and effect relationship between variables, and therefore you do. You have an independent variable with at least two levels. at least two groups that you're comparing, and you have a dependent variable.

You're looking at the possible effect that that independent variable has on the dependent variable. And note that I said possible cause and effect. This is really, really important language if you choose a causal comparative design. You cannot prove, you cannot state that you have a cause and effect relationship if you have a causal comparative design, because there are too many threats to validity. So make sure.

If you choose a causal comparative design, you're using correct terminology. And just as a reminder, when you choose a causal comparative design, you do so because it's too difficult, too unethical, or impossible for you as the researcher to manipulate the independent variable. Also note that oftentimes causal comparative designs are chosen because you want to explore a phenomenon that's happening. If a lot of causal comparative research has been done in a specific area, and you're finding a lot of causal comparative literature research in that specific area you're studying, it probably means that we need to start moving more toward more rigorous research, such as an experimental design, which we're going to talk about next.

So that's the causal comparative design. Actually, before we talk about experimental research, we're going to talk about the correlational research designs. A correlational design, like a causal comparative study, is used for exploratory or beginning research to determine if there is truly more rigorous research that's warranted.

The purpose of a correlational study is simply to examine if a relationship exists between variables. Is there a relationship between variables? like we said in the case of Charlie, is there a relationship between the presence of an anxiety disorder, so whether or not somebody has an anxiety disorder, and their academic achievement or their grade in a freshman college class. Other examples of correlational studies may be, is there a relationship between high school GPA and college board SAT scores? We know in the research that some research says there is and some research says there isn't.

If you choose to do a correlational design, it's important that you recognize that you cannot speak in terms of cause and effect. You simply have two variables of interest. One does not cause the other because you don't know the direction of the cause and effect.

You can't guarantee that one variable caused another. All you know is that they are varying together. Either, let's say we're looking at two scores such as GPA and SAT.

We just want to know, do they increase and do they decrease together? Also, you may be looking at predictive. I remember I said at the beginning there are two basic types of correlational designs. There's ones that look at relationship, and thus far everything I've described has been in terms of relationship.

There's also predictive studies. If you want to know do SAT scores predict college SAT scores, that you can look at what does one variable predict another variable now it's important to note and specially in talking about prediction that you have a solid theoretical or conceptual framework in which you're basing that study oftentimes with with predictive studies there's been multiple studies that has shown that there's relationship between certain variables or multiple variables and you wanna add to that conceptual or that model or that theoretical model. Let me give you an example of what I mean by that.

We know that there's been a lot of research on high school GPA and SAT scores, performance in college, and we've seen that not only does high school GPA predict SAT scores, but also it predicts the performance of high school students. gender and ethnicity and sometimes there's an interaction between gender and ethnicity and social economic status and high school GPA and the results of an SAT score. So let's say in the literature I find a strong conceptual base or a model that says gender and ethnicity and social economic status and GPA predict SAT scores. Let's say that that model predicts SAT scores predicts, let's say, 60% of the SAT score. That means there's still 40% of the model that's unexplained.

And maybe I have another framework or another theory that says parents' parental or parental attendance at college is another important piece of the puzzle. So then I'm going to test that. So it's important in correlational designs that you really have a strong theoretical or conceptual rationale for every variable that you plan to study, not just ones that you think, but there needs to be research and theory that backs it up.

Again, I want to reiterate, if you choose a correlational study, if you want to look at the relationship between two variables or multiple variables, note that you're you can only make statements about relationship, and you use this design because there's very little research in the area, and it's exploratory. You're only beginning research in the area, or the models that have been presented, specifically predictive models that have been presented, are not complete. There's still parts of it that are unexplained. Now we're going to focus our attention on experimental designs. Campbell and Stanley in 1963 purported that there are three...

primary experimental designs. And that's the pre-experimental design, the true experimental design, and the quasi-experimental design. And the distinguishing factor or distinguishing characteristic of all experimental designs is that the researcher manipulates the independent variable.

That is, the researcher actually conducts the treatment, does something to the participants. So let's look at these three types of designs. As we discuss these three types of designs, there are commonly used symbols in research text.

And actually Campbell and Stanley in 1963 are the ones that proposed these symbols. But anyway, there are these symbols in research text that help us visually see the designs that we're talking about. And so we're going to use those symbols as we talk about these different designs over the next few slides.

And the X. Simply means that exposure to treatment. That means there's been a treatment done.

O represents observation or the measurement, the dependent variable, really, what you're measuring. Multiple roles reflect multiple groups. And R reflects randomization.

And I'll define these more in a little bit. We're going to start by discussing pre-experimental designs. And the primary characteristic of a pre-experimental design is that the...

Researcher manipulated the independent variable. Oftentimes, pre-experimental designs are used as preliminary research or pilot studies to determine the effectiveness of a treatment to see if there's more research that's warranted. Campbell and Stanley really say that pre-experimental designs are weak and have little to no value.

It's important to note that pre-experimental designs are distinguished as separate from quasi-experimental designs in texts such as Campbell and Stanley, but not so much in Borg, Borg and Gall. So some research texts don't even acknowledge pre-experimental designs or they wrap them into quasi-experimental designs or the quasi-experimental design discussion. I bring up pre-experimental designs here because I want you to be able to recognize them and I want you to recognize that they are not rigorous studies.

They, like Campbell and Stanley said, they have no value. So for a dissertation or a thesis they really shouldn't be done. They are, however, often used in program evaluation for school systems and can be very effective for that purpose. An example of a question that may be asked for pre-experimental design is simply, Does parent scores on a parent skill assessment increase after participation in a successful parenting program? Let's look at what a design, that would be probably a one-shot case study.

Let's look at what that looks like in the next slide. Here are commonly used pre-experimental designs. As I just mentioned, the question I just proposed, you know, do parents increase in their skills or do parents have better parenting skills after a parenting intervention.

That's an example of a one-shot case study. Parents are given a parenting skill intervention, and then their parenting skills are measured. Now, you probably already see the weakness in this study.

I'll give you a moment to think about it. Do you have it? The problem with this is I can't say it was the intervention that caused the parenting skills. Perhaps parents already had the skills. Perhaps they were watching a...

a parenting program and that caused the score they got in the parent on the parenting skill let's say survey or measurement that i used um so i really can't say it was the intervention that caused caused the effect on the pair or caused the parenting skills additional pre-experimental designs are um the one group pre-test post-test again um here i give parents Here, going back to our parenting example, I would give parents a pretest to measure their parenting skills. I'd do the intervention, and then I'd give them the post-test. Now, this may be a stronger design, because I can see if there is an increase or decrease in parenting skills. But it might have been the pretest that caused the increase on the post-test. So again, I can't say it was the parenting skill class that actually caused.

the intervention or caused the increase or decrease in the parenting skills. Same, again, similar weakness in the post-test only non-equivalent group design. If I have two groups and I give one parenting skills and don't give it to the other and then measure them at the end, it may be simple group differences that were the cause, the differences in the parenting skills and not actually the intervention. So, these are pre-experimental designs. Let me give you one more example and see if you can identify the design.

Let's go back to our case of Charlie. Let's say Charlie has a group of students, freshman college students, who have an anxiety disorder. And he gives them an anxiety intervention and it helps them reduce their anxiety about their academics or in their classes. And then he post-tests to see if there is a decrease in their anxiety. Again, the problem is if he has one group, he does the intervention, he measures them, it may not be the treatment that actually caused the effect, the lowering, let's say, of the anxiety skills.

So again, pre-experimental designs, it's important to know what they are. They can be very effective in program evaluation, sort of exploring, is it even worth going on and... examining a treatment further. However, for a dissertation, they are not acceptable designs. As you can see, there's major threats to validity, and you can't talk about cause and effect relationships.

So let's look at two experimental designs that are often used and are some of the strongest designs that can be used for a dissertation. Let's first talk about quasi-experimental designs. Now, the quasi-experimental design is...

often used in educational research because it's more convenient and less disruptive than the stronger true experimental design. The purpose of a quasi-experimental design is to test the effectiveness of an intervention or treatment with a target population. It really allows the researcher to control the treatments and look to see if the intervention had an effect on the identified dependent variable.

Again, this is the most rigorous design aside from the true experimental design and often chosen in educational research because it's more convenient and less disruptive. Let's look at an example of that. Let's say that you wanted to look at problem-based pedagogy for teaching math. And so you ask the question, what effect does participation in a math lesson developed using a problem-based pedagogy have on teaching math? problem-based pedagogy have on second grade students' math achievement scores when compared to participation in a math lesson that was developed using traditional pedagogy.

And you have Mrs. Smith's class and Mrs. Jones's class. And Mrs. Smith, you train to use the problem-based pedagogy approach. And Mrs. Jones, she continues using the traditional approach. And your specific target population is second grade students.

Now, it would be really nice, we're going to talk about random assignment in a moment, it would be really nice to randomly assign students to either Mrs. Smith's class or Mrs. Jones's class. However, that's probably not possible in an educational situation. You can't just go randomly assigning students to classes.

Therefore, you use Mrs. Smith's class that's already been put together and Mrs. Jones's class. One gets the treatment, one doesn't, and you compare their math achievement scores at the end. You used already composed classes, and so that's a quasi-experimental design.

Let's look at, more specifically, let's look at specific types of quasi-experimental designs. If you choose to conduct a quasi-experimental design, you may use one of these specific designs. Probably most prominent is the one that I just explained. And, That's the non-equivalent pre-test, post-test control group design.

And that's the middle one here, represented with our little X's and O's. What you do is you pre-test. So you give both, let's say you gave Mrs. Smith's class and Mrs. Jones' class a pre-test. You want to find out what their math achievement was prior to the intervention.

You then give Mrs. Smith's class the intervention. You don't give it to Mrs. Jones' class. And then you post-test them.

This way, you can actually see if the groups were equivalent in their math achievement prior to the intervention. So you know if it was the intervention or not that caused the... post-test scores.

An interrupted time series design is very similar to that where you you create a baseline so you would actually give let's say multiple math achievement tests prior to the intervention you give one group the intervention and then you do multiple post-tests. Another group another design is the counterbalance design and this one can be very effective at looking at when you want to do multiple treatments and you give each group a different regimen of the treatment and then compare them so you can see what the effect is. Now this is a great design one of the weaknesses as you can see would be you you're adding multiple or you're adding more groups and again sometimes in educational research it can be difficult when you have multiple groups because each group needs to be fairly large.

When we talk about different When we talk about statistical convention and power, when we talk about an analysis, you'll see that you probably need anywhere between 30 and 60 individuals per group. Again, just to recap, a quasi-experimental design is often used in educational research. It includes both, it has two primary characteristics, and that's manipulation. You as the researcher actually cause the treatment.

You manipulate the independent variable. And there's always a control group, a comparison group. Now, talking about control comparison group, let me stop here for a moment and talk about that. There are different types of control groups, and you need to decide what type of control group you are going to use. This Kazdin 2003 book has a great discussion on different types of control groups you have.

Most of the time in... In educational research, you're going to use a no treatment control group of some sort or a nonspecific treatment group in which one group gets an intervention and the other group is the same in every single aspect, in every single way, except for that intervention. And let me also say this. When you're thinking about that control group, it's important that The element that you're studying is the only thing different. So, for example, I can't compare Mrs. Jones' math class to Mrs. Smith's English class, or compare a sixth grade math class to a fifth grade math class.

Everything about the intervention needs to be the same. So the content of the material, everything needs to be very, very similar except for the intervention. But anyway, when you're talking about control groups or a group that's there as a comparison group, it's important to identify that there are different types of groups.

And again, I highly, highly recommend Kasdan as you talk about this. And he talks about factors that you really need to consider when choosing a control, and that's the intent of your research, previous research that's been done, as well as ethical and practical considerations. Let me go back and just remind you, quasi-experimental design has two characteristics, manipulation and control.

It is often used in educational research because it's too impossible or not ethical to do the more rigorous true experimental design, which we're going to talk about next. And really the aim of a quasi-experimental design is to determine the effectiveness of some type of intervention. If you're thinking about a causal comparative study and you have the opportunity to do a quasi-experimental study, I highly, highly recommend you consider a quasi-experimental study.

It's a much more rigorous, it's a better research design. There's less threats to internal validity. The last design we're going to talk about is the true experimental design.

this design is truly the most rigorous design and ultimately the design that you want to choose if you have the opportunity to do so. The purpose of this design is to make sure that you is to examine the cause and effect relationship between variables. You really investigate the possible cause and effect relationship by exposing one or more experimental groups to some type of treatment that you as the researcher do, and you compare those results to one or more control groups not receiving the treatment. Again, this is considered the most rigorous design.

Examples of questions that you may ask if you're doing a true experimental design is, is there a difference in students' sense of community based on the type of technology they're using in a class? So, for example, let's say I have two classes. One class does all of their discussion via Blackboard discussion. Another class does all of their discussion via Skype. And I want to compare, does community differ among the students?

So everything in the classes are exactly the same except the technology that's used for the discussion. And obviously there's theoretical and conceptual reasons for why you would compare those two things. And let's say there are three characteristics of a true experimental design.

So I have already talked about manipulation. So I, as the researcher, am manipulating the independent variable. And I just talked about that.

I'm giving one group a discussion using discussion board. And I'm giving another group a discussion. I'm going to do it via Skype or via some type of online conferencing system.

There's a control group. So there's a comparison group. And the last thing that I... that characterizes or makes a true experimental design unique is the ability to randomize. So in this specific scenario, I would ask for a group of volunteers, and then I would randomly assign them to either the treatment group or the control group.

Now, the important thing to recognize about randomization is this. Campbell and Stanley in 1963 say, that if you randomize, you can assume group equivalence. You can assume a group equivalence. And so you can assume the groups are starting out the same, and so it's actually the treatment that caused the effect. As we've done with the other categories or other types of designs, let's look at specific types of true experimental designs.

One of the most rigorous designs is the post-test. only design. In this case, you randomize your groups. You do random assignment.

Notice I said random assignment, not random selection. Random assignment, so I have my group of volunteers. In the example I gave, I used college students, let's say. I have a group of them, and then I randomly assign one to treatment, one to non-treatment. I do the treatment, and then I post-test both of them.

There's also a pre-test, post-test equivalent group design in which is exactly what I just explained. However, you randomize them, you do a pre-test, you give one group the experiment or the treatment and then you, not the other group, and then you post-test them. Now Campbell and Stanley say that this design introduces a threat to validity and that's the testing threat to validity, so it's preferable to do the post-test only design.

There are other textbooks and researchers that disagree with that. An alternative is to combine those two and do a design called a Solomon 4 design. And here you can see two groups get the treatment, two groups get the pretest, and then you compare. And so you can see if the pretest caused any effect or introduced the testing threat to validity.

A Solomon 4 design is probably the strongest. The problem with a Solomon 4 design is sample size. You need a very, very large sample size to do a Solomon 4 design. Again, remember, 30 to 60 individuals per group probably. So oftentimes it's just not feasible.

It's too costly. You don't have enough individuals to do a Solomon 4 design. Now, we've just discussed the primary designs, and we've talked about the primary designs. But we've talked about the primary designs. discuss types of designs and specific, then specific designs within each of those types.

So true experimental design, you have post-test only design, pre-test, post-test equivalent group design, Solomon 4. When you are identifying your specific research design, you can identify the type, but then be as specific as possible. We've talked about the different types of designs and identifiers. identify the different types of designs.

We've also talked about reasons that you would choose the designs. And so you also need to talk about why you chose that specific design. Here with the true experimental design, it's the most rigorous research design when I want to look at a cause and effect relationship between variables.

And therefore, that's why I would use a true experimental design. Let's talk a little bit more about choosing your design. Here's a list of questions that can be helpful in identifying which design you're using for your research. And remember, I said you can use mixed designs. So let's say, for example, one of my independent variables is gender, which I'm not going to manipulate.

But one of my independent variables is a treatment that I, as a researcher, am going to manipulate. Obviously, since I'm not manipulating gender, I have a causal comparative design. But if I...

and manipulating some type of treatment depending on whether or not he's randomization I have a quasi experimental or true experimental design so just recognize that time you may have multiple design so for every variable specifically independent variables are on variables of interest you need to go through and ask these questions so let's go through and ask these questions and I want you to think about your research and I want you to think about on answer yes or no to these first of all Or answer the question. First of all, are you concerned with relationship or difference between variable? If you're concerned with the relationship between variables, then you have a correlational design.

If you're just looking at the relationship between two variables, which most research is more complex than that. If you're looking at the relationship just between two variables, we may need to talk. Your design may not be rigorous enough. But if you're just looking at the relationship between two variables, you have a basic correlational design.

If you're looking at multiple variables, the relationship of multiple variables to one, or examining a model, you have some type of predictive design. So we've already determined your design if you're looking at relationship. Now, if you're looking at the difference between variables, or are two groups different, you then have some type of group comparison.

So the next question you're going to ask is, will I manipulate the independent variable? If your answer is no, it's already occurring within the environment, then you have a causal comparative design, which is a type of expose factor design. So you have causal comparative.

If your answer to that question is yes, I am going to manipulate the independent variable, then you need to keep asking yourself these questions. The next question you need to ask yourself is, will I use a control group? If your answer to that is no, then you have a pre-experimental design and you need to rethink your design. If your answer to that is yes, that means you have some type of quasi or true experimental design.

So the next question you're going to ask yourself is, will you randomly assign your participants? So will you gather volunteer participants and randomly assign them? If you are randomly assigning them, then you...

have a true, probably some type of true experimental design. If you're not randomly assigning them, then you're going to have to have a pre-test and you have some type of quasi experimental design. So I want you to walk through for each of your questions or each of your ideas and I want you to identify what design you are choosing. You then need to ask yourself, is your design feasible and is it the best design, the strongest design that you can do and And does it have enough controls for internal validity? I also want you to ask yourself, what other research has been done in this area?

And is the design that I'm choosing warranted based on what's already been done? Now, there's two ways to think about this. Maybe a lot of the research in the area that you're looking at has been correlational and causal comparative.

And so you're going to do an experimental design. That's a great rationale. Is everything to this date has been exploratory?

When I did my dissertation, that was the case. Everything's been exploratory, and now we need more rigorous research. Another way to justify it is most of the research in the area that you're doing has been quasi-experimental because it's too impossible and too difficult to do a true experimental in an educational setting. And so you justify saying other research has...

Similar research has used this design and this design is thus warranted. So, again, go through these questions and ask yourself what design, help them guide and identify what design you're doing. Once you have identified the design, you need to ask yourself a few other questions and begin justifying that design. So, using the information that we've just gone over, using the information in your research text as well as the handout I've provided, Identify your design and justify that design. Justify that design using the literature in research text as well as empirical literature as I just explained.

The handout that I provided has some examples of identifying and justifying your research design. Here are a few additional examples that you may want to look at. Again, identify what you're examining, what you're doing.

identify the design that you're using, how you're implementing that design, and then justify why you're using that design. Why is it the most appropriate design based both, again, on research text as well as other research that has been done in the field. So take some time, work through, like I said, work through the handout.

There are some great questions at the end of the handout and write up a section that you could include in a dissertation thesis or your research plan.

Transcript for:Choosing the Right Quantitative Research Design

Transcript for:
Choosing the Right Quantitative Research Design