Research Design

Sema A Kalaian. Encyclopedia of Survey Research Methods. Editor: Paul . Lavrakas. Volume 2, Sage Publications, 2008.

A research design is a general plan or strategy for conducting a research study to examine specific testable research questions of interest. The nature of the research questions and hypotheses, the variables involved, the sample of participants, the research settings, the data collection methods, and the data analysis methods are factors that contribute to the selection of the appropriate research design. Thus, a research design is the structure, or the blueprint, of research that guides the process of research from the formulation of the research questions and hypotheses to reporting the research findings. In designing any research study, the researcher should be familiar with the basic steps of the research process that guide all types of research designs. Also, the researcher should be familiar with a wide range of research designs in order to choose the most appropriate design to answer the research questions and hypotheses of interest.

Generally, the research designs can be classified into one of three broad categories based on the nature of research, purpose of research, research questions, sample selection, data collection methods, and data analysis techniques: (1) quantitative research designs, (2) qualitative research designs, and (3) mixed-research designs.

Quantitative Research Designs

Quantitative research is a deductive theory-based research process that focuses primarily on testing theories and specific research hypotheses that consider finding differences and relationships using numeric data and statistical methods to make specific conclusions about the phenomena. Quantitative research designs can be classified into one of four broad research design categories based on the strength of the research design’s experimental control: (1) true experimental research designs, (2) quasi-experimental research designs, (3) pre-experimental research designs, and (4) nonexperimental research designs.

Although each of the categories of research design is important and can provide useful research findings, they differ in the nature of the evidence they provide in establishing causal relations between variables and drawing causal inferences from the research findings. Experimental designs are the most rigorous, powerful, and the strongest of the design categories to establish a cause-effect relationship. Nonexperimental designs are the weakest in terms of establishing a cause-effect relationship between variables because of the lack of control over the variables, conditions, and settings of the study.

True Experimental Research Designs

The true experiment is a type of research design where the researcher deliberately manipulates one or more independent variables (also called experimental variable or treatment conditions), randomly assigns individuals or objects to the experimental conditions (e.g., experimental or control groups) and controls other environmental and extraneous variables, and measures the effect of the independent variable on one or more dependent variables (experimental outcome). The experimental group is the group that receives the treatment, and the control group is the group that receives no treatment or sometimes a placebo (alternative treatment that has nothing to do with the experimental treatment). Thus, in a typical experimental study, the researcher randomly selects the participants and randomly assigns them to the experimental conditions (e.g., experimental and control), controls the extraneous variables that might have an effect on the outcome (dependent) variable, and measures the effect of the experimental treatment on the outcome at the conclusion of the experimental study.

It is important to emphasize that the experimental research design, if well conducted, is the most conclusive and powerful of all the research designs and the only research design that tests research questions and hypotheses to establish cause-effect relationships. For this reason it is sometimes called the “Golden Design.”

The simple randomized experimental designs with two groups can be conducted using one of the following four basic experimental designs:

Randomized Two-Group Post-test-Only Designs

The two-group randomized experimental design involves two groups of individuals or objects which, ideally, are randomly selected from the population and which are randomly assigned to the experimental and control (comparison) groups (a single independent variable with two levels: experimental and control groups). The effects of the experimental treatment on the dependent variable (experimental outcome) are measured at the conclusion of the experiment. It is represented as

Experimental Group: R E O

Control Group: R C O

where R is random assignment of individuals or objects, E represents the experimental treatment, C represents the control condition (no treatment or placebo treatment), and O represents the posttest observation (measurement).

An example of this design would be testing an alternative wording of the mail survey cover letter, compared to a standard cover letter, to determine whether the new cover letter raised cooperation. Households would be randomly assigned to either the standard or alternative cover letter. The resulting response rates between the two groups would represent the dependent variable used to test the hypothesis that the new wording raises cooperation.

Randomized Two-Group Pretest-Posttest Designs

This experimental design involves two groups of individuals or objects randomly selected from the population and randomly assigned to the experimental and control groups (a single independent variable with two levels: experimental and control groups). The two groups are pretested on the dependent variable before administering the experimental treatment and post-tested on the same dependent variable at the conclusion of the experiment. This design is represented as

Experimental Group: R O1 E O2

Control Group: R O1 C O2

where R is random assignment of individuals or objects, E represents the experimental treatment, and C represents the control condition (no treatment or placebo treatment). The O1 represents the pretest observation (measurement), and the O2 represents the posttest observation (measurement).

An example of this design would be a telephone survey questionnaire that measures the effects of new information on approval versus disapproval of a proposed city bond to fund the building of a new bridge. All respondents would be asked whether they favor or oppose the new bridge funding early in the questionnaire. Later in the questionnaire they would be asked the same favor-oppose question again, but a random half of them would first be told some information about the value of the new bridge and the other half would not be told this information. Nothing else in the questionnaire would change. The difference in answers between the before-and after-questions about the funding for the two groups would serve as the dependent variable to test the hypothesis that the new information raises support for the bridge funding.

Solomon Four-Group Designs

This experimental design is a combination of the randomized two-group posttest-only design and the randomized two-group pretest-posttest designs. It involves randomly selecting a sample of subjects from the targeted population and randomly assigning the random sample to one of four groups. Two of the groups are pretested (Experimental and Control Groups 1) and the other two are not (Experimental and Control Groups 2). One of the pretested groups and one of the not pre-tested groups receive the experimental treatment. All four groups are posttested on the dependent variable (experimental outcome). The design is represented as

Experimental Group 1: R O1 E O2

Control Group 1: R O1C O2

Experimental Group 2: R E 02

Control Group 2: R C O2

Here, the researcher has two independent variables with two levels. One independent variable is the experimental conditions with two levels (experimental and control groups), and the other independent variable is the pretesting condition with two levels (pretested and not pretested groups). The value of this design is that it allows the researcher to determine if the pretest (O1) has an effect on the resulting answer given in the posttest.

An example of this design would be one that builds on the previous example of the experiment to test the effect of the information about the value of the new bridge. However, in the Solomon four-group design, there would be two more randomly assigned groups of respondents, ones who were not asked whether they favored or opposed the bridge funding at the beginning of the questionnaire. Instead, one of these groups would be the second control group, asked only their opinions about the bridge funding later in the questionnaire. The other group would be the second experimental group, asked their opinions about the bridge funding only later in the questionnaire but after first being given the information about the value of the bridge. This design would allow the researchers to test not only the effects of the information but also whether the saliency of the bridge funding, by asking about it first before giving the new information, affected opinions given later about the funding.

Experimental Factorial Designs

Experimental factorial designs are extensions of single independent variable experimental designs to situations where there are two or more independent variables that are controlled by the researcher. Factorial designs allow the researcher to examine simultaneously the effects of one or more independent variables individually on the dependent variable (experimental out-come) as well as their interactions. These interactions cannot be examined by using single independent variable experimental designs.

The term factorial refers to experimental designs with more than one independent variable (factor). Many different experimental factorial designs can be formulated depending on the number of the independent variables. The Solomon four-group design is an example of a 2 × 2 factorial design with treatment conditions (treatment and control groups) crossed with pretesting conditions (pretested and not pretested groups).

Quasi-Experimental Research Designs

Quasi-experimental research is used in situations where it is not feasible or practical to use a true experimental design because the individual subjects are already in intact groups (e.g., organizations, departments, classrooms, schools, institutions). In these situations it is often impossible to randomly assign individual subjects to experimental and control groups. Thus, quasi-experimental designs are similar to experimental designs in terms of one or more independent (experimental) variables being manipulated, except for the lack of random assignment of individual subjects to the experimental conditions (i.e., experimental and control groups). Instead, the intact groups are assigned in a nonrandom fashion to the conditions. Types of quasi-experimental designs include nonequivalent control group designs, longitudinal research designs, and multilevel research designs.

Nonequivalent Control Group Design

The nonequivalent control group design involves assignment of intact nonequivalent groups (e.g., classrooms, schools, departments, and organizations) to experimental conditions (experimental and control groups). Thus, the intact groups are assigned to the treatment conditions and not the individual subjects, as was the case in the true experimental designs. For example, in a study of the effects of a new curriculum of students’ knowledge of science and attitudes toward science, some classrooms would be assigned to receive the new curriculum and others would not. Toward the end of the school year, all students are measured on their science knowledge and attitudes toward science. Because the effects are being measured at the level of the individual student, but the students themselves were not randomly assigned to the control and treatment condition, this is a quasi-experiment, not a true experiment.

Longitudinal Research Designs

Longitudinal, repeated-measures, or time-series research designs involve repeated measurement or observation on the same individuals at several points over a period of time. It is an elaboration of the one-group pretest-posttest design and focuses primarily on change, growth, and developmental types of research questions across many different disciplines such as medicine, public health, business, and social and behavioral sciences. Longitudinal designs, if well designed and conducted, are usually more complex, time consuming, and expensive than the other types of research designs.

Multi-Level Research Designs

Multi-level or hierarchical research designs involve the nesting of individuals (micro-level units) within organizations (macro-level units) and having explanatory independent variables characterizing and describing both levels. For example, in a two-level design, the emphasis is on how to model the effects of explanatory variables (predictors) at one level on the relationships occurring at another level. These multilevel and hierarchical structured data present analytical challenges that cannot be handled by traditional linear regression methods because there is a regression model for each level of the hierarchy. Thus, hierarchical models explicitly model the micro and macro levels in the hierarchy by taking into consideration the interdependence of individuals within the groups.

Pre-Experimental Research Designs

Pre-experimental research designs are simple designs with no control groups. These designs are questionable because they lack control and thus should be used for exploratory or preliminary examination of research problems.

One-Group Posttest Experimental Design

The one-group experimental design, also called the one-shot experimental design, takes a single group of subjects or objects exposed to a treatment (X) and observes and measures its effects on the outcome (O). This simple design is represented as

X → O

This is the most basic and simple design in experimental research. It is used as a starting point for preliminary examination of the precausal relationship of research problems for the purpose of developing better-controlled future experimental designs.

One-Group Pretest-Posttest Design

The one-group pretest–posttest design involves a single group of individuals or objects that are pre-tested or measured (O1), exposed to an experimental treatment (X), and posttested or measured (O2). This design is represented as

O1→ X → O2

Nonexperimental Research Designs

Nonexperimental or descriptive research designs aim to answer research questions about the current state of affairs, identify factors and relationships among them, and create a detailed quantitative description of phenomena. Thus, it provides a snapshot of the feelings, opinions, practices, thoughts, preferences, attitudes, or behaviors of a sample of people, as they exist at a given time and a given place. For example, measuring the attitudes of the employees in the organization toward adapting new technologies is an example of a research question that can be carried on using a nonexperimental descriptive survey research design. The following are short descriptions of some of these designs.

Nonexperimental Survey Research

Survey research is a systematic research method for collecting data from a representative sample of individuals using instruments composed of closed-ended and/or open-ended questions, observations, and interviews. It is one of the most widely used nonexperimental research designs across disciplines to collect large amounts of survey data from a representative sample of individuals sampled from the targeted population using a variety of modes such as face-to-face, telephone, mail, and electronic (Web-based and email). Each of these data collection modes has its own advantages and disadvantages in terms of cost, duration, and response rate. Thus, the key goal of nonexperimental survey research is to collect data and describe the behaviors, thoughts, and attitudes of a representative sample of individuals at a given point in time and place.

Survey research is considered one of the most important research designs, and survey instruments and survey methods are frequently used to collect data for the other quantitative, qualitative, and mixed research designs. For example, it can be used to collect data for correlational research studies, experimental studies, and quasi-experimental studies.

Correlational Research

Correlational research is a type of descriptive non-experimental research because it describes and assesses the magnitude and degree of an existing relationship between two or more continuous quantitative variables with interval or ratio types of measurements or discrete variables with ordinal or nominal type of measurements. Thus, correlational research involves collecting data from a sample of individuals or objects to determine the degree of the relationships between two or more variables for the possibility to make predictions based on these relationships. There are many different methods for calculating a correlation coefficient, which depends on the metric of data for each of the variables. The most common statistic that measures the degree of the relationship between a pair of continuous quantitative variables, having interval and ratio types of measurements, is the Pearson product-moment correlation coefficient, which is represented by the letter r.

Alternative correlation coefficients can be used when the pair of variables has nominal or ordinal types of measurement. If the pair of variables is dichotomous (a nominal type of measurement having only two categories), the Phi coefficient should be used. If the pair of variables has ordinal type of measurement, the Spearman rank order correlation coefficient should be used.

Another type of correlational research involves predicting one or more continuous quantitative dependent variables from one or more continuous quantitative independent variables. The most common statistical methods used for prediction purposes are simple and multiple regression analyses.

The significance of correlational research stems from the fact that many complex and sophisticated statistical analyses are based on correlational data. For example, logistic regression analysis and discriminant function analysis are quite similar to simple and multiple regression analyses with the exception that the dependent (criterion) variable is categorical and not continuous as in simple and multiple regression analyses. Canonical analysis is another statistical method that examines the relationship between a set of predictor (independent) variables and a set of criterion (dependent) variables. Path analysis and structural equation modeling are other complex statistical methods that are based on correlational data to examine the relationships among more than two variables and constructs.

Causal-Comparative Research

Causal-comparative or ex post facto research is a type of descriptive nonexperimental research because it describes the state of existing differences among groups of individuals or objects as they existed at a given time and place and attempts to determine the possible causes or reasons for the existing differences. Thus, the basic causal-comparative approach starts with selecting two or more groups with existing differences and comparing them on an outcome (dependent) variable. Also, it attempts to examine and explain the possible causes of the existing differences between the groups.

Some causal-comparative designs involve only two independent groups to be compared on a particular continuous dependent variable, for example, studying the differences between boys and girls on math achievement. In this causal-comparative study, the researcher needs to analyze the collected data using t-test for testing the research hypothesis that there are differences between the two independent sample means. Some other causal-comparative research designs involve more than two groups, for example, studying differences between white, black, and Hispanic students on math achievement. In this study, the researcher needs to use analysis of variance (ANOVA) to analyze the data.

Other causal-comparative designs involve studying differences between (among) two or more independent groups on two or more related dependent variables. In this case, multivariate analysis of variance (MANOVA) statistical procedure should be used to analyze the data to determine whether two or more independent groups differ on more than a single dependent variable.

It is important to note that the t-test, ANOVA, and MANOVA are parametric statistical procedures that require interval-or ratio-level data, a large sample size, and meeting the requirements of statistical assumptions (e.g., normality, independence of observations). The nonparametric counterparts for these statistical methods should be used with nominal-or ordinal-level data and when one or more of the assumptions are violated in the research study and when the sample size is small. For example, a nonparametric statistical method such as Mann-Whitney U is an alternative to the parametric t-test.

Meta-Analysis Research

The meta-analysis design is used to quantitatively and systematically summarize and synthesize the research results and findings from a collection of primary studies that address and test the same research question. Meta-analytic research methods have established five major general stages that guide meta-analysts in their systematic quantitative review. These stages include (1) formulating research problems, (2) collecting primary research studies, (3) evaluating primary studies, (4) analyzing and modeling the meta-analytic data, and (5) interpreting and presenting the meta-analytic results.

Generally, the key goals of meta-analysis methods are to (a) produce quantitative summary measures of the effect sizes, (b) assess the heterogeneity (variation) among the effect sizes, and (c) model and explain the heterogeneity between the effect sizes using known study and sample characteristics as exploratory variables in the specified meta-analytic regression model.

Qualitative Research Designs

Qualitative research is inductive and context-specific research that focuses on observing and describing a specific phenomenon, behavior, opinions, and events that exist to generate new research hypotheses and theories. The goals of qualitative research are to provide a detailed narrative description and holistic interpretation that captures the richness and complexity of behaviors, experiences, and events in natural settings. Thus, qualitative research is an inductive research process, logically emerging from the specific phenomena to general conclusions and theories about the phenomena based on data collected by observations, documents, physical artifacts, interviews, and focus groups.

Case Study

Case study is an in-depth examination and intensive description of a single individual, group, and organization based on collected information from a variety of sources, such as observations, interviews, documents, participant observation, and archival records. The goal of the case study is to provide a detailed and comprehensive description, in narrative form, of the case being studied.

Ethnographic Research

Ethnographic research is a qualitative research design that is used for studying social groups, cultures, and human interactions in natural cultural and social settings. The goal of the ethnographic study is to provide a detailed, in-depth, and holistic narrative description of the group and the cultural setting being studied. The primary ethnographic data collection methods are in-depth interviews and participant observation to comprehensively describe a cultural and social setting.

Phenomenological Research

Phenomenological research, or phenomenology, is a qualitative research method in which the researcher attempts to understand and explain how an individual or a group of individuals experience a particular phenomenon from the individual’s or individuals’ own perspective(s). The primary method of data collection used in phenomenology is in-depth interviews of individuals who have experienced the phenomenon.

Action Research

Action research is a systematic research inquiry conducted by teachers, principals, school counselors, managers, or any other educational or organizational practitioners in the educational and organizational setting to collect information about educational and organizational practices and operations to resolve matters of concern or a problem in a particular setting such as classroom, playground, library, department, or company. Simply stated, action research is a study conducted by educational and organizational practitioners to help them to develop alternative reflective practices that lead to positive changes within their educational and organizational settings.

Historical Research

Historical research is a systematic process for searching, exploring, summarizing, and reporting past information and events using primary and secondary sources of historical data to gain understanding of historical events, issues, and policies. Primary sources of historical data are the original firsthand artifacts, documents, observations, oral presentations, diaries, photographs, and audio-visual recordings of past events. Secondary sources are secondhand nondirect oral and written documentations of past events that are summarized and documented by others and not the original primary sources.

Grounded Theory Research

Grounded theory research is an inductive qualitative research design that is used for generating and developing theories and explanations based on systematically collected qualitative data. The data collection process in grounded theory research is usually an ongoing iterative process that starts with collecting and analyzing qualitative data that leads to tentative theory development. Then, more qualitative data are collected and analyzed that lead to further clarification and development of the theory. The qualitative data collection and further theory development process continues until the particular theory is developed that is “grounded” in the data.

Mixed-Methods Research Designs

Mixed-methods research designs involve research studies that employ both quantitative and qualitative research methodologies to address the proposed research questions. Thus, mixed research methods combine the deductive and inductive inquiries of the scientific research methods as well as use a variety of data collection and analysis methods. The quantitative and qualitative methods can be conducted concurrently or sequentially to address a research question or questions. The mixed-methods research designs require from the researcher a considerable amount of time and energy as well as training in both quantitative and qualitative research designs. However, one of the significant advantages of the mixed-methods research design is that it provides a more comprehensive and enhanced image of the research problem that is under investigation than would either one of the designs (quantitative or qualitative) by itself. Specifically, the mixed-methods research designs can be classified into three types: exploratory, explanatory, and triangulation.

Exploratory Mixed-Methods Research Designs

Using this design, the researcher first conceptualizes a qualitative research study. Second, the researcher collects and analyzes the qualitative data. Third, the researcher uses the findings from the qualitative data analysis to conceptualize a quantitative research study. Finally, the researcher collects and analyzes the quantitative data to validate the qualitative findings.

Explanatory Mixed-Methods Research Designs

Using this design, the researcher first conceptualizes a quantitative research study. Second, the researcher collects and analyzes the quantitative data. Third, the researcher conceptualizes a qualitative research study. Finally, the researcher collects and analyzes the collected qualitative data to clarify and enhance the quantitative research findings.

Triangulation Mixed-Methods Designs

Using this design, the researcher simultaneously conceptualizes quantitative and qualitative research studies. Then, the researcher simultaneously collects and analyzes both quantitative and qualitative data. Finally, the researcher uses the results from the quantitative and qualitative studies to validate findings from both studies.