Experimental Design Requirements and Limitations Paper

Experimental and Quasi-Experimental Design

As you are learning, quantitative designs are designated by type according to their purpose. At the broadest level of categorization, designs are experimental, quasi-experimental, or non-experimental in nature. Experimental designs are those meant to measure a cause-effect relationship and can do so because subjects are randomly assigned to control and experimental conditions making the groups equivalent at the onset. Quasi-experimental studies are also intended to show a cause-effect relationship; however, this design is employed when random assignment is not possible or would pose an ethical issue. Therefore, the fundamental difference between experimental and quasi-experimental designs is random assignment. In both designs, the researcher manipulates one or more independent variables. In experimental designs, the researcher also uses random assignment as described above, and therefore groups are assumed to be equivalent prior to any intervention. 

Experimental Designs

True experimental designs consist of at least one randomly assigned control group and experimental group.  Randomized experiments are considered the gold standard in experimental design because they reduce the plausibility of alternative explanations for the effects a researcher is observing, therefore possessing a very high level of internal validity. The key design feature is the random assignment of participants to conditions based only on chance.  

It is very important not to confuse random assignment with random sampling. The purpose of randomness differs for these two procedures. Random assignment allows the researcher to control for pre-existing differences between participants that could influence the results and jeopardize the validity of the research. Through random assignment, you are able to create equivalent groups before conducting any experiment or intervention so that when you compare the groups after the intervention, differences found between the groups are more likely due to the intervention versus pre-existing differences. For example, if you have two groups of employees (one that is a control group and one that participates in an IT security training), and you compare them after the experimental group participants in the training, you want to make sure that any differences between those groups in IT security knowledge is due to the training and not knowledge differences that existed before the intervention.

Quasi-Experimental Designs

Quasi-experimental designs are one of the more frequently used designs you are likely to encounter. Well-designed quasi-experimental designs can show clear causal relationships between variables but lack the randomization that is required of a full experimental design. A quasi-experimental design differs from experimental design in that it involves a less controlled environment and does not use randomly assigned independent variables. This design is often used when it is not feasible to assign participants to different conditions.

Quasi-experimental designs have specific requirements. For example, in order for the design to qualify as quasi-experimental, the design must either include a control or comparison group and pretest for the outcome of interest (e.g., dependent variable). One exception to this rule is if the design includes only one group, but the participants are measured in a time-series fashion (longitudinally), and the treatment is implemented, discontinued, and then implemented. These are typically called regression discontinuity designs or interrupted time-series designs.

Pre-experimental designs can include a design with a pretest with a posttest without a control group or a design with a control group but no pretest. In other words, the independent variable is manipulated, but there is either no pretest or no control group. Pre-experimental designs have low internal validity, given the lack of rigor of the design. Refer to the figure below for determining which design is applicable (experimental, quasi-experimental, or pre-experimental).

Figure 1.

Study with a manipulated variable a true experiment, quasi-experiment, or pre-experiment (Shutay, 2019)

One of the design features that a researcher can manipulate is the use of control groups. Adding a control group can help the researcher improve the validity of inferences by being able to describe what would have happened to the treatment group had they not received the treatment.

The pretest is another design feature that the researcher can manipulate. Pretests (along with post-tests) help the researcher by adding observations over time and, with the addition of a control group, allow the researcher to discern whether the groups differed at the beginning of the study. These manipulations help lower the possibility of internal validity threats.

If we go back to the IT training example, if the two groups were not equivalent due to a lack of random assignment, we would need to consider baseline performance.  Figure 2 provides an example where two groups (experimental and control) are not equivalent prior to the intervention and therefore pre-existing differences need to be statistically controlled through the researcher’s data analysis techniques. Specifically, the researcher could use a repeated-measures ANOVA or an ANCOVA to control for baseline performance. Without considering baseline performance, one would incorrectly conclude that the training was not effective.

Figure 2.

Employee Baseline and Post Treatment Performance (Shutay, 2020)

Another type of quasi-experimental design is the interrupted time-series design. This design is one in which a number of observations are made on the same variable over time and is often used to assess the impact of some treatment. The important design feature of the interrupted time-series design is that several observations are made before the treatment is implemented, during the treatment period, and after the treatment period. This design is a strong alternative to a randomized design when such a design is not feasible. Multiple observations help the researcher decrease validity threats and offer the researcher more information regarding whether a treatment is working (or perhaps, if the treatment only worked for the first few weeks or only works after several weeks of implementation), and whether any improvements last after treatment. The interrupted time-series design is fundamentally different than the time-series designs discussed in Week 5 given that an independent variable is manipulated. 

References

Shutay, J. (2020). Study with a Manipulated Variable a True Experiment, Quasi-Experiment, or Pre-Experiment  [image].

Shutay, J. (2020). Employee Baseline and Post-Treatment Performance [image]. 

Assignment: Evaluate Experimental Design Requirements and Limitations

Instructions

For this assignment, write a paper that evaluates the research conducted in three peer-reviewed articles on a technology topic that employs an experimental research design. Address all components for each article before moving on to the next article. Your paper should address the following components:

• What was the research problem?
• What variables were analyzed, and what were the hypotheses tested?
• What types of statistical analyses were used?
• Which, if any, of the general strengths and weaknesses of experimental designs did the study demonstrate? Are there any threats to validity? Explain.
• Evaluate the quality of the research study
  • Was an experimental design the most appropriate design and why or why not?
  • How credible are the conclusions of the study?

REFERENCE 

https://methods.sagepub.com/book/sage-hdbk-quantit…

https://sk.sagepub.com/

https://methods.sagepub.com/video/an-introduction-…