In most of the last half of the 20th century the goal of quantitative research in the humanities and very often in other areas as well, such as medicine, was to obtain statistically significant results. This is no longer the goal of quantitative research in the social sciences.
Although many researchers and statisticians have suggested greater emphasis on effect sizes and statistics like confidence intervals, we need a more radical change in how we understand what the goals of research are. In this video we propose that researchers adopt the view that research is fundamentally about model building and model testing. This concept is used in data mining and also the great Nobel prize winning physicist Richard Feynman described research in physics as making guesses and then carefully testing the guess. In modern parlance we can call this building models and testing models.
Although this paradigm for understanding the goals of research is simple and may sound very general, there are specific guidelines that are important for conducting research properly within this paradigm. In this video 10 guidelines are given for properly conducting research and reporting results. In some academic fields some of these guidelines are already being violated as researchers leave the old paradigm of seeking statistical significance to an alternative paradigm.
Ten guidelnes for best practices for quantitative research in the social sciences are:
1. Be clear about what stage your research is in from initial guesses to confirmation of a theory in a hypothesis test. Do not exaggerate the stage of the research.
2. State clearly whether you are building a model with training data or testing a model with testing data.
3. Data Mining: A set of predictor variables that is not consistent, i.e., contains unrelated predictors is unlikely to be validated with test data! Data mining expert Sam Roweis emphasized this.
4. Data Mining: A set of predictors with no basis in theory is “a shot in the dark”. If the effect sizes found are small, then this is very much a shot in the dark.
5. The benefits of qualitative research: Some researchers see a strict divide between qualitiative research and quantitative research. However, very often it is qualitative research that develops content expertise that provides the best basis for what Richard Feynman called the guess.
6. Extraordinary proposed relationships require extraordinary support.
7. Strive to remove selection bias from your data! This might be the most important guideline for best practices!
8. Make friends with your data and use data visualization.
9. Pay close attention to effect sizes, confidence intervals, measurement error, rates of false positives and false negatives, power of the statistical test, etc.
Use p-values as a guide.
P-values have become very intensely debated in recent years. Virtually everyone acknowledges that they have been abused. Swinging to opposite extreme by declaring them verboten is ill-advised.
10. Early exploratory research is now welcome.
Research into areas that have little research literature, and therefore the literature review section of a paper might be almost non-existent, would be heretical in the old paradigm.
Hopefully this video provides a clearer understanding of what the goals of research in the social sciences are in the 21st century, and how we can conduct research and report results in a responsible manner.
Note also that in this video only the transition from a search for statistical significance to a model-building-and-testing philosophical framework are considered. These approaches are part of the essentially positivist tradition in research. Other philosophical frameworks based on other traditions are important but are not the focus of interest in this video.
The lecturer, David Cochrane, conducts research in the fringe area of astrology and his guidelines are important for researchers in astrology as well as education, psychology, other social sciences, and other fields.