While scientific journalism is not a side effect of the pandemic, its popularity is. Marketing departments are taking advantage of this popularity: When lean or agile don’t quite catch on as buzzwords, products and services are advertised as “scientifically proven.” But what is actually scientific? In some cases, universities have quite clear definitions.
Anyone who knows the definitions will quickly realize that not everything that is touted as scientific is by a long shot. We have compiled a small checklist that you can ask when someone claims that their figures, data and facts are scientifically verified. For the quality criteria of scientific work, I have not only relied on experience and gut feeling, but on reliable sources. I recommend for example “Scientific work… easy to understand” by Rödiger Voss. These criteria form a good basic framework for a thorough review according to scientific standards.
- Have the results been obtained independently of the researcher?
- Has the data been collected independently?
- Would other researchers reach the same conclusions?
- Is the methodological procedure standardized?
- Have the results been compared with results from other surveys of the same design?
- Is a distinction made between quantitative and qualitative aspects?
- Does the “taken” sample allow general statements to be made regarding the issue/research variable?
- Is the sample representative of the population?
- Am I measuring what I want to measure?
- Does the data actually represent what I want to measure, i.e., the variable I want to measure?
- Face Validity: What does “common sense” say?
- Expert Validity: Were experts consulted and involved?
- Predictive Validity: The validity of a prediction can only be seriously verified in retrospect; biased pseudo-research is subject to errors such as self-fulfilling prophecies or destroying prophecies.
- What is the accuracy/reliability of the measurements?
- Are the measured differences genuine or did measurement errors occur?
- Do the same results reoccur with repeated measurements? Is the data reproducible?
- Are the results thoroughly checked? For example, the following methods may be used:
- Re-test (test-retest is concerned with the degree of accuracy with which a particular characteristic is repeatedly measured).
- Split-half method (by random selection, the test results are divided into two equal halves)
- Parallel test (two or more tests on the same item, e.g. test variant A and variant B)
Honesty and probity
- Are the sources used credited? Citing sources makes it possible to verify which ideas and findings underlie a paper. In addition, sources can be used to document that the work is not plagiarism, i.e. an attempt to cheat.
- Are the general ethical standards of scientific work and research adhered to?