It also implies that the domain context of a test has no or almost no influence on the assessment of scientific reasoning skills. The position of domain generality entails that the core set of scientific reasoning skills is very similar or the same in domains like chemistry or physics. Whether scientific reasoning is domain-general or domain-specific has been an issue for decades. This article will focus on one aspect of the assessment of scientific reasoning: the domain generality of scientific reasoning tests when applied to higher education students. While there is no shortage of scientific reasoning tests in the literature, a lot of them are of unknown quality and many assumptions remain untested ( Opitz et al., 2017).

These skills are not only relevant in conducting scientific studies but also in professional practice. Not all conceptualizations of scientific reasoning include the same skills, but there is considerable overlap and at its core the set comprises skills like formulating questions, formulating hypotheses, gathering evidence, evaluating evidence, explaining results, and communicating results ( Fischer et al., 2014 National Research Council, 2012). The assessment of scientific reasoning has been highlighted as a particularly important challenge of this century ( Osborne, 2013). We advise against using it in high-stakes situations involving domain comparisons. Taken together, our analyses indicate that the scientific reasoning test is neither entirely domain-general, nor entirely domain-specific.

Thus, the tree- and lasso-based methods provide a clear added value to test evaluation. Those methods indicated instead that all items were easier for physics students compared to biology students. We did not find this bias by using methods based on classical test theory. All three DIF analysis methods indicated a domain bias present in about one-third of the items, mostly benefiting biology students. We gave the scientific reasoning test to 507 university students majoring in physics, biology, or medicine.

Additionally, we evaluated the usefulness of these new, tree- and lasso-based, methods to analyze DIF and compared them with methods based on classical test theory. This study applied three recently developed methods of analyzing differential item functioning (DIF) to evaluate the domain generality assumption of a common scientific reasoning test. A significant problem that assessments of scientific reasoning face at the level of higher education is the question of domain generality, that is, whether a test will produce biased results for students from different domains.

Lmu spss code