QRM Conference 2022
The school of Quantitative Research Methods in Education held its annual conference on June 13-14 in Gothenburg
Keynote presentation
Methodological issues in research on effects of schools and teachers on student outcomes
Prof. Dr. Sigrid Blömeke, Director and Professor, University of Oslo, Centre for Educational Measurement (CEMO)
This talk will draw on three recent studies on effects of schools and teachers on student outcomes conducted with national and international data. The focus will be on the respective methodological challenges involved, in particular the theoretical modelling of relations and its transformation into measurement models. Each topic will be assigned 30–40 minutes including questions.
The first topic is the modelling of mediation processes and is based on Blömeke et al. (2022a). Opening up the black box: Teacher competence, instructional quality, and students’ learning progress (in Learning and Instruction).
The second topic is the modelling of moderation processes and is based on Blömeke et al. (2022b). The role of intelligence and self-concept for teachers’ competence (in Journal of Intelligence).
The third topic is the generalizability of results, for example in case of studies that include several countries, and is based on Blömeke et al. (2021). School innovativeness is associated with enhanced teacher collaboration, innovative classroom practices, and job satisfaction (in Journal of Educational Psychology)
Workshops
Analyzing large-scale assessment data using R–package RALSA
Dr. Plamen Mirazchiyski, International Educational Research and Evaluation Institute (INERI), Slovenia
June 14
The workshop will be divided into two parts. The first part will make a short introduction to ILSAs' complex sampling and assessment designs using TIMSS as an example. A specific stress will be put on the consequences of these designs on the analysis of studies' data using examples.
The second part will introduce the analysis software (RALSA) for analyzing data from ILSAs. The data preparation and analysis features will be explained. The overview will also include demonstrations on using both the command line and the graphical user interface. The remaining time will be devoted to guided hands-on training using the graphical user interface of the package. Through these guided exercises all features will be demonstrated. If the time permits, at the end of the workshop assignments will be given to the participants to complete on their own with assistance from the instructor.
The workshop will use data from Nordic countries participating in the IEA’s Trends in International Mathematics and Science study (TIMSS) 2019 (grades 4 and 9). Regardless of the study and the cycle, RALSA always applies the correct estimation techniques, given the study design and implementation.
By the end of the workshop the participants are expected to have gained the following:
- Knowledge, understanding and appreciation on the ILSAs design and methodology, as well as the statistical complexities and issues for analyzing their data;
- Knowledge and understanding on the computational routines used in ILSAs; and
- Skills to analyze data using an R package, tailored for ILSAs design.
Target group and previous knowledge requirements
The workshop is intended both for analysts who do not have yet the knowledge and experience using ILSAs’ data, as well as for researchers with more experience.
Previous experience with R is welcome, but not a prerequisite, RALSA has intuitive and easy to use syntax, as well as graphical user interface. Working knowledge on basic statistics is required. All materials for the workshop, including the software, will be provided free of charge. The participants will need a computer to install the software and perform the sample analyses.
An Introduction to Propensity Score Matching for Causal Inference
Dr. Isa Steinmann, Centre for Educational Measurement at the University of Oslo (CEMO), Norway
15 June 2022
This workshop will introduce how propensity score matching methods can be used to address cause and effect questions in empirical research.
Causal research questions aim to isolate effects of a treatment, independent of other effects and differences between the treated and untreated. The most straightforward way to answer causal research questions is to conduct randomized trials like in pharmaceutical studies, for instance. In many fields including education, randomized trials are however very difficult to conduct, due to practical, ethical, or financial reasons, among others. Therefore, educational research often has to rely on observational data, that is, information that stems from pure observations of educational processes and outcomes without an interference of the researchers. Under specific circumstances, it is however possible to isolate causal effects anyway. This workshop will focus on propensity score matching methods both from a theoretical and applied perspective. Example studies will help to understand the assumptions and prerequisites behind this method and to evaluate its scope critically.
This workshop will first revisit Rubin’s potential outcome framework, a model that formalizes cause and effect questions, and the issue of selection bias. Second, different propensity score matching and balance check methods will be introduced. Lastly, the estimation of effects and the central advantages of propensity score matching over other methods will be discussed.
Target group and previous knowledge requirements
“The workshop is an introduction to the topic of propensity score matching and is primarily targeted at researchers who have not yet worked with the method themselves. For more experienced participants, further literature will be provided. Basic knowledge of causal inference is welcome but not a prerequisite.
The R demonstrations are for illustrative purposes and will be designed so that participants without R experience can follow as well. Laptops and software are not necessary.”
Math Anxiety – negative impact and potential solutions
Jonatan Finell1, Ellen Sammallahti2, Hanna Eklöf1, Johan Korhonen2 & Bert Jonsson1
1Umeå University, Sweden, 2Åbo Akademi University, Finland
It is well established that math anxiety has a negative relationship with math performance (MP). The Attentional Control Theory (ACT) suggests that anxiety can negatively impact the attentional control system and increase one's attention to threat-related stimuli. Within the ACT framework, the math anxiety (MA)—working memory (WM) relationship is argued to be critical for MP. The first study is based on meta-analyses that provide insights into the relationship between MA and WM. Through database searches with pre-determined search strings, 1,346 unique articles were identified. After excluding non-relevant studies, data from 57 studies and 150 effect sizes were used for investigating the MA—WM correlation using a random-effects model. This resulted in a mean correlation of r = −0.168. The correlation varied as a function of different factors, such as: age, the use of numerical or alphabetical based cognitive instruments, or anxiety measures that are more cognitively focused rather than affectively. Applying a similar method, a second study synthesised published interventions (50 studies, 75 effect sizes) that attempt to reduce math anxiety. The results show that interventions on math anxiety report a moderate effect size for reducing math anxiety d = -0.46, and improving math performance d = 0.58. These effect sizes varied depending on intervention length and sample age. Categorisations of the intervention into (1) motivation, (2) emotion, and (3) cognitive support, revealed no significant between group differences
Understanding Globalized Primary Classrooms: A multivariate investigation of the associations between cultural and linguistic diversity, teacher-self-efficacy, classroom emotional climate and reading achievement in the Norwegian context
Jacqueline Peterson1, Maria Therese Jensen2, Njål Foldnes3
1,3Norwegian Reading Centre, University of Stavanger, Norway, 2Centre for Learning Environment, University of Stavanger, Norway
Teachers experience numerous demands in the classroom, including the need to recognize the uniqueness of students through differentiated instruction. In teaching reading, such differentiation becomes even greater in classrooms with culturally and linguistically diverse (CLD) students. Yet, research to date is limited on how the CLD in a classroom relates to teachers’ self-efficacy, classroom emotional climate and literacy skills. With the job-demands resource model (Demerouti and Bakker, 2011) as an overarching framework, this study aims to investigate CLD for its relation to teacher self-efficacy and students’ perceptions of classroom emotional climate and its subsequent association to reading achievement. In line with previous research and theory, we define cultural and linguistic diversity as a job demand and teacher self-efficacy as a personal resource. Classroom emotional climate is measured as students’ perception of the emotional climate of the class. Structural equation modeling will be used to investigate associations between the aforementioned variables on data drawn from the Two Teachers project (See Solheim et al., 2017). The sample for the current study consists of 150 classrooms containing 2880 students. Class size will be included as a moderating variable in the analysis, while reported SES and parents’ educational levels will be controlled for in the model. The results of the study will contribute to our understanding of how CLD classrooms are experienced by primary teachers, via self-efficacy, and students, through classroom emotional climate, and the potential implications of these experiences on learners’ reading achievement.
Timeseries analysis extends content analysis to exploring distribution of a topic among data
Jöran Petersson
Faculty of Education and Society, Malmö University, Malmö, Sweden
This poster shows that methods imported from timeseries analysis could benefit the use of content analysis through raising new research questions and allowing enhanced results by showing where, in a sequence of data, the studied phenomenon occurs. For methodological reasons, timeseries analysis in educational research has been effectively absent and, until recently, content analysis in educational research typically meant to look for the mere existence of some explored theme or for comparing their frequency in two data sets. A timeseries requires that the data are an ordered sequence of units of analysis. One example is the analysis of a classroom conversation where the data may follow clock-time. Another example is analysing exercises in a textbook, where the data may not follow clock-time since each exercise can require different amounts of time to solve. A first outcome of a timeseries analysis is a moving average diagram, which is a diagram of the same kind as temperature curves used in climate science for displaying temperature changes over time. For the case of analysing exercises in a mathematics textbook, this diagram shows the changes in the intensity of the explored learning object throughout the textbook. When analysing a classroom conversation, it allows the researcher to compare two moving average diagrams, each displaying where in the conversation two specified topics occur. Specifically, determining the correlation between two timeseries allows the researcher to explore how two learning objects or conversation topics interact with each other. Hence, timeseries analysis provides a new tool for the researcher.
