Validity evidence and its sources.

Developing new psychometric instruments.

Adapting existing psychometric instruments.

Examples with PISA 2022 data for SELF-ERAS 2025 workshop

SLR MLR

Bayesian Structural Equation Modeling (SEM) has received increasing interest due to its capacity to address challenges in the frequentist approach, such as nonconvergence, Heywood cases, small sample sizes, and inadmissible solutions. A defining feature of Bayesian SEM is the use of a priori distributions, which play a fundamental role in parameter estimation, model interpretation, and uncertainty quantification. While researchers may express skepticism regarding the subjectivity of priors, they represent a substantial advantage of Bayesian methods, enabling the integration of prior knowledge about parameters before observing the data. However, the choice and specification of a priori distributions remain an underexplored aspect of Bayesian SEM. This protocol outlines the goal of a scoping review designed to explore the application of a priori distributions in Bayesian SEM, with a particular focus on confirmatory factor analysis and full SEM models. Key aspects of a priori distribution usage will be examined, including their application across different dimensionality structures and model parameters. Special attention will be given to priors for variances, loadings, regression coefficients, and covariances, both in terms of distribution families and hyperparameter values, highlighting their impact on posterior distributions, model estimation, and performance. By synthesizing recent literature, this review will identify trends, challenges, and gaps in the use of a priori distributions within the Bayesian SEM framework. The findings aim to promote informed decision-making regarding prior elicitation and enhance the robustness of Bayesian SEM applications.

From Classical Theories to Modern Applications

This presentation by Jorge Sinval delves into the critical processes and guidelines for adapting psychometric instruments within organizational and work psychology contexts. It highlights the importance of adhering to international standards to ensure the validity evidence, and equivalence of adapted tests across diverse populations. Drawing from the International Test Commission (ITC) guidelines and other prominent frameworks, the slides outline best practices for pre-conditions, test development, confirmation, administration, scoring, and documentation during test adaptation. By addressing challenges such as cultural and linguistic differences and providing actionable criteria for evaluation, the presentation aims to promote ethical, transparent, and effective adaptation methodologies that meet the unique needs of specific populations and contexts.

This presentation, authored by Jorge Sinval, explores the nuanced process of psychometric instrument development within organizational and work psychology. It addresses the critical decision between adapting existing tools and creating new ones, emphasizing the importance of non-redundancy, validity, and comprehensive reporting. Drawing on established frameworks and guidelines from leading researchers, the slides provide actionable insights for choosing, evaluating, and constructing psychometric instruments tailored to specific contexts. The presentation also highlights questionable practices, strategies for instrument selection, and detailed steps for developing robust and reliable scales, ensuring their alignment with theoretical constructs and practical applications.

Validity definition, standards, sources of validity...

This presentation explores the fundamental role of psychometrics in organizational and work psychology, focusing on Classical Test Theory (CTT). CTT provides the foundation for understanding how psychological tests are developed, evaluated, and interpreted. Attendees will learn key concepts such as reliability, and measurement error, along with practical applications for improving assessments in workplace settings.

Bayesian Confirmatory Factor Analysis (BCFA) is a statistical method that incorporates prior distributions for model parameters, offering an alternative to traditional frequentist approaches like maximum likelihood estimation in Confirmatory Factor Analysis (CFA) (Hoofs et al., 2017). BCFA has been utilized in various fields such as psychology, medicine, and education to assess the latent structure of data and determine relationships between constructs (Gracia et al., 2017; Santirso et al., 2018; Liu et al., 2021). It imposes an a priori structure on the data, allowing researchers to confirm or refute specific models based on prior information (Falkenström et al., 2014). BCFA has been employed to examine factor structures, including correlated-factor models, bi-factor models, higher-order factor models, and single general-factor models (Liu et al., 2021). Additionally, BCFA has been used to test existing models across different populations, demonstrating its versatility and applicability in diverse settings (Heinzel, 2023). In the context of confirmatory factor analysis, BCFA provides a robust framework for model estimation, particularly in cases where traditional assumptions may lead to poor model fit (Pan et al., 2017). It allows for the incorporation of informative priors based on expert knowledge or flat uninformative priors, enhancing the analysis process (Bott et al., 2016). BCFA has also been compared with other estimation techniques like penalized maximum likelihood and Markov Chain Monte Carlo methods, showcasing its effectiveness in estimating models with small sample sizes (Lüdtke et al., 2020).

The Burnout Assessment Tool (BAT) is a recognized psychometric tool for measuring burnout. It introduces a novel approach to burnout as a multidimensional construct, consisting of two second-order dimensions — core and secondary symptoms. Previous studies have examined the psychometric properties of the BAT-23 and BAT-12 in Portugal. However, no research has yet explored the validity of the BAT-S (secondary symptoms version) in Portugal or the psychometric properties of the BAT-12 in Portuguese university students. This study seeks to evaluate the psychometric characteristics of the Portuguese versions of BAT-S in working professionals and BAT-12 in university students. The investigation focuses on the validity evidence based on the internal structure (dimensionality, reliability, and measurement invariance) as well as the validity evidence based on the relations to other variables. The study utilized three non-probabilistic convenience samples comprising two worker groups from cross-sectional designs: a public administration organization (n = 399) and various other organizations (n = 458), along with a group of university students from a Portuguese public university in a longitudinal design (nwave 1 = 369; nwave 2 = 234). For the BAT-S work version, the original BAT dimensionality was confirmed with all items displaying satisfactory mean square fit statistics (Minfit = 0.99; SDinfit = 0.18; Moutfit = 0.98; SDoutfit = 0.15). Adequate internal consistency was achieved (ωi ≥ .79), and measurement invariance was established between male and female workers. Validity evidence based on the relations to other variables showed correlations that align with the constructs' nomological network, with core symptoms exhibiting moderate/strong correlations with colleagues’ incivility, clients’ incivility, negative change, and bureaucracy, while the secondary symptoms presented weaker correlations with the same variables. For the BAT-12 student version, the original dimensionality was confirmed, demonstrating good internal consistency (ωi ≥ .79) and measurement invariance across sex and longitudinal measurements. In terms of validity evidence related to other variables, the BAT-12 scores displayed strong negative latent correlations with academic engagement, satisfaction with education, psychological capital, and social support while revealing strong positive latent correlations with psychological distress and dropout intention. The study's findings provide supporting evidence for the validity of the Portuguese versions of BAT-S and BAT-12. The promising validity evidence suggests that both tools are suitable for measuring burnout in workers and university students, respectively.

The Burnout Assessment Tool (BAT) has is gaining increased attention as a sound and innovative instrument in its conceptualization of burnout. BAT has been adapted to several cultures and countries, revealing promising validity evidence. This paper aims to present the psychometric properties of the Singaporean student and workers versions of the BAT in both the 12-item versions. BAT’s validity evidence based on the internal structure (dimensionality, reliability, and measurement invariance) and validity evidence based on the relations to other variables are the focus of research. A cross-sectional study was conducted with two non-probabilistic convenience samples: one from university students (n = 1350) and one from workers (n = 269). BAT’s original dimensionality was confirmed, items’ fit was satisfactory, good internal consistency estimates were obtained, and measurement invariance was achieved between students and workers, and between male and female participants. In terms of validity evidence based on the relations to other variables, burnout presented positive small/moderate latent correlations with kiasuism, conformity, and pessimism, while the latent correlations of burnout with optimism, and social support were negative small/moderate. Students’ cumulative GPA was shown to be related with academic burnout scores, while the final course GPA of the workers did not present a statistically significant correlation with job burnout. In conclusion, the psychometric properties of the BAT make this freely available instrument a promising way to measure and compare burnout levels of students and workers in Singapore.

The initial survey was conducted to gauge the foundational knowledge and experience that our postgraduate students in the Transport Engineering program have regarding the course topic, "Computational Psychometrics Applied to Engineering." The intention behind this preliminary assessment was to understand the students' current familiarity and proficiency with the subject matter. This survey considered various aspects such as their understanding of key concepts, prior exposure or experience in computational psychometrics, and its application in engineering. It also assessed their ability to integrate and apply this knowledge in real-world engineering scenarios. The results of this survey will play a crucial role in tailoring the course content and teaching methodologies to best suit the students' learning needs and to ensure an effective and comprehensive learning experience.

The Burnout Assessment Tool (BAT) has is gaining increased attention as a sound and innovative instrument in its conceptualization of burnout. BAT has been adapted to several cultures and countries, revealing promising validity evidence. This paper aims to present the psychometric properties of the Singaporean student and workers versions of the BAT in both the 12-item versions. BAT’s validity evidence based on the internal structure (dimensionality, reliability, and measurement invariance) and validity evidence based on the relations to other variables are the focus of research. A cross-sectional study was conducted with two non-probabilistic convenience samples: one from university students (n = 1350) and one from workers (n = 269). BAT’s original dimensionality was confirmed, items’ fit was satisfactory, good internal consistency estimates were obtained, and measurement invariance was achieved between students and workers, and between male and female participants. In terms of validity evidence based on the relations to other variables, burnout presented positive small/moderate latent correlations with kiasuism, conformity, and pessimism, while the latent correlations of burnout with optimism, and social support were negative small/moderate. Students’ cumulative GPA was shown to be related with academic burnout scores, while the final course GPA of the workers did not present a statistically significant correlation with job burnout. In conclusion, the psychometric properties of the BAT make this freely available instrument a promising way to measure and compare burnout levels of students and workers in Singapore.

t-test and ANOVA

Data Analysis with R: Descriptive and Inferential Statistics: Statistical Decision Theory

Data Analysis with R: Descriptive and Inferential Statistics: Graphics

Data Analysis with R: Descriptive and Inferential Statistics: Descriptive Statistics

Data Analysis with R: Descriptive and Inferential Statistics: Creating and Manipulating Datasets in R

Data Analysis with R: Introduction to R and RStudio

Data Analysis with R: Course Introduction

Descriptive statistics Plots

PIRLS 2016

Sample size of complete answers per administrative region for Brazil and Portugal.