Generalized Total Entropy Fit Index: A new fit index to compare bifactor and correlated factor structures in SEM and network psychometrics

New Pre-Print: "Generalized Total Entropy Fit Index: A new fit index to compare bifactor and correlated factor structures in SEM and network psychometrics" (link: https://osf.io/preprints/psyarxiv/5g3hb), by Golino, Jimenez, Garrido, & Christensen. Read the extended summary below.

Bifactor models are popular but beware of bias in fit indices. Recent studies show that traditional indices (like CFI, RMSEA) may favor bifactor over correlated traits models, even if the true data generation mechanism is a correlated traits model. This affects model selection and interpretation in psychology research.

Recent research by Kan et al. (2024) warns against using approximate fit indices like CFI and RMSEA when comparing models, as they may favor more complex bifactor models. They suggest using relative fit indices such as BIC and AIC instead. Usually, to address potential bias in bifactor models, researchers are urged to consider a broader range of factors beyond fit indices, including theory and factor interpretability. However, the field has largely avoided directly addressing the limitations of traditional fit indices, either by advocating for a more comprehensive approach or by avoiding bifactor models altogether. This highlights a significant gap in the field and the need for new fit indices that are not based on traditional covariance matrix comparisons.

# Limitations of Recent Simulation Studies

Recent studies investigating fit indices for comparing correlated traits and bifactor models have faced limitations in their approach. Cudeck and Henly (1991) identified four critical types of discrepancy in statistical modeling: Discrepancy due to Approximation (DA), Discrepancy due to Estimation (DE), Sample Discrepancy (SD), and Overall Discrepancy (OD). However, research by Greene et al. (2019), Morgan et al. (2015), Bonifay and Cai (2017), and Kan et al. (2024) has predominantly focused on generalizability (minimizing OD) without adequately addressing verisimilitude (minimizing DA). This narrow focus potentially overlooks crucial aspects of model performance, particularly when examining various sample sizes. Furthermore, these studies have not incorporated error at the population level in their simulations, limiting their ability to capture realistic evidence of both estimation-based and approximation-based divergence. A more comprehensive approach to evaluating fit indices in bifactor modeling should consider both generalizability and verisimilitude, while also incorporating population-level error to provide a more accurate representation of real-world scenarios.

# Present Study:

This paper introduces the generalized total entropy fit index (GenTEFI), an extension of the total entropy fit index (TEFI) developed by Golino et al. (2021). While TEFI showed high power in detecting correct population structures, it was limited to single-level structures. GenTEFI expands this capability to accommodate more complex two-level structures, including bifactor models with multiple general factors. In the paper we implemented three simulation studies to address limitations in recent research by generating data from population models with error, allowing for the quantification of fit indices' accuracy in capturing both sampling and approximation discrepancies. These simulations compare GenTEFI's performance against traditional fit indices at both sample and population levels, with a focus on differentiating between correlated traits and bifactor structures under various data generation mechanisms.

Simulation 1: True data generation mechanism: correlated traits model with population error.

Simulation 2: True data generation mechanism: bifactor model with population error.

Simulation 3: True data generation mechanism: bifactor model with population error, comparing the correct structure to over- or under-factored structures.

# Results:

The study's results demonstrate that the Generalized Total Entropy Fit Index (GenTEFI) is the most effective measure for distinguishing between correlated traits and bifactor structures, both at the population and sample levels, regardless of the true data generation mechanism. GenTEFI showed a balanced accuracy of 94-95% across all tested conditions, outperforming other indices such as BIC, AIC, RMSEA, and SRMR. While BIC and AIC performed well at the sample level, they struggled at the population level, indicating poor capacity to detect Discrepancy due to Approximation (DA). GenTEFI proved particularly effective when the data generation mechanism was a correlated traits model, though it showed some limitations with high loadings, moderate factor correlations, and fewer items per factor.

Balanced Accuracy (Simulation 1):

# Hit Rate (Percent Correct) Simulation 1:

# Hit Rate (Percent Correct) Simulation 2:

The second simulation compared bifactor structures (reflecting the true data generating mechanism) with correlated traits structures. It is well known that traditional fit indices have a bias towards bifactor models, and therefore it is expected that in the second simulation these fit indices perform well.

At the population level, BIC, AIC, CFI, RMSEA, and SRMR achieved the highest mean percent correct (100%) for both lower and higher population error. GenTEFI had a mean percentage correct of 95.83% for both error levels at the population.

At the sample level, CFI, AIC, and SRMR presented a mean percentage of 100% for higher and lower population error, respectively, while RMSEA presented percent correct scores of 99.98% and 100%, for higher and lower error. BIC presented percent correct scores of 95.23% and 95.94%, for higher and lower population errors, while GenTEFI presented percent correct scores of 93.98% and 94.46%.

In the second simulation, with a true data generation mechanism being a bifactor model, all fit indices presented a high or very high accuracy for the population and sample levels. This indicates that all fit indices are good at capturing DE, DA, and OD when the true data generation mechanism is a bifactor model.

# Hit Rate (Percent Correct) Simulation 3:

In the third simulation, which compared correct bifactor structures to misspecified ones, RMSEA emerged as the most accurate index, with all fit indices showing high accuracy (80% or above at the population level and over 90% at the sample level). GenTEFI, BIC, and RMSEA performed particularly well in identifying overfactored general factor structures, especially with moderate loadings on the general factor. Based on these findings, the researchers recommend using GenTEFI as the primary metric for deciding between correlated traits and bifactor models. If GenTEFI suggests a bifactor model, traditional fit indices like RMSEA and BIC can be used in combination with GenTEFI to determine the most appropriate structure among competing bifactor models.

# Empirical Example 1:

We analyzed data from the 12-item Experiences in Close Relationships (ECR-12; Lafontaine et al., 2015)scale to compare the fit of a correlated-factors model against a bifactor model. The ECR-12 is purported to measure two attachment factors, anxiety and avoidance, each measured by six items. In general, the ECR published literature has not considered a bifactor model of attachment for the responses to this scale. The data was obtained from the Open Source Psychometrics Project (openpsychometrics.org), which contained full responses by 27,883 persons from the United States.

The items were responded via a 5-point Likert scale with options strongly disagree (1), disagree (2), neither agree nor disagree (3), agree (4), and strongly agree (5). Cronbach’s alpha reliability was .858 for avoidance and .840 for anxiety.

The factor models were estimated using robust maximum likelihood. The two-factor CFA model produced a good fit to the data with CFI = 0.95, RMSEA = 0.07, and SRMR = 0.05. Additionally, AIC = 1,825,558.68 and BIC = 1,825,558.68 for this model. The standardized loadings on the avoidance factor ranged from .54 to .80 (M = .70), while the loadings on the anxiety factor ranged from .50 to .79 (M = .6785). The two attachment factors were estimated to be approximately orthogonal, with a correlation of -0.002.

The bifactor model produced an even better fit to the data according to all factor-analytic fit indices, with CFI = 0.97, RMSEA = 0.05, and SRMR = 0.03, AIC = 1,819,293.09, and BIC = 1,819,293.09 for this model.

However, the standardized loadings did not support this bifactor structure as none item of the avoidance dimension had loadings above .12 on the general factor (M = -0.01), while the anxiety items had a mean loading of .67 on it. Conversely, the anxiety items did not load on the anxiety group factor well with a mean loading of -0.13, while the avoidance items had a large mean loading of .70 on the avoidance group factor.

When Exploratory Graph Analysis in the ECR data, the two-dimensional structure of the avoidance and anxiety items became very clear (see Figure below).

The generalized total entropy fit results point to the first-order two-community structure with avoidance and anxiety factors (TEFIfirst?order = -9.74) as fitting better than the bifactor structure with a general factor (TEFIsecond?order = -6.22).

The lower TEFI value for the first-order structure, compared to the second-order TEFI value, indicates that a two-community structure presents a lower level of disorganization, fitting the data better.

# Importance of the Empirical Example 1:

The two-factor structure aligns closely with attachment theory as developed by Bowlby and Ainsworth. This theory conceptualizes attachment in terms of anxiety about abandonment and avoidance of intimacy. Lafontaine et al. (2015) points that the two-dimensional factor structure of the ECR has been observed consistently across various populations, cultures, and languages. This robust replication supports the validity of the two-factor model. The paper emphasizes that anxiety and avoidance are designed to be orthogonal (uncorrelated) dimensions. This independence is theoretically important and has been supported by research, although small correlations are sometimes observed. Therefore, a bifactor model does not make theoretical sense, although it is shown to fit the data better than a correlated-traits structure with two factors, according to the traditional indices (CFI, RMSEA, SRMR, AIC, and BIC). The new Generalized Total Entropy Fit index, on the other hand, identified the two-factor structure as fitting the data better than the bifactor structure.

?

?