Penn State Worry Questionnaire-11 validity in Colombia and factorial equivalence across gender and nonclinical and clinical samples

Full reference: Ruiz, F. J., Monroy-Cifuentes, A., & Suárez-Falcón, J. C. (2018). Penn State Worry Questionnaire-11 validity in Colombia and factorial equivalence across gender and nonclinical and clinical samples. Annals of Psychology, 34(3), 451–457. https://doi.org/10.6018/analesps.34.3.30281

Study type: Psychometric measure validation; Factorial invariance analysis; Measurement property study

Background and objectives

The Penn State Worry Questionnaire (PSWQ) is considered the gold standard measure for assessing worry related to generalized anxiety disorder (GAD). However, debate exists regarding the factor structure of the instrument: some studies have found better fit with a one-factor structure (pathological worry), while others have reported two factors. This discrepancy is particularly relevant in Spanish-speaking contexts, where translated versions have shown inconsistent psychometric properties.

A significant problem in PSWQ research is the use of negatively worded items (e.g., "it does not worry me that..."), which have demonstrated lack of practical utility and fail to correlate adequately with other psychological variables. Some authors have suggested using only the 11 positively worded items (PSWQ-11), which generates better comprehension in Spanish speakers and improved psychometric properties.

This study had two primary objectives: (1) to explore the validity and psychometric properties of the PSWQ-11 in Colombian nonclinical and clinical samples, and (2) to examine measurement invariance (factorial equivalence) of the instrument across gender and clinical/nonclinical status. The research is framed on the premise that a valid instrument must measure the same construct consistently across different population groups.

Method

Participants

The study included two separate samples. Sample 1 consisted of 710 nonclinical participants (71.4% female), with ages ranging from 18 to 89 years (M = 27.51, SD = 10.18). Educational level varied: 44.8% had secondary school education, 34.9% were undergraduate students or technical education graduates, and 20.3% held graduate degrees. 7.4% of participants reported receiving psychological/psychiatric treatment, and 3.7% reported psychotropic medication consumption. All participants were Colombian and responded to an anonymous online survey distributed through social media, Facebook, and institutional websites.

Sample 2 consisted of 335 clinical participants (74% female), with ages ranging from 18 to 63 years (M = 27.40, SD = 9.93). These participants were evaluated at an institutional psychological consultation center in Bogotá during clinical psychology consultations. Most clinical participants reported emotional symptoms as the primary reason for consultation (91%), while 7.2% reported social skills deficits. Only 7.2% reported psychotropic medication consumption.

Experimental design

The study employed a cross-sectional correlational design to evaluate psychometric properties and multigroup factorial invariance analysis. The factor structure of the PSWQ-11 was examined, internal consistency evaluated, and measurement invariance analyzed at configural, metric, and scalar levels. The grouping variables were gender (male/female) and clinical status (clinical/nonclinical), allowing examination of whether the instrument's structure remained equivalent across these groups.

Materials and apparatus

Penn State Worry Questionnaire-11 (PSWQ-11). Version of the PSWQ including only the 11 positively worded items, eliminating the 5 negatively worded items. The original PSWQ was designed to assess the persistent and nonspecific worry characteristic of GAD, with scores ranging from 16 to 80. The Spanish version used was recommended by Sandín et al. (2009) after finding that negatively worded items were difficult for Spanish speakers to understand and showed poor psychometric properties.

Depression, Anxiety, and Stress Scales-21 (DASS-21). A 21-item self-report instrument with 4-point Likert scale assessing depression, anxiety, and stress symptoms. The Spanish translation was used to evaluate negative emotional symptoms and correlations with the PSWQ-11. The instrument has shown good psychometric properties in Colombian samples (Ruiz, García-Martín, Suárez-Falcón, & Odriozola-González, 2017).

Generalized Anxiety Disorder-7 (GAD-7). A 7-item self-report instrument with 4-point Likert scale designed as a diagnostic and severity measure of GAD. The Spanish translation was used to evaluate anxiety symptoms related to worry.

Ruminación and Rumination Response Scale - Short Form (RRS-SF). An instrument measuring the tendency to ruminate over depression symptoms. Used to evaluate expected correlations with the PSWQ-11.

Data were analyzed using LISREL software (version 8.71) for confirmatory factor analysis (CFA) and SPSS 20 for descriptive statistics and correlations.

Procedure

Sample 1 participants responded to an anonymous online survey distributed through social media, institutional web pages, Facebook profiles, and emails. The survey was titled "Survey of Emotional Health in Colombia" and was completed online at www.encuestasalud.com. After data collection, a general form was sent to participants explaining the study's purposes.

Sample 2 participants completed questionnaires at the beginning of treatment at a clinical psychology clinic. All participants provided informed consent and confirmed their voluntary participation anonymously.

In addition to the PSWQ-11, Sample 1 participants completed the DASS-21, RRS-SF, and selected DASS-21 sections. Other participants completed the PSWQ-11, DASS-21, and RRS-SF (N = 370), while others completed GAD-7 measures (N = 340).

Data analysis

Missing data imputation was conducted using weighted least squares (WLS) estimation with polychoric correlations, recommended for large samples with more than 20 items (Holaday, Chacón, Micheo, Barbero-García, & Vila-Abad, 2010). Bivariate normality was analyzed via chi-square test and RMSEA, with RMSEA values below 0.1 in all correlations (supporting polychoric correlation matrices).

Confirmatory factor analysis (CFA) of a one-factor model was conducted to evaluate internal structure validity. Models were compared using RMSEA differences (Δ RMSEA < .01), CFI differences (Δ CFI ≥ -.01), and NNFI differences (Δ NNFI ≥ -.01) per Cheung and Rensvold (2002) and Chen (2007) criteria. Ninety-five percent bootstrap confidence intervals (CI) were calculated to explore internal consistency of the PSWQ-11 in Samples 1 and 2, as well as the overall sample.

To examine measurement invariance, standard procedures (Cheung & Rensvold, 2002; Millsap & Yun-Tein, 2004) were followed, analyzing three models: unconstrained multigroup baseline model, metric invariance model (equality constraints on factor loadings across groups), and scalar invariance model (equality constraints on factor loadings and intercepts). Metric and scalar invariance were examined across gender and across clinical/nonclinical participants using RMSEA, CFI, and NNFI changes as decision criteria.

Results

Descriptive psychometric quality of items

Table 1 presents descriptive statistics and corrected item-total correlations for both samples. All items showed significant correlations with total scores, with corrected correlations ranging from .67 to .88 in Sample 1 and .65 to .85 in Sample 2, indicating excellent item discrimination.

Regarding internal consistency, Table 2 reports alpha and omega coefficients with 95% confidence intervals (CI). In Sample 1 nonclinical (n = 710), alpha was .949 (CI [.942, .954]) and omega was .949 (CI [.943, .954]). In Sample 2 clinical (n = 335), alpha was .933 (CI [.920, .942]) and omega was .934 (CI [.921, .944]). In the total sample (n = 1045), alpha was .951 (CI [.946, .955]) and omega was .952 (CI [.947, .956]). These values indicate excellent internal consistency across all cases, well above the minimum acceptable threshold of .70.

Mean scores were M = 27.47 (SD = 10.44) in Sample 1 nonclinical and M = 36.29 (SD = 10.13) in Sample 2 clinical, showing statistically significant difference between clinical and nonclinical participants (t = -12.80, p < .001).

Validity evidence based on internal structure: Dimensionality

Results of the chi-square test for bivariate normality showed this assumption was accepted in 22% of correlations. However, RMSEA values were below 0.1 in all correlations, supporting the use of polychoric correlation matrices to conduct CFA.

The one-factor PSWQ-11 model showed excellent fit. Goodness-of-fit values were χ²(44) = 430.73, p < .01; CFI = .99; NNFI = .98; and RMSEA = .092, CI [.084, .099]. CFI and NNFI above .95 indicate very good fit, while RMSEA = .092 is considered acceptable per Kelloway (1998) guidelines, though slightly higher than the stricter standard of Hu and Bentler (1999). Overall, the one-factor model fit was highly satisfactory.

Standardized factor loadings ranged from .72 to .94 as shown in Figure 1, indicating all items contributed significantly to the pathological worry construct. Items 7 and 6 showed the highest loadings (.94 and .89, respectively), while Item 10 showed the lowest loading (.79).

Measurement validity: Factorial invariance

Table 3 presents results of metric and scalar invariance analyses across clinical/nonclinical groups and genders.

Invariance across clinical and nonclinical samples: The unconstrained multigroup baseline model showed RMSEA = .0955, CFI = .984, NNFI = .984. The metric invariance model (factor loadings constrained equal) showed RMSEA = .0952 (Δ RMSEA = -.0003), CFI = .983 (Δ CFI = -.001), NNFI = .981 (Δ NNFI = -.000), satisfying all minimal change criteria. The scalar invariance model (factor loadings and intercepts constrained) showed RMSEA = .1010 (Δ RMSEA = .0058), CFI = .979 (Δ CFI = -.004), NNFI = .078 (Δ NNFI = -.003), also satisfying minimal change criteria below .01.

Invariance across genders: The baseline model showed RMSEA = .0893, CFI = .985, NNFI = .985. The metric invariance model showed RMSEA = .0884 (Δ RMSEA = .0009), CFI = .984 (Δ CFI = -.001), NNFI = .982 (Δ NNFI = .001). The scalar invariance model showed RMSEA = .0861 (Δ RMSEA = .0023), CFI = .983 (Δ CFI = -.001), NNFI = .983 (Δ NNFI = .001), satisfying all criteria.

These results indicate the PSWQ-11 measures the same construct (pathological worry) equivalently across clinical and nonclinical participants, and across males and females.

Validity evidence based on relationships with other variables

The PSWQ-11 showed significant correlations with all concurrent criterion scales examined (Table 4). Positive correlations were found with DASS-21 Depression (r = .58, n = 710, p < .001), DASS-21 Anxiety (r = .46, n = 242, p < .001), DASS-21 Stress (r = .67, n = 710, p < .001), and GAD-7 (r = .73, n = 340, p < .001). Positive correlations were also found with RRS-SF Rumination (r = .65, n = 370, p < .001) and RRS-SF Reflection (r = .49, n = 370, p < .001). A negative correlation was found with the Acceptance and Action Questionnaire (AAQ; r = -.33, n = 370, p < .001), consistent with the expected relationship between worry and experiential avoidance. The magnitudes of these correlations were generally moderate to large, supporting convergent validity of the instrument.

Regarding clinical discrimination, the PSWQ-11 significantly discriminated between clinical and nonclinical participants. Clinical participants showed significantly higher scores (M = 36.29, SD = 10.13) than nonclinical participants (M = 27.47, SD = 10.44), t = -12.80, p < .001. There were no statistically significant differences in PSWQ-11 scores between males and females in any sample, suggesting the instrument functions equivalently across genders.

Discussion and conclusions

This study provides strong evidence that the abbreviated Spanish version of the PSWQ (PSWQ-11) is a valid and reliable instrument for measuring pathological worry in Colombian samples. Findings support previous recommendations by Sandín et al. (2009) to use only the 11 positively worded items, eliminating the 5 negatively worded items that demonstrated lack of practical utility in Spanish-speaking populations.

A particularly important finding was the factorial equivalence of the PSWQ-11 across gender and clinical/nonclinical participants. These measurement invariance results are critical because they justify valid score comparison across these groups. In the absence of data supporting PSWQ factorial equivalence, it would not be legitimate to compare scores between male and female samples, or between clinical and nonclinical participants. The current study extends previous findings by Brown (2003) to Spanish-speaking contexts and clinical populations, confirming the PSWQ measures the same construct across these groups.

Several limitations merit mention. First, systematic diagnostic status information was obtained only from clinical participants. Second, certain validity aspects were not fully analyzed in the current study (e.g., discriminant validity, sensitivity to treatment effects, etc.). Third, due to time constraints, not all participants completed all questionnaires, resulting in slightly reduced sample sizes for some correlation analyses. Fourth, the sample was predominantly female (approximately 300 male/female participants in each category), though this was sufficient to conduct robust factorial invariance analyses. Finally, although all participants could respond to all questionnaires, some chose not to complete certain questionnaires, so only data from those completing all measures were used.

In conclusion, the abbreviated Spanish version of the PSWQ (PSWQ-11) recommended by Sandín et al. (2009) can be used as a valid measure of worry related to GAD in Colombia. The instrument demonstrates excellent internal consistency (α = .95), acceptable one-factor structure, and measurement invariance at metric and scalar levels across gender and clinical/nonclinical participants. Additionally, the PSWQ-11 showed validity evidence based on relationships with measures of emotional symptoms (depression, anxiety, stress) and ruminative processes (rumination and reflection). The measurement invariance data found in this study across multiple contexts justifies comparison of scores between males and females, and between clinical and nonclinical participants.

Significance and contribution

This study contributes significantly to the field of worry measurement in Spanish-speaking contexts, specifically in Colombia. By demonstrating that the abbreviated PSWQ version (PSWQ-11) provides solid psychometric properties and factorial equivalence across genders and clinical status, this work validates the use of this instrument in research and clinical practice. This has important implications for diagnostic assessment in GAD, clinical research on worry mechanisms, and assessment of therapeutic change in acceptance and commitment therapy protocols and other psychotherapeutic approaches targeting pathological worry reduction. The finding that the positively worded version shows better properties than the original version with negative items also has implications for translation and adaptation of psychometric instruments to Spanish-speaking languages.