Latent Variable Modeling ED231E, Spring 2004 …Latent Variable Modeling ED231E, Spring 2004 05.12.04 Page 1 of 11 Assignment 4 For this assignment, I have selected a growth model

Latent Variable Modeling ED231E, Spring 2004

05.12.04 Page 1 of 11

Assignment 4 For this assignment, I have selected a growth model with parallel processes for variables measuring quantitative literacy and whether students report being “good at math.” The folk-rationale for this model (without any particular grounding in self-efficacy or attribution theory, etc.) is that if a student has high quantitative skills, and demonstrates these skills by performing well on the quantitative literacy items, the student will also tend to report being good at math. It is posited that quantitative skills is a variable that might capture the overall math ability or skills of a particular student and that the item regarding being good at math will relate highly to these overall abilities or skills. The first step to conducting a growth model with parallel processes is to estimate a growth model for each process separately (determine the shape of the growth curve, fit the model without covariates, modify the model). After estimating these processes separately, a joint analysis of both processes will be conducted and then covariates will be added. Growth Model for Process 1: Quantitative Literacy from Grades 7-10 Excluding 14 cases that have missing data on all four time points, descriptives are included in Table 1. There are a total of 3102 different cases included for this entire process. But for each time point, a different number of subjects are used in the estimation. Although not necessarily true, I made the strong assumption that data is missing at random. N Mean Std. Dev. Variance Minimum Maximum Grade 7 3065 50.313 10.119 102.392 35.04 79.16 Grade 8 2581 54.761 11.688 135.601 31.69 83.42 Grade 9 2241 60.156 12.847 165.047 37.36 89.85 Grade 10 2040 64.831 13.946 194.482 37.43 89.65 Table 1. Descriptive statistics for quantitative literacy IRT scores from Grade 7 to Grade 10. As can be seen from the mean scale scores, there is a small, positive increase in the average quantitative score from Grade 7 to Grade 10. Qualitative literacy scores are highly related (see Table 2). Since we don’t observe large increases in scores, it suggests that this construct of quantitative literacy measures a general math ability and not specific to any particular type of course or instruction. For the purpose of this assignment, I included all students even if they dropped out or were missing data at any time point. If I excluded students if they dropped out or were missing data at any time point, the sample size would decrease to 1659 students. Since I’m not sure about why these students dropped out or if they are missing data points at random, I decided to keep all students in the analyses. Another option would be to use only the observations for students who did not drop out during the course of the study which would allow us to examine students who were missing data at random (and would exclude students who dropped out at some point in the study because drop out might in some way be related to the outcomes of interest).


05.12.04 Page 2 of 11

Grade 7 Grade 8 Grade 9 Grade 10 Grade 7

- - -

Grade 8 0.710 (n=2550)

- - -

Grade 9 0.669 (n=2214)

0.817 (n=2037)

- - -

Grade 10 0.656 (n=2019)

0.760 (n=1837)

0.844 (n=1821)

- - -

Table 2. Correlations for quantitative literacy IRT-based scale scores from Grade 7 to Grade 10. All correlations are significant of the 0.01 level (2-tailed).

Individual growth trajectories for a select group of 10 students suggest a positive linear trend (Figure 1).

Figure 1. Observed values of quantitative literacy performance for 10 selected students from Grade 7 to Grade 10

Assuming linear growth, I tested several different models. The first model used fixed time scores and no covariates. The intercepts are fixed at 1 and the slopes are considered equidistant and linear. In addition, the intercepts of the slopes are fixed at 0. The chi-square test of model fit did not indicate a good fit (but as discussed in class and in previous assignments, this does not take into consideration the nested nature of the data, so significant values are interpreted with caution). The CFI and TLI are high and the SRMR is low but the RMSEA 90% confidence interval which suggests that this model can be improved.


05.12.04 Page 3 of 11

The modification indices, suggest that correlating qlt8 with qlt9 would significantly improve the fit. In this second model (Model 2), I correlated these residuals. Table 3 compares these two models. Model 2 fits the data better than Model 1. The chi-square is not significant (which is good), the AIC and BIC are lower than Model 1 and the RMSEA 90% confidence interval includes 0. The SRMR is low and the CFI and TLI are higher than Model 1. The chi-square difference test (114.761-3.116=111.645 with 1 df) suggests that there is an actual improvement from Model 1 to Model 2. The loglikelihood test to compare the two models gives us a value of 111.644 with 1 df which is another piece of evidence that there is significant improvement in fit in Model 2 from Model 1. Process 1, Model 1 Process 1, Model 2 Chi-square value 114.761 3.116 df 5 4 p-value 0.0000 0.5381 CFI 0.984 1.000 TLI 0.981 1.000 RMSEA 0.084 0.000 SRMR 0.079 0.012 Akaike’s Information Criterion (AIC) 70587.423 70477.777 Schwarz’s Bayesian Criterion (BIC) 70641.781 70538.175 Loglikelihood H0 value -35284.711 -35228.889 Table 3. Indices of model fit for Model 1 and Model 2 for quantitative literacy from Grades 7-10. Process 1, Model 1: fixed time scores, intercepts fixed at 1, slopes are equidistant and linear, intercepts of the slopes are fixed at 0 Process 1, Model 2: fixed time scores, intercepts fixed at 1, slopes are equidistant and linear, intercepts of the slopes are fixed at 0, qlt8 and qlt9 correlated

Figure 2 provides a schematic of Model 2 for quantitative literacy growth from Grades 7 to Grade 10. Note that the residuals are correlated for Grades 8 and 9. A substantive reason why these residuals might be correlated is that is that even though there is a transition between middle school and high school, there is still a strong relationship in terms of errors on these items because students aren’t affected much with instruction between these two grade levels. Residuals associated with their performance in these two grade levels tend to be related because they are dealing with other issues and probably aren’t concerned with performance on these exams.


05.12.04 Page 4 of 11

Figure 2. Model 2 includes correlated residuals for Grade 8 and 9. The R2 values for the observed values range from 0.783 (qlt7) to 0.939 (qlt10). These high values indicate that the latent variables are measuring these observed variables well. The relationship between the initial starting level (performance in grade 7) and growth is positive and significant which indicate that students who perform well on these quantitative items in grade 7 also tend to grow faster over the years. Students who start lower in grade 7 do not grow as fast as students who start higher. This is not surprising given that students who are generally better at math tend to take courses that are more rigorous and challenging than students who start off lower. Math is a cumulative in that, you can’t perform well in Algebra II without doing well in Algebra I. Quantitative literacy items capture this general math ability, so students who do well in math initially will probably continue to do well in math over the years. The estimated mean change from Grade 7 to Grade 8 is 4.531. Table 4 compares the observed means and the estimated means. There is very little difference between the observed and estimated values which provides further evidence that this model fits the data well. Grade Time Score Estimated Means = intercept + slope(time score) Observed Means

7 0 50.286 + 4.531(0) = 50.286 50.313 8 1 50.286 + 4.531(1) = 54.817 54.761 9 2 50.286 + 4.531(2) = 59.348 60.156 10 3 50.286 + 4.531(3) = 63.879 64.831

Table 4. Estimated and observed means for Process 1. There is greater variation in terms of where students start off in Grade 7 and the slopes as they progress but in general, it is not too surprising that there is a general positive linear change in scores because one would expect students to acquire greater proficiency in mathematics as they progress. This brings up the issue of whether there is some systematic reason for the missing data points (students who don’t do well tend to drop out of school). The linear change in scores


05.12.04 Page 5 of 11

is also somewhat expected because I am assuming that quantitative literacy is something that has to do with a construct of general math ability. Growth Model for Process 2: Being “Good at Math” from Grades 7-10 Similar to the first process of looking at change in quantitative literacy from Grade 7-Grade 10, this second process, being good at math, excludes only those cases (9 cases) that have missing data at all time points, descriptives are included in Table 5. Since this is a parallel processes analysis, I multiplied the scores by 10 to put it on a similar scale as the quantitative literacy scores. N Mean Std. Dev. Variance Minimum Maximum Grade 7 3057 22.45 10.436 108.900 10 50 Grade 8 2670 22.52 10.437 108.931 10 50 Grade 9 2353 22.69 9.765 95.353 10 50 Grade 10 2252 23.97 10.359 107.299 10 50 Table 5. Descriptive statistics for “being good at math” responses from Grade 7 to Grade 10 (N=1470).

As can be seen from the mean scale scores, there does not seem to be much change in responses from Grade 7 to Grade 10. However, when looking at correlations (Table 6), though significant, these correlations are not as high as the qualitative literacy scores. Students responses over time to these good at math item is not very related. For example, the Grade 7 and Grade 10 responses have the lowest correlation (0.324) and the highest correlation is only 0.490 (Grade 9 and 10). These are Likert items so the change over the years is expected to be choppy and might explain why the correlations are lower. Grade 7 Grade 8 Grade 9 Grade 10 Grade 7

- - -

Grade 8 0.455 (n=2625)

- - -

Grade 9 0.339 (n=2315)

0.391 (n=2164)

- - -

Grade 10 0.324 (n=2222)

0.417 (n=2058)

0.490 (n=2031)

- - -

Table 6. Correlations for “being good at math” responses from Grade 7 to Grade 10. All correlations are significant at the 0.01 level (2-tailed).

Individual growth trajectories for a select group of 10 students do not suggest a positive linear trend (Figure 2). There does not seem to be a clear linear trend as observed with the quantitative literacy variables. But as mentioned earlier, these are Likert items so the change over the years is expected to be choppy.


05.12.04 Page 6 of 11

Figure 3. Observed values of “being good at math” for 10 selected students from Grade 7 to Grade 10.

Thought it appears as though we can not assume linear growth, I still proceeded to first test out a linear growth model. The first model was the basic model in which the slopes were considered to be equidistant and linear and the intercepts of the slopes were fixed to 0. Modification indices for this model suggested that correlating the residuals for good7 and good8 would provide a better fit. In Model 2, these residuals were correlated. In the modification indices for this second model, it was suggested that freeing trend with good10 would provide a better fit. In Model 3, this parameter was freely estimated. The summary model fit indices (Table 7), suggest that a linear growth model with the grade 10 slope free to be estimated (Model 3) seems to fit the best fit. I also tried estimating the slope for good9 (Model 4) but Model 3 seemed to be the best fit.


05.12.04 Page 7 of 11

Process 2,

Model 1 Process 2, Model 2

Process 2, Model 3

Process 2, Model 4

Chi-square value 40.149 23.105 3.348 35.407 df 5 4 3 3 p-value 0.0000 0.0001 0.3398 0.000 CFI 0.981 0.990 1.000 0.982 TLI 0.977 0.984 1.000 0.965 RMSEA 0.048 0.039 0.006 0.059 SRMR 0.027 0.018 0.011 0.036 Akaike’s Information Criterion (AIC) 75637.957 75622.914 75605.156 75637.216 Schwarz’s Bayesian Criterion (BIC) 75692.330 75683.328 75671.612 75703.671 Loglikelihood H0 value -37809.979 -37801.457 -37791.578 75637.216 Table 7. Indices of model fit for Model 1, 2, 3 and 4 for “being good at math” for Grades 7-10. Process 2, Model 1: fixed time scores (0, 1, 2, 3) Process 2, Model 2: fixed time scores (0, 1, 2, 3), good7 and good8 correlated Process 2, Model 3: time score for grade 10 estimated (0, 1, 2, *), good7 and good8 correlated Process 2, Model 4: time score for grade 9 and 10 estimated (0, 1, *, *), good7 and good8 correlated

The chi-square difference test indicates that there isn’t an improvement between Models 3 and 4 but there is an improvement between Models 1 and 2 and Models 2 and 3 (Table 8). The loglikelihood ratio test leads to similar conclusions. Chi-square Difference Model 1 40.149 (5) Model 2 23.105 (4) 17.044 (1)* Model 3 3.348 (3) 19.757 (1)* Table 8. Difference in chi-square values for Model 1, 2, and 3. * indicates as significant improvement. With the exception of good10, the R2 values for the other observed variables are fairly low. This indicates that the latent variables are not measuring the observed variables very well. Good10 seems to be measured fairly well with an R2 of 0.844 but the other variables are not being measured as well. So even though the fit statistics indicate that Model 3 is the best choice thus far, there are probably better models that explain the observed variables a little better. Similar to the quantitative literacy variables, below is a table that compares the observed means with the estimated means for each grade level (Table 9). Grade Time Score Estimated Means = intercept + slope(time score) Observed Means

7 0.000 22.415 + 0.234 (0.000) = 22.415 22.45 8 1.000 22.415 + 0.234 (1.000) = 22.649 22.52 9 2.000 22.415 + 0.234 (2.000) = 22.883 22.69 10 7.936 22.415 + 0.234 (7.936) = 24.272 23.97

Table 9. Estimated and observed means for Process 2.


05.12.04 Page 8 of 11

There seems to be a large change in perceptions from Grade 9 to Grade 10. The difference between the estimated means and the observed means are roughly similar which is a good sign but this huge jump from Grade 9 to Grade 10 doesn’t seem exactly right. Reasons why students might respond with a huge positive change in their perception of their math skills from Grades 9 to 10 needs to have more of a substantive reasoning but for the purposes of this assignment, perhaps this has something to do with a change in school size. High schools are much larger than middle schools, so students move from Grade 8 to Grade 9 into a much different type of school. After a year of this change, they might start to compare their knowledge, skills and performance with the other students and start to feel more comfortable and better about themselves which results in a huge change in responses to this item from Grade 9 to 10. Obviously, there is no research grounding for this hypothesis and I don’t didn’t test take into consideration the nested nature of the data. The relationship between where students start off and how fast they grow is small but positive (0.110). This would indicate that students who have a higher perception of their math skills in the 7th grade tend to continue to have this positive attitude and even have a slightly higher increase in this positive attitude as time goes on –compared to 7th graders who have lower initial positive attitudes. Figure 4 provides a schematic for modeling perceptions of being good at math from Grade 7 to Grade 10.

Figure 4. Model 3 includes correlated residuals for Grade 7 and 8 and slopes for good10 freely estimated. In Model 3, the residuals for good7 and good 8 are correlated. Again, to offer a substantive reason for this correlation, I hypothesize that perhaps there is something in common about the being in middle school that is somewhat related over and above what is explained by the latent variables. One’s attitude in middle school remains pretty stable or at least somewhat related, which accounts for the correlated residuals for these grades. In general, this process of modeling change of attitudes or perceived efficacy in this area of math is hypothesized to be related to actual performance on mathematics items. More specifically,


05.12.04 Page 9 of 11

mathematics is defined as skills in quantitative literacy. To determine what the relationship between these different variables over time will explored next. Joint Analysis of Process 1 and Process 2 After running separate growth models for both processes, a joint analysis of both processes was conducted. Four students were excluded because they were missing information on all 8 variables (N=3112). The first model assumed linear growth for both processes (Joint, Model 1). The second model (Joint, Model 2) used the information from Processes 1, Model 2 and Process 2, Model 3. In Process 1, Model 2 (quantitative literacy), residuals for Grade 8 and Grade 9 are correlated. In Process 2, Model 3 (good at math), the Grade 10 time score is freely estimated and Grade 7 and Grade 8 residuals are correlated. I regressed the slopes of Process 1 on the intercept of Process 2 and the slope of Process 2 on the intercept of Process 1 to look for a predictive relationship in Joint, Model 3. The modification indices for Model 3 suggested that I also correlate the residuals for the cross-processes time specific correlations between Grade 7 and Grade 8 variables (Joint, Model 4). Model 4 seemed to fit the data the best and will be interpreted below. Fit indices for these four different models are presented in Table 10. The AIC and BIC are the lowest in Model 3 compared to Model 1, 2 and 4. In addition, the loglikelihood difference tests comparing the different models suggests that Model 3 fits the data the best. Joint,

Model 1 Joint,

Model 2 Joint,

Model 3 Joint,

Model 4 Chi-square value 171.718 28.568 32.178 20.779 df 22 19 21 19 p-value 0.0000 0.0731 0.0562 0.3491 CFI 0.983 0.999 0.999 1.000 TLI 0.979 0.998 0.998 1.000 RMSEA 0.047 0.013 0.013 0.005 SRMR 0.048 0.013 0.016 0.015 Akaike’s Information Criterion (AIC) 145937.565 145800.415 145800.025 145792.626Schwarz’s Bayesian Criterion (BIC) 146070.511 145951.490 145939.014 145943.702Loglikelihood H0 value -72946.782 -72875.207 -72877.012 -72871.313 Table 10. Indices of model fit for Model 1, 2, 3 and 4 for joint analysis of Process 1 (quantitative literacy for Grade 7-10) and Process 2 (good at math for Grade 7-10). Joint, Model 1: parallel processes, fixed time scores (0, 1, 2, 3) Joint, Model 2: parallel processes, qlt8 and qlt9 correlated, time score for grade 10 estimated (0, 1, 2, *), good7 and good8 correlated Joint, Model 3: parallel process, qlt8 and qlt9 correlated, time score for grade 10 estimated (0, 1, 2, *), good7 and good8 correlated, slopes of Process 1 regressed on intercept of Process 2, slope of Process 2 regressed on intercept of Process 1 Joint, Model 4: parallel process, qlt8 and qlt9 correlated, time score for grade 10 estimated (0, 1, 2, *), good7 and good8 correlated, slopes of Process 1 regressed on intercept of Process 2, slope of Process 2 regressed on intercept of Process 1, correlated residuals between Grade 7 and Grade 8 variables across processes


05.12.04 Page 10 of 11

To improve the fit even more and/or to test a particular research question, time variant or invariant covariates should be added to the model. The addition of a time variant covariate such as the rigor of the courses taken at each grade level would help to answer questions of whether the types of courses taken at each grade level impact ones attitude toward math and quantitative literacy skills. Adding a time invariant covariate such as gender would help determine whether gender plays in a role in perceptions of math skills and quantitative literacy skills. In this fourth model without any covariates, regressing the slope for Process 1 onto the intercept for Process 2 is significant. Quantitative literacy is predicted by the intercept of the being good at math variable. But the intercept of being good at math is not predicted by quantitative literacy. The estimate is actually small but negative which is counterintuitive. If students reported lower self efficacy in math, they tended to score higher on the quantitative literacy items. One explanation for this is that students who know more about math will tend to know the breadth and depth of mathematics and thus will probably feel less confident about their skills in the field. Where students who don’t know or understand how complex mathematics is might over estimate their skills in math. The intercepts are highly related to one another which suggest that being good at math and having high quantitative skills are related to one another. The residual errors that were correlated for this fourth model are all significant which suggest that there is some relationship to these errors over and above what the latent variables are explaining. In this model, Grade 7 and Grade 8 variables were significant which suggests that there is something about these cross-time processes that are related. The following figure (Figure 5) describes the relationship between Process 1 and Process 2. I purposefully excluded the arrows going from the intercept and slope latent variables to the observed variables for graphical purposes.

Figure 5. Model 4 for parallel processes excluding arrows from intercepts (level1 and level2) and slopes (trend1 and trend2) to the observed variables.


05.12.04 Page 11 of 11

Adjacent errors were correlated for Grade 7 and Grade 8 good at math variables and Grade 8 and 9 quantitative literacy variables. I have some weak hypotheses for why this might be the case, but my hunch is that if the nested nature of the data is taken into consideration, these hypotheses don’t apply. The cross-process correlations for Grade 7 and 8 variables suggest that there is some sort of shared experience in these grade levels that are significant over and above the latent variables. One reason why we don’t see the similar sort of shared relationship in Grade 9 and 10 might have something to do with the students being in high school and taking a wider range of classes. Maybe in middle school, the classes are fairly homogeneous and there isn’t much of a choice of courses for students to explore. This might change in high school and so there is less of a relationship between these variables. There was also a big change in attitudes from Grade 9 and 10 in terms of being good at math which suggests that there is something going on between Grade 9 and 10 that changes the attitudes or perceptions of students. The estimated means for the intercepts are close to the observed means which is a good sign. There is significant variation for both processes in terms of these initial means. The purpose of parallel processes was to examine a hunch that these two processes are correlated. Based on the models, I would conclude that there is some relation over time between performance on quantitative literacy items and perceptions about math skills.

Documents

Latent Variable Modeling ED231E, Spring 2004 …Latent Variable Modeling ED231E, Spring 2004 05.12.04 Page 1 of 11 Assignment 4 For this assignment, I have selected a growth model