Challenges and Opportunities in Evaluating Two MSP Projects

Authors: Jane Butler Kahle, Yue Li, Sarah Woodruff

PennSTI
Comparisons Before and Three Years After Teacher Initial Participation in PennSTI
Differences were found between pre and final subscale means on the teaching ("What I do in class") and learning ("What my students do") subscales for teachers participating in the PennSTI (Table 3). These analyses used raw mean scores. Science teachers reported slightly more use of inquiry by their students, while chemistry teachers significantly increased their own ("What I do in class") use of inquiry.

 

Table 3. Comparison of Subscale Mean Scores for Penn Science and Chemistry Teachers: Pre (Spring 2005) and Final (Fall 2007).

Course	Subscale	Time	n	Mean	SD	t-value	df	p-value	n2
Science	What I do in class	Pre	16	3.51	0.39	-1.20	15	0.250	0.09
		Final	16	3.65	0.38
	What my students do	Pre	15	3.38	0.40	-1.86	14	0.084	0.20
		Final	15	3.65	0.32
Chemistry	What I do in class	Pre	15	3.07	0.28	-3.55	14	0.003	0.47
		Final	15	3.43	0.34
	What my students do	Pre	15	3.10	0.46	-2.04	14	0.060	0.23
		Final	15	3.40	0.38

1 = "Almost Never" and 5 = "Very Often"

During the spring of 2005, 2006, and 2007, the student questionnaire was administered to science or chemistry students in one typical class of each participating teacher. Although the student groups changed, they were taught by the same teacher. Item Response Theory (IRT)-Rasch Model-was used to analyze the student data. For the teaching and learning subscales that request frequency ratings, a Rasch measure of 0 indicates that the respondent selected "Almost Never" for all subscale items. A Rasch measure of 100 indicates that the respondent choose "Very Often" for all subscale items.

Science students reported more use of standards-based instruction on both the teaching ("What my teacher does") and learning ("What I do") subscales after teacher participation (Table 4). Likewise, chemistry students of PennSTI teachers reported increased frequency of inquiry learning after teacher participation (Table 5).

Table 4. Comparison of Rasch Mean Subscale Scores of Penn Science Students: Pre, Post Year 1, and Post Year 2.

Subscale	Time	n	Rasch Mean	SD	F-value	df	p-value	n2
What I do in class	Pre	555	46.68 *	4.83	18.30	2, 1323	< 0.001	0.03
	Post Yr 1	413	48.41	4.93
	Post Yr 2	358	48.10	4.36
What my teacher does in class	Pre	552	60.50 *	12.31	7.20	2, 1316	0.001	0.01
	Post Yr 1	410	62.89	13.10
	Post Yr 2	357	63.29	10.98

* Mean scores for Post Year 1 and Post Year 2 are significantly higher than Pre mean scores.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

Table 5. Comparison of Rasch Mean Subscale Scores of Penn Chemistry Students: Pre and Post Year 2.*

Subscale	Time	n	Rasch Mean	SD	t-value	df	p-value	n2
What I do in class	Pre	362	46.70	5.87	-2.89	610	0.004	0.01
	Post Yr 2	250	48.08	5.80
What my teacher does in class	Pre	362	57.16	12.19	-1.56	576	0.120	0.004
	Post Yr 2	249	58.60	10.62

* The questionnaires were not administered Post Year 1.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

 

MSPinNYC
Comparisons Before and One Year After Teacher Participation in the MSPinNYC Summer 2006 Institutes
Item Response Theory (IRT)-Rasch Model-was used in order to provide the project with rigorous and appropriate analyses. Comparisons were made between Rasch mean scores before the institute and one year after participating as a TRT teacher in an institute (Table 6). Significant differences were found (in favor of more frequent use of inquiry and other standards-based instruction) on both the teaching and learning subscales one year after teacher participation.

Table 6. Comparison of Rasch Mean Scores for Teaching and Learning Subscales for MSPinNYC TRT Teachers: Spring 2006 and Spring 2007.

Subscale	Time	n	Rasch Mean	SD	t-value	df	p-value	n2
What I do	Spring2006 (Pre)	17	57.77	8.81	-2.04	16	0.058*	0.21
	Spring2007 (Post)	17	61.13	7.95
What my students do	Spring2006 (Pre)	17	49.45	7.21	-2.66	16	0.017	0.31
	Spring2007 (Post)	17	53.31	6.23

* Using alpha = 0.1.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

Students in TRT teachers' classes completed the student questionnairepre and post teacher participation in MSPinNYC.  Four different institutes were offered in 2006 and the findings for the learning subscales are displayed in Table 7 by specific institute/class.

Table 7. Comparison of Rasch Mean Scores for Learning Subscale for Students of MSPinNYC TRT Teachers: Spring 2006 and Spring 2007.

Subscale	Time	n	Rasch Mean	SD	t-value	df	p-value	n2
Mathematics A	Spring2006 (Pre)	59	46.38	2.27	-2.36	130	0.020	0.04
	Spring2007 (Post)	73	47.50	3.01
Mathematics B	Spring2006 (Pre)	37	46.32	2.39	-3.13	91	0.002	0.10
	Spring2007 (Post)	56	47.99	2.59
Living Environment	Spring2006 (Pre)	99	45.77	2.64	1.72	182	0.088*	0.02
	Spring2007 (Post)	85	45.01	3.35
Chemistry	Spring2006 (Pre)	68	44.97	3.17	-2.19	138	0.030	0.03
	Spring2007 (Post)	73	46.12	3.00

* Using alpha = 0.1.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

Mathematics A and B and chemistry students indicated significantly more use of problem solving and other active learning strategies on the learning subscale after their teacher participated in an MSPinNYC institute. However, for students in Living Environment classes the reverse was true.

Comparison between Participant and Non-Participant Teachers in MSPinNYC Spring 2007

Comparisons also were made between the responses of the two groups of teachers and those of their students. No statistically significant differences were found for any of the subscale comparisons on the responses of teachers who participated in the Summer 2006 institutes (TRT team members) and non-participant Hub school colleagues. However, significant differences were found in comparing responses of the two groups of students (Table 8).

Table 8. Comparison of Rasch Mean Scores for the Learning Subscale for Students of MSPinNYC Participant and Non-Participant Teachers: Spring 2007.

Subscale	Time	n	Rasch Mean	SD	t-value	df	p-value	n2
Mathematics A	Participants	73	47.50	3.01	1.82	179	0.071*	0.02
	Non-Participants	110	46.64	3.17
Mathematics B	Participants	56	47.99	2.59	2.94	97	0.004	0.08
	Non-Participants	43	46.34	2.97
Living Environment	Participants	86	45.09	3.40	-1.65	322	0.101	0.01
	Non-Participants	238	45.72	2.96
Chemistry	Participants	98	46.68	3.03	2.31	121	0.023	0.04
	Non-Participants	25	45.05	3.56

* Using alpha = 0.1.
0 = respondent chose "Almost Never" for all subscale items.
100 = respondent chose "Very Often" for all subscale items.

Mathematics A and B students of TRT teachers indicated significantly more frequent use of active learning strategies than did students of non-participating teachers. No statistically significant differences were found between the responses of students in Living Environment classes taught by participating teachers and those taught by non-participating teachers.  Chemistry students of participating teachers, compared to students whose teachers had not participated, indicated significantly more frequent use of standards-based or inquiry activities on the learning subscale.