Frequently Asked Questions About MIT’s Subject Evaluation Pilot
Why are we piloting new subject evaluation questions?
It is essential to the educational mission of MIT to continuously improve the quality of its subject design and delivery over time. This can be accomplished by evaluating our teaching, reflecting on the results of that evaluation, and planning improvements for the next iteration of our teaching (see figure).
Although there are many ways to evaluate one’s own teaching, student evaluations are the most commonly used source of that data. However, student feedback is only as useful as the tools we use to measure it (d’Apllonia & Abrami, 1997; Medina et al., 2019). Student evaluations of a subject can be valid, reliable, and helpful for faculty/instructors “provided the instrument asks the right questions” (Benton & Ryallis, 2016, p. 7). Thus, the first part of the subject evaluation pilot is the trial adoption of a revised set of questions. The second part of the pilot involves providing time for students to complete the evaluations in class. Doing so will help to improve the response rates and welcome more student voices into the feedback process (Benton & Ryallis, 2016; Medina et al., 2019). In the last part of the proposed process, a teaching development specialist (TLL staff) will be available to help participating instructors and departments process and use student feedback (Marsh & Roche, 1997).
What are the limitations of the current subject evaluation questions (SEs)?
In the scholarly literature, a wealth of evidence shows that the use of student ratings of questions like those that currently appear on the standard MIT subject evaluations can be problematic. These issues include:
Students are not well-equipped to answer some of the questions posed, such as whether the instructor has sufficient disciplinary knowledge/expertise.
Student ratings from standard SE questions are not a valid indicator of teaching effectiveness or student learning. Research has shown no correlation between student ratings using traditional questions and student learning outcomes or teaching effectiveness (AAUP, 2016; Braga, et al., 2014; Carrell & West, 2010; DesLauriers, 2019; Hornstein & Law, 2017; Johnson, 2003; Uttl, et al., 2017).
Students’ ratings of a subject (based on standard SE questions) are predicted better by how satisfied they are with their grades than other factors (e.g., instructional quality; Kogan et al., 2022).
Ethnicity, race, physical attractiveness, and instructor’s age influence responses to traditional SE questions (Ambady & Rosenthal, 1993; Anderson & Miller, 1997; Arbuckle & Williams, 2003; Basow, 1995; Cramer & Alexitch, 2000; Linse, 2017; Reid, 2010; Wachtel, 1998; Worthington, 2002).
Overall, women are systematically rated lower than men on traditional SE questions because of their gender (Kreitzer & Sweet-Cushman, 2021).
For an excellent critique of standard, end-of-semester teaching evaluations see: Stark, P.B. & Freishtat, R. (2014) An Evaluation of Course Evaluations, ScienceOpen. Philip Stark is a faculty member in the Department of Statistics at UC Berkeley. See also this summary of Evaluating Student Evaluations of Teaching (Kreitzer & Sweet-Cushman, 2021) and this collection of articles from R.J. Kreitzer, UNC-Chapel Hill.
What kind of information will be collected?
The pilot questions ask students to consider their experiences in the subject, such as
The various aspects of teaching and learning within the learning environment (classroom, lab, design, or production space, etc.)
Their own engagement with the subject within and outside of the designated learning space, and
The impacts of specific instructor actions and approaches on their learning.
What are the questions on the pilot SE?
The questions for the pilot SE form were designed based on research for the design of course evaluations (e.g., Medina et al., 2019) and work from STEM Demonstration Projects sponsored by the Association of American Universities (AAU). The pilot questions are available here.
Can a department still add its own department-specific questions to the pilot set?
Yes, we acknowledge and respect the need to gather department-specific information from subject evaluations. Therefore, the process for adding department-specific questions will remain unchanged. Please keep in mind that the length of the evaluation may impact student response rates.
Can individual instructors have their own subject-specific questions added to the SE?
Yes! The process for adding subject-specific questions will remain unchanged.
What can faculty and instructors learn about their teaching from the pilot evaluations?
There are many potential benefits for instructors participating in the pilot SE process. The pilot SE questions are focused on scholarly teaching criteria and, as such, are hypothesized to provide more detailed and informative student ratings than the traditional SEs (McCarthy, 2012). The pilot SE form also has a more comprehensive set of questions than the traditional SE form:
Some questions focus on course design, which is as important as teacher’s performance for student perceptions of quality and engagement (Levinsson et al., 2024)
The pilot SE questions provide multiple pathways for faculty and instructors to demonstrate effective teaching
The pilot SE questions may provide richer insight into the student experience
The variety of pilot SE questions allows departments, faculty, and instructors to emphasize one or several items that are most important to focus on (McCarthy, 2012)
The pilot SE questions can provide benchmarks for monitoring changes over time (Medina et al., 2019)
Set teaching goals
Reflect on teaching philosophy
Documentation in annual reviews and/or promotion/tenure
Focus on self-improvement rather than comparison to others
Providing time in class for students to complete the SE is essential to the success of the pilot. Setting aside class time increases inclusivity so that feedback reflects a majority of students’ perceptions rather than only a subset. Additionally, it provides instructors with the opportunity to discuss the importance of SEs and how they are used, which is related to more constructive student feedback.
A consultation with teaching development specialists in TLL can help with SE interpretation.
Avoid oversimplification of data about a complex process
Examine distributions of students’ responses, not just averages
Detect important trends within and across courses/semesters
Support teaching improvement plans that interpret the feedback within the course-specific context in collaboration with pedagogy experts
Consider the role of SEs in a more holistic approach to evaluating teaching effectiveness
Is there someone who can help individual instructors and/or departments make sense of the responses?
Yes! Staff from the Teaching + Learning Lab (TLL) will be available to collaborate with individual faculty/instructors and/or department heads to interpret the student feedback and plan actionable changes to instruction based on that feedback.
How might participation in the pilot impact tenure and/or promotion?
We encourage department heads to consider the most appropriate use of student feedback for faculty and instructors in their departments. Because of the emphasis on the student learning experience in the pilot SEs, the results will not be directly comparable to traditional SEs. That being said, faculty and instructors will receive richer, more detailed, and more actionable feedback that will better support them in self-reflection processes involved in the T/P process.
Must every faculty member/instructor in a participating department use the pilot subject evaluations?
Although we encourage widespread faculty/instructor participation, individual faculty and instructors within a participating department can opt out of the pilot. Students in subjects whose instructors have not opted in will receive the standard/traditional version of the SEs. We will work with department administrators to ensure that the relevant SE questions are provided to students.
In subjects with multiple instructors, will all teaching staff need to opt in?
No. All instructors must agree to participate in the pilot for the subject to be included in it.
What should students be told about the pilot?
We encourage instructors and faculty to stress the importance of their participation in the pilot and engagement with the questions. We will provide you with text/language you can use when speaking with your students about the pilot.
Which departments are participating?
We will update this information as departments sign on to participate in the pilot.
What are the next steps for departments that would like to learn more?
Department heads and individual faculty/instructors should reach out to sepilot@mit.edu to learn more.
Ambady, N., & Rosenthal, R. (1993). Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology, 64(3), 431–441. https://doi-org.ezproxy.canberra.edu.au/10.1037/0022-3514.64.3.431
Carrell, S. E., & West, J. E. (2010). Does professor quality matter? Evidence from random assignment of students to professors. Journal of Political Economy, 118(3), 409–432. https://doi-org.ezproxy.canberra.edu.au/10.1086/653808
Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251–19257. https://doi-org.ezproxy.canberra.edu.au/10.1073/pnas.1821936116
Kreitzer, R. J., Sweet-Cushman, J. (2021). Evaluating Student Evaluations of Teaching: a Review of Measurement and Equity Bias in SETs and Recommendations for Ethical Reform. Journal of Academic Ethics, 1-12).
Levinsson, H., Nilsson, A., Mårtensson, K., & Persson, S. D. (2024). Course design as a stronger predictor of student evaluation of quality and student engagement than teacher ratings. Higher Education. https://doi-org.ezproxy.canberra.edu.au/10.1007/s10734-024-01197-y
Marsh, H.W. & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52(11), 1187-1197. https://psycnet.apa.org/doi/10.1037/0003-066X.52.11.1187
McCarthy, M. (2012). Using student feedback as one measure of faculty teaching effectiveness. .In M.E. Kite (Ed.), Effective evaluation of teaching: A guide for faculty and administrators. Retrieved from the Society for the Teaching of Psychology website: http://teachpsych.org/ebooks/evals2012/index.php
Medina, M. S., Smith, W. T., Kolluru, S., Sheaffer, E. A., & DiVall, M. (2019). A review of strategies for designing, administering, and using student ratings of instruction. American Journal of Pharmaceutical Education, 83(5), 7177. https://doi-org.ezproxy.canberra.edu.au/10.5688/ajpe7177
Reid, L. D. (2010). The role of perceived race and gender in the evaluation of college teaching on RateMyProfessors.Com. Journal of Diversity in Higher Education, 3(3), 137–152. https://doi-org.ezproxy.canberra.edu.au/10.1037/a0019865
Rosen, A. S. (2018). Correlations, trends and potential biases among publicly accessible web-based student evaluations of teaching: A large-scale study of RateMyProfessors.com data. Assessment & Evaluation in Higher Education, 43(1), 31–44. https://doi-org.ezproxy.canberra.edu.au/10.1080/02602938.2016.1276155
Stark, P. B & Freishtat, R. (2014). An Evaluation of Course Evaluations, ScienceOpen DOI: 10.14293/S2199-1006.1.SOR-EDU.AOFRQA.v1
Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22–42. https://doi-org.ezproxy.canberra.edu.au/10.1016/j.stueduc.2016.08.007
Worthington, A. C. (2002). The impact of student perceptions and characteristics on teaching evaluations: A case study in finance education. Assessment & Evaluation in Higher Education, 27(1), 49–64. https://doi-org.ezproxy.canberra.edu.au/10.1080/02602930120105054