Southern Illinois University Edwardsville Logo
Apply to SIUE
Academic Innovation & Effectiveness
Academic Innovation & Effectiveness
Innovation Header

Valuing People Through An Improved Process For Student Evaluation of Teaching (SET)

Action Project


Institution: Southern Illinois University at Edwardsville
Submitted:
Contact:

Denise Cobb

Email:

pcobb@siue.edu

Phone:

618-650-5609


Timeline:
Planned project kickoff date: 09/07/2010
Target completion date: 09/04/2012
Actual completion date:
A. Give this Action Project a short title in 10 words or fewer:
Valuing People Through An Improved Process For Student Evaluation of Teaching (SET)
B. Describe this Action Project's goal in 100 words or fewer:
Reporting to the Provost and the Faculty Senate, this action will establish a Student Evaluation of Teaching Committee to carry out the following activities: 1) develop a set of standard SET procedures, 2) create and validate a SET instrument, 3) create documents detailing the administration of the SET instrument, the acceptable uses of resulting data, and develop a framework for continued review of the instrument, 4) pilot the instrument, 5) openly document the progress and goals of the committee on-line for all faculty to observe and 6) establish procedures for the continued review of the SET instrument.
C. Identify the single AQIP Category which the Action Project will most affect or impact:
Primary Category:  Valuing People
D. Describe briefly your institution's reasons for taking on this Action Project now -- why the project and its goals are high among your current priorities:
Student Evaluation of Teaching (SET) first emerged as a university concern in 2004 with the formation of a Faculty Senate Committee. This effort was followed by a School of Education committee in 2004-05, a School of Nursing committee in 2008, and a Curriculum Council of Faculty Senate SET Committee in 2009. Although all of these committees have worked hard to strengthen student evaluation of teaching processes, there is still considerable variation in these the form, content, administration, and use of SETs at SIUE. Faculty members across the various schools and colleges have expressed concerns about the ways SETs are carried out within their departments. Some faculty members are genuinely concerned about the disparate standards for evaluating effective teaching between individuals and/or departments. Likewise, faculty members have expressed concerns about using instruments that have not been validated. As an AQIP institution, SIUE seeks to identify opportunities for improvement and act upon them. The Higher Learning Commission has asked SIUE to take a more "scholarly approach" to assessment measures while demonstrating our commitment to "valuing people" by addressing concerns expressed by faculty. These factors make this project a priority for SIUE's continuous improvement efforts.
E. List the organizational areas - -institutional departments, programs, divisions, or units -- most affected by or involved in this Action Project:
This project will affect all tenure-track faculty within Academic Affairs who are teaching courses.
F. Name and describe briefly the key organizational process(es) that you expect this Action Project to change or improve:
A strengthened SET process will allow faculty to make better use of student data as they work to improve the quality of their teaching and to promote student learning outcomes. It will also improve the personnel evaluation system used by faculty to attain tenure and promotion, as SETs are commonly used as one measure of teaching effectiveness.
G. Explain the rationale for the length of time planned for this Action Project (from kickoff to target completion):
Based on the agreement with Faculty Senate, much of the work required to accomplish the six goals listed previously will occur in the '10-'11 academic year. Because the work needs approval from both the Provost and Faculty Senate, we envision a 1 1/2 to 2 year process to develop a form, validate it, pilot test, review and revise policies, and to attain appropriate approval to make recommendations to university policy.
H. Describe how you plan to monitor how successfully your efforts on this Action Project are progressing:
We will monitor success by providing regular communication of our activities, accomplishments, and progress to the faculty. We will do so through a variety of mechanisms including regular updates via email, brown bag sessions, and periodic meetings with Faculty Senate. Regardless of outcome, the key to success for this project is to encourage faculty involvement and engagement in this discourse. Continuous improvement efforts will be supported by sustained faculty engagement with these issues. It is the charge of the committee to create structured opportunities for information sharing and to facilitate faculty input. Such opportunities include a website with discussion forum, regular email updates, brown bag discussions, liaison meetings, surveys of departments and faculty, etc.

I. Describe the overall "outcome" measures or indicators that will tell you whether this Action Project has been a success or failure in achieving its goals:

The outcome associated with the project relates to the accomplishment of the six goals of the project listed above. The results of this process will be shared with the Provost and Faculty Senate, at which time a decision can be made as to the full-scale implementation of a campus-wide SET program and the adoption of revised policies regarding use and administration. Because the use of a standard validated end of the semester course evaluation form is controversial, success will not be measured by the adoption of a university-wide standard SET instrument. Project success may be determined by the broader awareness and consistent application of university policy regarding SET administration and use. As this project is designed to value people, SIUE is committed to listening to faculty and being responsive to their needs. Policy recommendations and decisions that emerge from this project should integrate faculty perspectives and attend to the concerns that arise during the outreach efforts.

For more information about the SET project, please visit the website: www.siue.edu/innovation/assessment/set/


Last Action Project Update: 09/28/2012

A. Describe the past year's accomplishments and the current status of this Action Project.

The charge for this AQIP SET committee originally included the follow ing:
1) develop a set of standard Student Evaluation of Teaching (SET) procedures, 2) create and validate a SET instrument, 3) create documents detailing the administration of the SET instrument, the acceptable uses of resulting data, and develop a framework for continued review of the instrument, 4) pilot the instrument, 5) openly document the progress and goals of the committee on-line for all faculty to observe and 6) establish procedures for the continued review of the SET instrument.

The project accomplished all of the goals above in the 2010-2011 academic year. The 2011-12 year was spent implementing the new SET core instrument and procedures. In the fall of 2011, the AQIP committee and Faculty Senate worked and approved an implementation plan. The implementation plan included: 1) a plan to transition Assistant and  Associate professors to using the new instrument without interrupting their SET data used for promotion and tenure, 2) model scripts for faculty to use when administering the SET instrument and, 3) a plan for continuous review of the SET instrument and policies. All of those were accomplished and the new SET and policies instrument was implemented throughout the university in spring of 2012.  Additionally, the SET implementation committee completed a small study on on-line courses. A validation study was done to determine if the new SET core instrument was appropriate for on-line courses delivered predominately on-line. The results of this pilot study will be shared with the SIUE Faculty Senate in Fall 2012. The policy approved by the Faculty Senate did not apply to on-line courses because of concerns about whether the items were appropriate. As such, the validation sub-committee developed and conducted a pilot study to determine how well the items work in online courses.

Review (10/02/2012):

The College is commended for undertaking this project in a logical and rigorous manner. Given that the issue of student evaluation of teaching is of interest in many institutions, the College may w ish to consider presentation or publication of its efforts and the results of the pilot validation study it conducted in order that others may benefit from w hat it has learned.

B. Describe how the institution involved people in work on this Action Project.

The committee’s membership included faculty members from the College of Arts and Sciences, School of Education, School of Engineering, School of Nursing, School of Pharmacy, Library Information Services, and the Office of the Provost. The committee also included representation from the student-body and Information Technology Services (ITS). The main SET committee met several times during the fall semester to develop the implementation plan. A representative of the SET committee met monthly with Faculty Senate’s Curriculum Council to gain input when needed, to insure on-going collaborative communication, and to work toward approval of the implementation plan. In addition, e-mails were sent to the university community to ensure transparent communication. In the spring, during the implementation stage, the AQIP committee chair and/or Assistant Provost met with department chairs to review their current SET policies and practices. This occurred through chairs’ councils of the various colleges/schools and in private one-on-one  meetings. This outreach was a critical support for departments during the implementation stage.

Review (10/02/2012):

The College's involvement of multiple constituencies is notable. SET is of value to both students and faculty and the College involved both of these communities in the function of this project.

C. Describe your planned next steps for this Action Project.

This project will be retired in fall of 2012. Although continued review of the SET core instrument and SET policies will remain in place as prescribed by the plan approved by the SIUE Faculty Senate and confirmed by the Provost and the Chancellor. Continuous review of the ways in which the items work is critical as we work to better understand how faculty members can best use these data to support student learning and the ways in which the data may be used for evaluation purposes.

The SET Continuous Review committee, a subcommittee of the Committee on Assessment, meets every three years and functions to oversee continuous review and validation of the SIUE Student Evaluation of Teaching Core Instrument (SETCI) . The Committee shall be constituted of a minimum of four faculty members, including the Director of Assessment (as a voting member) and an additional liaison from the Committee on Assessment. Faculty members w ill be chosen based on their expertise in psychometric measurement, survey design, and statistics. Appointments are made jointly by the Director of Assessment and the Committee on Assessment and approved through Faculty Senate. Appointments to the Committee shall normally be for a one year term; reappointment is permitted. All members of the Committee will be voting members. The Committee will be responsible for the continuous review and validation of the SIUE Student Evaluation of Teaching Core Instrument (SETCI), and making recommendations to the Committee on Assessment and Faculty Senate on the basis of the data collected.

This group must continue to find ways to collaborate with faculty, administrators, and students in the continuous review process. The committee must share results and work with constituencies to improve processes.

Review (10/02/2012):

Although the College notes that the Continuous Review Committee should collaborate with multiple constituencies, it does not appear from the information provided that students are represented on this Committee. The College may wish to consider inclusion of student representation on the Committee to ensure all relevant interests are included and considered. The College also may wish to consider whether inclusion of faculty whose expertise lies in areas other than "psychometric measurement, survey design, and statistic" may lead to a broader ongoing assessment of the usefulness of the SET instrument.

D. Describe any "effective practice(s)" that resulted from your work on this Action Project.

This project is an exemplary example of shared governance at Southern Illinois University Edwardsville w hereby an Academic Quality Improvement Program (AQIP) committee comprised of faculty, staff, students, and administrators worked together to improve student evaluation of teaching (SET) processes on our campus. The leadership of the committee, Faculty Senate and the Provost’s Office recognized that developing a thoughtful approach to student evaluations of teaching “involved integrating the technical requirements of good measurement with the political process of building consensus around shared values (Arreola, 2004: 1).”

Review (10/02/2012):

The broad involvement of multiple interested and affected constituencies in this project is notable. However, it is not evident from the material provided that new effective practices resulted from this project, other than the SET process itself. The institution may find it beneficial to consider further what lessons and effective practices emerged during the conduct of this project.

E. What challenges, if any, are you still facing in regards to this Action Project?

Some challenges emerged during the implementation of the project. Certainly some issues were easily resolved with concerted attention and collaboration. Other challenges remain, but we are confident that they do not threaten the overall success of the initiative.
1) We must develop a policy regarding records retention for the raw data and summary reports.
2) Some courses, for example those organized in blocks taught by multiple instructors, are not easily serviced by the university’s CourseEval software system.
3) The new policy requires that all open ended comments are in typed form to ensure students’ anonymity. This change created some challenges for programs since some had not been doing this work in the past. The optional qualitative feedback is critical to programs, and we do not want to discourage use of those data. We recognize that there is a significant work load required for some departments to transcribe the comments. We continue to w ork w ith departments to explore on-line options for the qualitative portion.
4) Some programs have moved to on-line SET delivery and management. There is some understandable w orry about low response rates. The current policy prohibits incentives or punishments for participation or lack thereof. As such, programs have limited avenues for increasing response rates. This creates an opportunity to work with programs to identify ways to approach the problem institutionally or at school/college levels to increase response rates through other means recommended in the literature.
5) We must continue to find ways to collaborate with faculty, administrators, and students in the continuous review process. The committee must share results and work with constituencies to improve processes.

Review (10/02/2012):

The College has identified further actions that may enhance the effectiveness of this project. From what is presented here, it appears that SIU-E is making progress on this project and is encouraged to continue to learn from it and make even greater advancement to harvest the opportunities identified.

facebookoff twitteroff vineoff linkedinoff flickeroff instagramoff googleplusoff tumblroff foursquareoff socialoff