Southern Illinois University Edwardsville Logo
Apply to SIUE
Academic Innovation & Effectiveness
Academic Innovation & Effectiveness
Innovation Header

Meta-Assessment Responding to the Systems Appraisal and NSSE Results

Action Project


Institution: Southern Illinois University at Edwardsville
Submitted: 2005-05-23

Timeline:
Planned project kickoff date:
Target completion date:
Actual completion date:
A. Give this Action Project a short title in 10 words or fewer:

Meta-Assessment Responding to the Systems Appraisal and NSSE Results

B. Describe this Action Project's goal in 100 words or fewer:

This year-long Action Project will examine the effectiveness of the Senior Assignment as both a culminating senior experience and as an assessment device. A team of faculty researchers with expertise in curriculum and quantitative and qualitative methods will design and conduct a research study, the results of which will inform the General Education Reform and Program Review processes.

C. Identify the single AQIP Category which the Action Project will most affect or impact:

Primary Category: Helping Students Learn

D. Describe briefly your institution's reasons for taking on this Action Project now -- why the project and its goals are high among your current priorities:

In preparing for the 2004 Strategy Forum, SIUE examined the recently completed Action Projects, considered feedback from the annual updates on those projects and on the Systems Portfolio, and considered campus discussions in a variety of arenas (faculty listserv, Academic Affairs Conference, Faculty Senate, and AQIP Task Forces). One concern that repeatedly surfaced was that NSSE data has not been as useful as we had hoped. Over the last two years, SIUE has explored different ways to use NSSE for quality improvement. The measures for SIUE's strategic plan, for example, rely on a number of questions from NSSE along with other measures. To this point, though, the NSSE data provide a general measure of the health of the strategic plan, but not in actionable ways.

SIUE used NSSE data in a focused study to examine the effectiveness of pilot freshman seminar courses to meet the defined course objectives. The focused study compared responses from freshmen who enrolled in one of the pilot freshman seminar classes with those freshmen who did not. Results of the data analysis provided supporting evidence for a recommendation from a Curriculum Council review committee that the Faculty Senate accept and approve a freshman seminar requirement. Based in part on NSSE data analysis, the review committee also recommended that a freshman seminar requirement be included in upcoming general education reform.

The experience with using NSSE for decision making and action encouraged us to look for ways in which we could use NSSE more actively than we had in the past. A common question that emerged throughout the discussions was why only 70% of SIUE seniors respond that they have a culminating senior experience, even though the Senior Assignment (a culminating senior experience) is a graduation requirement. The response to that question should be 100%. The System Appraisal Feedback Report adds to the discussion about the Senior Assignment. One of the few areas cited for pressing or outstanding opportunity for improvement was the Senior Assignment. As the reviewers noted, "While all programs have implemented Senior Assignment as a baccalaureate requirement, the reported data are process measures…rather than outcome measures to determine the levels of learning achieved by the students within the departments." The Senior Assignment has been a marquee program that SIUE has presented at AAHE assessment conferences, AAC&U Assessment and General Education conferences, the Higher Learning Commission annual meetings, the Illinois Board of Higher Education assessment workshops, and other regional, national and international venues. SIUE understands that corrective feedback regarding the Senior Assignment can provide an opportunity to take an area of strength and look for ways to strengthen it.

This year-long Action Project will examine the Senior Assignment from both a culminating senior experience and an assessment perspective. These two perspectives will help to determine if student learning is taking place and if evidence of the learning is being captured and utilized effectively. To most fully understand the Senior Assignment and the impact that it has, both qualitative and quantitative measures will be employed. Once the study has been completed, the results will be forwarded to the General Education Reform Committee to guide its deliberations. The results will also be forwarded to the Curriculum Council and the Committee on Assessment to guide their discussions about Undergraduate Program Reviews.

E. List the organizational areas - -institutional departments, programs, divisions, or units -- most affected by or involved in this Action Project:

Academic Affairs; Committee on Assessment; Curriculum Council

F. Name and describe briefly the key organizational process(es) that you expect this Action Project to change or improve:

Senior Assignment; General Education Reform; Undergraduate Program Review

G. Explain the rationale for the length of time planned for this Action Project (from kickoff to target completion):

Existing processes are in place and well developed to support this Action Project. All undergraduate programs have existing Senior Assignments, and all baccalaureate graduates complete Senior Assignments. A research group, chaired by the Assistant Provost for Planning and including the Director of Undergraduate Assessment, a representative from the Committee on Assessment, a representative from the Curriculum Council, and faculty with expertise in curriculum and qualitative and quantitative research methods, will be formed in Spring 2005 to begin coordinating work on this Action Project.

H. Describe how you plan to monitor how successfully your efforts on this Action Project are progressing:

Establishment of a research project team, design of study, collection and analysis of data

I. Describe the overall "outcome" measures or indicators that will tell you whether this Action Project has been a success or failure in achieving its goals:

Research report to be used in the redesign of the General Education program and the Undergraduate Program Review process

J. Other information (e.g., publicity, sponsor or champion, etc.):

A paper detailing the results of the study will be completed.N/AN/A

K. Project Leader and contact person:
Contact Name: Susan L. Thomas, Associate Provost for Academic Planning

Email: suthoma@siue.edu

Telephone: (618) 650-3772


Last Action Project Update: 2007-08-20
A. Describe the past year's accomplishments and the current status of this Action Project.

The personnel changes that had delayed progression on this project were resolved and the project was completed. During the year, a research team examined the Senior Assignment (SRA) and produced a report that is guiding changes to both the Senior Assignment and the assessment program as a whole. The report included eight recommendations, ranging from clarification of the purpose of the SRA to identifying programs that need additional support for implementing the SRA. The report ended with the following conclusion: "This research team found that there is engagement with the SRA at SIUE. The program would benefit from improvements, as outlined above (eight recommendations). However, we also found that programs generally do implement an SRA assessing students' majors along with writing and oral communication, and that students are generally aware of this. Further, we found that the willingness of students, faculty, and administrators to engage in a research project such as ours, and to allow us to make public the results, shows clearly that SIUE embodies both the letter and the spirit of ongoing program assessment."

Review (09-05-07):

Southern Illinois University at Edwardsville (SIUE) has made reasonable progress given the changes in personnel demonstrating the AQIP principle "Focus." Faculty, staff, and administrators have integrated their shared focus into their individual work goals and decision-making strategies. The institution is to be commended on completing a research project and formulating eight recommendations. You may want to review these recommendations and determine if they focused on the process of conducting the Senior Assignment (SRA) and student satisfaction with the experience itself or if they included ideas on how well students learned or how to improve student outcomes.
B. Describe how the institution involved people in work on this Action Project.

The research team included faculty from the College of Arts and Sciences and the Schools of Engineering, Education, and Business. After reviewing the characteristics of Senior Assignments, the team identified a number of different implementation methods across departments and selected at least two examples of each method for faculty interviews. Team members interviewed 16 faculty. In addition, the team surveyed graduating seniors at commencement. The Director of Assessment interviewed all department chairs regarding their successes and challenges in managing the assessment program at the department level. The Director of Assessment has begun meeting with departments in the summer to look at improvements and changes based on the work completed last year. The report has been distributed to the Provost Senior Staff, is being shared with department chairs and program directors in meetings with the Director of Assessment, and will provide foundation for the work of the Committee on Assessment in the upcoming year. It has also been posted on the SIUE AQIP website.

Review (09-05-07):

Sound research methods including interviews with faculty and students in addition to survey data were used. Broad cross samplings of SRAs from Colleges and Schools throughout the institution were used in the research study. The final report was distributed to all constituents of the university via use of the SIUE AQIP website. This demonstrates use of the AQIP principle concerning "Involvement." Broad-based faculty, staff, and administrative involvement encourage better decisions and strengthen individual and group ownership of systems, activities, and initiatives. Individuals understand how what they do affects others within and outside the organization, and appreciate how their work helps further the Institution's mission.
C. Describe your planned next steps for this Action Project.

While some of the work on this project had been delayed because of personnel changes, the work defined by the Action Project has been completed, and the project is ready to be retired. Follow-up work has continued and resulted in a reworking of the department reporting mechanisms on assessment results. One of the findings that the research team reported at the 2007 HLC Annual Meeting was that the Senior Assignment has matured as an assessment device, and while in the early years, the SRA results led to program improvements, some of which were significant, the current SRA results tend to be routine. The major programmatic changes have been made. Now the value of the SRA has shifted from being a program change driver as in the early years to being a monitoring device. It remains an enriching and integrative student learning experience. The challenge has shifted to considering how SIUE will change its assessment program now that key elements of the program, such as the SRA, have matured. That work has been begun by the new Director of Assessment.

Review (09-05-07):

The project resulted in reworking the department reporting mechanisms on assessment results indicating a refinement of the process of using information. The AQIP principle of "Leadership" is evident. Leaders have demonstrated a responsibility to make sure that everyone understands and values the institution's mission, goals, and directions and have used the information gathered from this project to understand and inform work goals and decision-making strategies. However, authors mentioned that the Senior Assignment has matured into a ""monitor" versus a "program change driver". The institution may want to revisit how the senior assignment and data collected can be used to determine if students have really learned. Perhaps tests scores, assignment grades from the experience or other learning objects with grading rubrics might provide more definitive information for making key programmatic changes. Visit the following link for a reference on learning objects http://www.reusability.org/read/. The "Instructional Use of Learning Objects" is a new book that connects learning objects to instruction and learning. SIUE has promoted change in a part of the assessment program as a result of this action project indicating evolution in the use of the continuous quality improvement process.
D. Describe any "effective practice(s)" that resulted from your work on this Action Project.

The work of the research team was reported at the HLC 2007 Annual Meeting and appeared in Volume 3 of the Collection of Papers on Self-Study and Institutional Improvement, 2007. Follow-up work may lead to best practices, but those are not yet identified.

Review (09-05-07):

Although SIUE has not identified any effective practices per se, they have demonstrated that the organization has learned how to use the quality improvement process to change assessment techniques to promote improvement. This demonstrates another AQIP principle "Learning." By always seeking more effective ways to enhance student achievement through careful design and evaluation of programs, courses, and learning environments, both the institution and its employees demonstrate an enthusiastic commitment to organizational and personal learning as the route to continuous improvement.
E. What challenges, if any, are you still facing in regards to this Action Project?

The primary challenges connected with this Action Project are related to the changes that SIUE is facing in its general education program reform project. The results of this Action Project, and particularly the findings and recommendations from the research team report, will provide guidance as the university necessarily modifies its assessment program to align with a new general education program, which is expected to begin implementation in 2008. The review to the 2006 Annual Update pointed out that "to examine an important institutional program that has history, tradition and academic rationale behind it is bund (sic) to face some inherent problems." The difficulties in reconsidering an established assessment program are in some ways moderated because of changes in general education, to which that assessment program is attached. The risk associated with general education reform provides an opportunity to rework the assessment plan in ways that might be difficult without general education reform.

Review (09-05-07):

SIUE is on the right track to align the assessment program with the general education reform. Obtaining timely, meaningful data will help to shape future changes.
F. If you would like to discuss the possibility of AQIP providing you help to stimulate progress on this action project, explain your need(s) here and tell us who to contact and when?

None

Review (09-05-07):

No comment.

facebookoff twitteroff vineoff linkedinoff flickeroff instagramoff googleplusoff tumblroff foursquareoff socialoff