state unive rs i ty of new yo r k U n i v e r s i t y F a c u l t y S e n a t e Guide for the Evaluation of Undergraduate Academic Programs SENATE U N I V E R S I T Y FAC U LTY The Stat e Un i ve r s i ty o f NewYork 2001 revision Originally issued by the University Faculty Senate of the State University of New York in June, 1983. The Undergraduate Committee of the University Faculty Senate of the State University of New York and the Faculty Council of Community Colleges revised the Guide for the Evaluation of Undergraduate Academic Programs during the years of 1999-2001. The University Faculty Senate adopted this revision on October 27, 2001. Undergraduate Committee Members 1999-2002 Fredrick L. Hildebrand, Co-Chair Marvin LaHood, Co-Chair John DeNisco Anne Donnelly, Past Co-Chair David Elliott Joanne Freeman, Student Assembly Daniel Marrone James McElwaine John Pipkin Rose Rudnitski Malcolm Sherman Augustine Silveira Carole Torok Judith Wishnia Joseph Hildreth, President University Faculty Senate Kathryn E. Van Arnam System Administration Liaison Joseph G. Flynn, Past President University Faculty Senate Representatives of the Faculty Council of Community Colleges Robert Axelrod, President Cathleen McColgin, Liaison Robert Jubenville, Liaison TABLE OF CONTENTS Preface Program Review Curriculum The Faculty The Students Support Services Administrative Support Bibliography 3 4 6 8 10 11 12 13 PREFACE The S worked to impr UNY Faculty Senate has always ove the quality of academic experiences. Realizing that i m p rovement re q u i res current and accura t e knowledge, the SUNY Faculty Senate’s Undergraduate Programs and Policies Committee undertook in 1983 the deve l o p - ment of a Guide for the Evaluation of Undergraduate Academic Programs. In formulating the Guide, that committee attempted to identify the major factors that might be expected to influence the quality and scope of academic programs in order to develop questions that could elicit information important to a meaningful assessment: faculty, students, curriculum, administration, and academic support services. Because of the quality of the original Guide, the SUNY Faculty Senate reissued it in 1990. This 2001 revision of the Guide for the Evaluation of Undergraduate Academic Programs builds on the excellent work of the original committee. This revised Guide continues and increases the centrality of assessment in program review. Consistent with the Characteristics of Excellence adopted by the Commission on Higher Education of the Middle States Association of Colleges and Schools (MSACHE) and the Final Report of the the Provost’s Task Force on the Assessment of Student Learning Outcomes, the Guide fosters a constant awareness of the relationship between any specific programmatic activity and the overall mission, goals, and objectives of the program and the institution. The revised guide also benefits from two decades of research about and practice of assessment and Guide for the Evaluation of Undergraduate Academic Programs program review. Most recently, Departmental Assessment: How Some Campuses Are Effectively Evaluating the Collective Work of Faculty by Jon Wergin and Judi N. Swingen analyzes current policies and practices and offers useful conclusions. Because SUNY academic programs undergo periodic evaluation by SUNY, MSACHE, the New York State Education Depart m e n t , and, sometimes, specialized accrediting associations, the revised Guide for the Evaluation of Undergraduate Academic Programs suggests criteria for a comprehensive evaluation of programs, criteria that are useful and adaptable to any review. Campuses should coordinate program review with anticipated reviews of external evaluators and professional accrediting bodies. The Committee strongly suggests that where there is an external specialized agency, that that program review be accepted as the SUNY program review. Like the original 1983 Guide, this 2001 revision will benefit individual departments making an internal study, those responsible for evaluations that include similar programs at more than one campus, or a campus embarking on an institutional self-study. The results can serve as a baseline for demonstrating improvement and a source of evidence for accountability to various constituencies. Regardless of the motivation for conducting the study, the program review process will contribute important information to the institution’s short- and long-range planning processes and will result in the i m p rovement of the academic experiences of the students. n 3 PROGRAM REVIEW The Undergraduate Committee of the SUNY Faculty Senate in cooperation with representatives of the Faculty Council of Community Colleges developed the Guide for the Evaluation of Underg raduate Academic Pro g rams to encoura g e i m p rovement through a self-study pro c e s s of planning, implementing, and eva l u a t i n g . Although program review is on a five-toseven- year cycle, institutions should collect, analyze, and use data continuously. In an effort to be comprehensive, the Faculty Senate Undergraduate Committee has considered documents from the Middle States Association of the Commission on Higher Education, the New York State Education Department, and specialized accrediting associations. Then, the Committee organized the Guide so that the sections and subsections studied would be easily adaptable to any review. The mission, goals, and objectives of the program and the institution provide the necessary framework for evaluation. All activities derive their relevance from this framework. Consequently, assessment of the program evaluates the collective effectiveness of the activities in achieving the mission, goals and objectives. Similarly, mission, goals, and objectives determine the appropriate criteria, standards, and measures for a program review. Quality is difficult to define; consequently, faculty, students, and administrators should all engage in the dialogue to clarify its meaning in the program being reviewed. Using multiple measures to collect comprehensive sets of data results in more valid assessments of Guide for the Evaluation of Undergraduate Academic Programs 4 quality and success. Open sharing and interpretation of data encourage the use of evidence to support conclusions. Ultimately the effectiveness of a program review depends upon the degree of involvement of the reviewers. As the Report of the Provost’s Advisory Task Force on the Assessment of Student Learning Outcomes states, “each campus is responsible for overseeing the process t h rough which the assessment of academic major programs takes place, following existing curriculum and governance procedures.” Personal identification with the institution, belief that actions will effect i m p rovement, and an institutional climate that supports change will encourage participation in the process. A recognition of the need for external accountability combined with a sense of internal responsibility motivates faculty, students, and administrators to become involved. Finally, the process of generating criteria and standards encourages the participants’commitment to the results of the review. Primary responsibility for assessment rests with the faculty, students, administrators and staff directly involved in the program. The institution should offer appropriate training in criteria, standards, and measures for a program review. This Guide uses Wergin’s definitions: “criteria refers to the kinds of evidence collected as markers of quality; standards are the benchmarks against which the evidence is compared; and measures are the methods used to collect the evidence.” The following sections suggest criteria but leave the reviewers to determine the appropriate standards and measures. PROGRAM REVIEW (CONTINUED) Confidence in the quality of each program and, therefore, the fulfillment of the insti - tution’s undergraduate academic mission, results from substantial achievement in the areas identified in this Guide for the Evaluation of Undergraduate Academic Programs. By evaluating the effectiveness and efficiency of the programs and using the results for planning improvements, institutions demonstrate accountability to their internal and external constituencies. However, each campus, each program, or each study will not find all of the criteria equally relevant for their purposes. A standard set of criteria cannot adequately evaluate the characteristics of diverse p ro g rams, some of whose strengths re s i d e in their uniqueness. Therefore, circumstances should stimulate reviewers to formulate, articulate, and provide rationales for the criteria employed. n Guide for the Evaluation of Undergraduate Academic Programs 5 CURRICULUM TThis includes the courses and other educational experiences, the methods of deliver he undergraduate academic major as a program of study is defined by the curriculum. y, and the structure of requirements and electives into which these are arranged to give coherence to the program. Although some traditional curricular goals underlie all academic programs, curricula for programs bearing the same name differ across institutions as a function of variations in the training and interests of the faculty, in the background and interests of the students, and in the availability of resources both on and off the campus. The resultant variations in curricula offer students and employers the desirable situation of having choices; however, differences make design and publication of a program’s expectations especially important. Faculty must determine the programmatic goals, objectives, purposes, and effects for the major. Consequently, faculty, students, administrators, and staff must engage in an active review of the combination of in-depth study and specialized skills in an academic discipline, general education, and intellectual growth that will best meet the needs of their students. MISSION, GOALS, AND OBJECTIVES O O O O Prepare a program mission statement that outlines the goals and their relationship to the institutional mission statement. Delineate the program’s goals and objectives. Define quality as it relates to the program. Identify the learning outcomes students should demonstrate. PROGRAM DESIGN O O O O O O O O O Specify the degree requirements for the program, using the format of the catalog description of the program. Describe the congruence between course and program goals and national standards and expectations in the discipline or profession, as appropriate. Describe the congruence between course and curricular goals, courses, and prerequisite patterns. Explain the balance between breadth and depth designed in the program. Describe the methods used to ensure comparable learning outcomes among multiple sections of a course. Describe efforts to assure that required courses and electives are offered on a schedule to meet the needs of various student constituencies. Describe internship opportunities and the rationale for assigning credit. Describe departmental procedures including student participation for the development, review, and evaluation of courses. Describe the advisement procedures and the way the department assesses advisement effectiveness. 6 Guide for the Evaluation of Undergraduate Academic Programs CURRICULUM (CONTINUED) O ASSESSMENT O O O O O O O O O Analyze the effectiveness of the program in achieving its goals and objectives in the discipline. Describe the procedures, criteria, and methods used for this assessment. Analyze the effectiveness of the program in achieving its goals and objectives in general education. Describe the procedures, criteria, and methods used for this assessment. Analyze the effectiveness of the program in achieving its goals and objectives for intellectual growth. Describe the procedures, criteria, and methods used for this assessment. Describe discipline-, college-, and community-related student activities, and how the program promotes and supports faculty and student involvement and effort. Describe how the program responds to the needs of the community, if applicable. Provide follow-up data on student placement for the last five years. Indicate year(s) of data collected and total number of student responses: percentage employed in the field, percentage employed elsewhere, percentage seeking employment, and percentage continuing education. Describe the results of any alumni follow-up surveys that reflect the student outcomes. Describe the dissemination of the results of the program review to appropriate constituencies. Describe the planning processes, including those to be used to formulate and implement changes based on program review analysis. Describe the most recent evaluation of the program, what was learned from the evaluation, and what improvements resulted. Guide for the Evaluation of Undergraduate Academic Programs 7 Part-time Full-time THE FACULTY The quality of the faculty is critical to the quality of an undergraduate academic program. The qualifications of the faculty constitute traditional measures, and they are useful to show the extent to which the faculty is prepared to fulfill the mission of the program. The quality of the program also depends upon the availability of the faculty to the undergraduates and the effectiveness of the interactions, in other words, the extent to which the faculty creates a participatory culture. The following are categorized according to the five criteria for the evaluation of academic employees in the Policies of the Board of Trustees. MASTERY OF SUBJECT MATTER Faculty Summary Number of faculty assigned to the program Men Women Minorities Credentials Bachelor’s Degree Master’s Degree Doctorate Experience 0-3 years 4–7 years 8–11 years 12–15 years 16–24 years 25+ years EFFECTIVENESS IN TEACHING O O Describe the hiring procedures, including formulation of the job description, publication of the position, representation on the search committee, and responsibility for the final decision. Include copies of faculty vitae with the report. Explain how the training and interests of the faculty contribute to appropriate breadth of the program’s mission. Indicate areas, if any, in which greater strength would be beneficial. Continued next page… Guide for the Evaluation of Undergraduate Academic Programs 8 THE FACULTY (CONTINUED) O EFFECTIVENESS IN TEACHING (continued) O O Analyze the teaching loads and how they are distributed among faculty by rank, fulltime, part-time, and teaching assistants (number of courses/number of students). Highlight faculty innovations in teaching. Describe the program’s procedures for evaluating effectiveness in teaching. SCHOLARLY ABILITY O Describe the recent scholarly and creative contributions of the program faculty that are important to the program goals. EFFECTIVENESS OF UNIVERSITY SERVICE O Describe the faculty’s service to the university, such as committee work, administrative work, public service, and other activities that contribute to the fulfillment of the program’s mission in relation to the university and the community. CONTINUING GROWTH O O Identify the steps taken to assure that faculty members maintain currency in their disciplines and the activities that result in the continuing growth of the faculty. Describe the relative weight of the five criteria for evaluation and promotion in the Policies of the Board of Trustees in the consideration of rewards: mastery of subject matter, effectiveness in teaching, scholarly ability, effectiveness of University service, and continuing growth. Guide for the Evaluation of Undergraduate Academic Programs 9 O THE STUDENTS In the evaluation of the effectiveness of an undergraduate academic program, it is essential to consider the students it serves and those it might ser ve in the future. Student needs influence the design of the curriculum, the faculty to implement it, and the services to support it. The quality and success of the program depends upon the extent to which it meets the needs of its students. STUDENTPROFILES Annual cohorts for the last five years: Total Number of Majors Year to Year to to to to to Graduates Year O O O O O Describe the program strategies used to recruit students. Identify the program minimum requirements for admitting students. Explain the acceptance ratio. Compare the student diversity of the program to that of the institution, other institutions, the region, and the state. Analyze the enrollment patterns over the last five years. STUDENTNEEDS O O O O O O O O Describe the goals of the students entering the program. Describe the financial needs of the students enrolled. Describe the academic needs of the beginning and transfer students. Describe the special needs of groups of students, such as nontraditional, international, disabled, and underprepared students. Explain any academic placement procedures. Describe the diverse learning styles of the students and strategies for engaging them. Describe any orientation activities designed to introduce students to the program. Analyze attrition patterns and describe efforts to improve retention. Analyze the time students take to complete degrees. 10 Part-time Full-time Number Guide for the Evaluation of Undergraduate Academic Programs SUPPORT SERVICES Tinstructional efforts of the faculty. Furthermore, support services should contribute he academic and student services of the campus provide important support to the directly to the richness of students’academic lives. SPECIAL STUDENTSERVICES O O O O O What provisions are made for groups of students with special needs in this program for the following groups: Nontraditional students International students Students with disabilities Students who need special remedialor tutorial services Nonresidential students GENERAL STUDENTSERVICES O O O O O O O What provisions are made for all of the students, such as in the following areas: Orientation Diagnostic evaluation and placement Library and media resources Computer resources Health and counseling services Career advisement Job placement O O O Describe student and faculty satisfaction with services that support the program. Describe the general campus environment and climate. Explain their impact upon student performance. Describe how student affairs and academic affairs are coordinating efforts to contribute to student success. Guide for the Evaluation of Undergraduate Academic Programs 11 ADMINISTRATIVE SUPPORT An academic program exists within the context of an institution, and it is effective in so far as it contributes to the mission of that institution. Similarly, the institution must be committed to the quality and effectiveness of the program. The health and continued vitality of an undergraduate academic program is critically influenced by the quality and continuity of leadership and support that its faculty and students receive from the administration of the campus, starting at the level of the president. Furthermore, the institution’s leadership plays an important role in fostering a climate that supports change and makes clear the consequences of not improving. Consequently, administrators also have a need for training as evaluators. O O O O O O O O Explain the effective ways in which the administration encourages program review. Explain how the leadership helps to create an environment and a climate for academic excellence. Analyze the fairness of the reward structure of the institution and the program. Describe the ways by which the administration empowers faculty and students. Explain how budget decisions reflect the concern for quality programs and support academic robustness. Provide an organizational chart that shows the relationship of the program to the rest of the institution. Explain how the program is represented in the institution’s governance bodies and planning processes. Describe faculty development and support efforts by administration in the program area. Guide for the Evaluation of Undergraduate Academic Programs 12 BIBLIOGRAPHY In addition to the research and analysis presented in Departmental Assessment: How Some Campuses Are Effectively Evaluating the CollectiveWork of Faculty (American Association for Higher Education), we appreciate the permission to include the Re f e re n c e s used by Jon F.Wergin and Judi N. Swingen in their monograph. Barr, F. B., and J. Tagg. (November/December 1995). “From Teaching to Learning: A New Paradigm for Undergraduate Education.” Change 27(6): 12-25. Bensimon, E.M., and H.F. O’Neil. (1998). “Collaborative Effort to Measure Faculty Work.” Liberal Education 84(4): 22-31. Csikszentmihalyi, M. (1997). Finding Flow: The Psychology of Engagement With Everyday Life. New York, NY: Basic Books. Dill, D. (June 1999). “Implementing Academic Audits: Lessons Learned in Europe and Asia.” Unpublished manuscript prepared for the Academic Audit Seminar, Chapel Hill, NC. Dill, W.R. (July/August 1998). “Specialized Accreditation: An Idea Whose Time Has Come? Or Gone? Change 30(4): 19-25. Eckel, P. (Winter 1998). “Thinking Differently About Academic Departments: The Academic Department as a Team.” In Teams and Team-work in Institutional Research, edited by S. Frost. New Directions for Institutional Re s e a rch, no. 100. San Francisco, CA: Jossey-Bass. ———, B. Hill, and M. Green. (1998). On Change: En Route to Transformation. Washington, DC: American Council on Education. Fairweather, J. (1996). Faculty Work and the Public Trust: Restoring the Value of Teaching and Public Service in American Academic Life. Boston, MA: Allyn & Bacon. Ga f f, J. G., J. L. Ra t c l i f f, and Associates. (1997). Handbook of the Underg raduate Curriculum: A Comprehensive Guide to Purposes, Structures, Practices, and Change. San Francisco, CA: Jossey-Bass. H a w o rth, J.G., and C.F. Conrad. (1997). Emblems of Quality in Higher Education: Deve l o p i n g and Sustaining High-Quality Programs. Boston, MA: Allyn & Bacon. Joint Committee on Standards for Educational Evaluation. (1994). Program Evaluation Standards. 2nd ed. London, England: Sage. Kuh, G.D. (1999). “How Are We Doing? Tracking the Quality of the Undergraduate Experience, 1960’s to the Present.” Review of Higher Education 22(2): 99-119. Larson, R.S. (1997). “Organizational Change From the ‘Inside’: A Study of University Outreach.” Dissertation, Michigan State University. Massy, W.F. (1997). “Teaching and Learning Quality Process Review: The Hong Kong Programme.” Quality in Higher Education 3(3): 249-262. ———, A. K. Wi l g e r, and C. Colbeck. (July/August 1994). “Ove rcoming ‘Hollowe d ’C o l l e g i a l i t y : Departmental Cultures and Teaching Quality.” Change 26(4): 10-20. Menges, R.J., and Associates. (1999). Faculty in New Jobs: A Guide to Settling In, Becoming Established, and Building Institutional Suppor t. San Francisco, CA: Jossey-Bass. Guide for the Evaluation of Undergraduate Academic Programs 13 Mets, L.A. (1998). “Implementation Strategies in a University Setting: Departmental Responses to Program Review Recommendations.” Dissertation, University of Michigan. O’Neil, H.F., E.M. Bensimon, M.A. Diamond, and M.R. Moore. (1999). “Academic Score c a rd . ” Unpublished manuscript, University of Southern California. Pitts, J.M., W.G. White Jr., and A.B. Harrison. (1999). “Student Academic Underpre p a re d n e s s: Effects on Faculty.” Review of Higher Education 22(4): 343-365. Preskill, H., and R.T. Torres. (1999). “Evaluative Inquiry for Organizational Learning.” in Learning Around Organizations: Developments in Theory and Practice, edited by M. E. Smith, L. Araujo, and J. Burgoyne. London, England: Sage. Serow, R.C., C.E. Brawner, and J. Demery. (1999). “Instructional Reform at Research Universities: Studying Faculty Motivation.” Review of Higher Education 22(4): 411-423. Smith, V. (1998). “The Futures Project: A Two-Year Activity of 18 Independent Colleges and Universities in California.” Final report to the James Irvine Foundation. Staw, B.M. (1984). “Motivation Research Versus the Art of Faculty Management.” In College and University Organization: Insights From the Behavioral Sciences, pp. 63-84, edited by J.L. Bess. New York, NY: New York University Press. Thune, C. (June 1999). “European Developments in Academic Quality Assurance.” Paper presented to the Academic Audit Seminar: Lessons Learned in Europe and Asia, Chapel Hill, NC. Tierney,W.G. (1999). Building the Responsive Campus: Creating High Performance Colleges and Universities. London, England: Sage. Wergin, J.F. (1994). The Collaborative Department: How Five Campuses Are Inching Toward Cultures of Collective Responsibility.Washington, DC: American Association for Higher Education. ———, (June 1998). “Assessment of Programs and Units: Program Review and Specialized Accreditation.” Paper presented at the AAHE Assessment Conference, Cincinnati, OH. ———, and J.N. Swingen. (2000). Departmental Assessment: How Some Campuses Are Effectively Evaluating the Collective Work of Faculty. Washington, DC: American Association for Higher Education. Zusman, A. “Issues Facing Higher Education in the Twenty-First Century.” (1999). In American Higher Education in the Twenty-First Century, pp. 109-150, edited by P.G. Altbach, R.O. Berdahl, and P.J. Gumport. Baltimore, MD: Johns Hopkins. Guide for the Evaluation of Undergraduate Academic Programs 14 NOTES Guide for the Evaluation of Undergraduate Academic Programs 15 NOTES 16 Guide for the Evaluation of Undergraduate Academic Programs