Course & Program Evaluation

In every course we teach and as a part of the administration of a our programs of study, utilizing opportunities and mechanisms that help us reflect on the effectiveness and value of our course offerings is essential part of our improvement process.

Use the links below and in the navigation to the left to find information on some of the activities and mechanisms that we have in place to help us maintain a quality program.

For more details and additional links see The Penn State Quality Assurance e-Learning Design Standards.

12 quality assurance standards that all Penn State WEB courses should include.

  1. Navigation
    The course has a consistent and intuitive navigation system enabling students to quickly locate course information and materials.
  2. Student Orientation
    A course orientation is used to familiarize the students with the course.
  3. Syllabus
    Students have easy access to a course syllabus which contains crucial course information and requirements they need to know about the course prior to starting.
  4. Instructor Response and Availability
    Instructor response time and availability is clearly communicated to the student.
  5. Course Resource Requirements
    Hardware, software, or specialized resources required are clearly communicated to the students.
  6. Technical Support
    Information regarding access to technical support is clearly communicated to the students.
  7. Accessibility Requirements
    The course adheres to University policies and guidelines regarding accessibility.
  8. Learning Objectives
    The course contains learning goals and objectives.
  9. Learning Activities and Assessment
    The course learning activities and assessment serve to stimulate student interactions with the course content and determine how well student performance achieves the course goals and learning objectives.
  10. Copyright Requirements
    The online course adheres to the current University policies for the use of third-party copyrighted material or is able to provide evidence of appropriate copyright clearance.
  11. Course Functionality
    All aspects of the course perform properly and support student progress.
  12. Student Input for Course Improvements
    Opportunities are provided to gather input from students on an on-going basis in order to inform course improvements.

Motivation

Review of courses and the overall curriculum is a part of the department's strategic initiatives for the growth of sustainable online programs. The MAS online program will benefit from formalized and rigorous scrutiny of the topics covered as well as the manner in which these are covered and build on each other from course to course throughout the program.

Overview

Each semester a small number of courses will be identified for review and will involve examination of aspects of the overall course as well as the content and approaches taken in each of the course lessons as depicted below. An instructional designer facilitates this course/curriculum review process.

Typically three faculty who are currently teaching the course or have recently taught the course will be assigned as reviewers. Ideally, one of these faculty might also be teaching this same course in residence. These faculty will meet on a regular basis throughout the semester to discuss the lessons as they are presented in the course. Meeting throughout the semester, as the course is instructed, keeps reviewers close to the materials and student use of these. Not only are overall course topics and issues discussed (worksheet) but each lesson also is reviewed (worksheet). The faculty reviewer's work results in recommendations for the course and individual lessons.

At some point in the semester additional faculty are assigned as 'Link Reviewers'. Typically these are faculty that are currently teaching or have recently taught a course that leads up to or a course that builds on the course under review. These faculty, having a vested interest in the associated course, looks for topics that set a foundation for or need foundational understanding from the prerequisite course. The recommendations of where these links should exist and the degree to which these links are supported are added to the recommendations to the course.

Outcomes

  1. The recommendations from the faculty course reviewers and the faculty link reviewers is used to generate a list of development tasks that are needed to revise the course. As much as is possible, these changes are put in place immediately. Should the amount of revision warrant, supplemental contracts for faculty development of these items will be offered.
  2. A list of concepts, topics, methods and/or procedures are developed that will articulate what a student must know or be able to do as a result of participating successfully in the course. Consequently, these then become the basis of coverage for the problems that are given on the final exam in the course.
Overview

Peer review of teaching—like the peer review of research—is a widely accepted mechanism for promoting and assuring quality academic work, and is required for the purpose of promotion and tenure at Penn State. For provisional faculty (not yet tenured), it is recommended that peer reviews should occur at least once per year and in a variety of courses. For faculty being reviewed for promotion, it is better to have a series of peer reviews over time rather than several in the fall immediately preceding the review. The peer review process in resident instruction typically involves a faculty reviewer observing a peer’s classroom. The reviewer then summarizes their observations in a document that is to be included in the reviewee’s dossier. As there are no online sessions to observe, we utilize a peer review guide.

The Peer Review Guide for Online Teaching at Penn State is based on the “Seven Principles for Good Practice in Undergraduate Education,” a summary of 50 years of higher education research that addressed good teaching and learning practices.

Good online teaching...

  1. Encourages communication between students and faculty,
  2. Encourages active learning strategies,
  3. Gives prompt feedback,
  4. Emphasizes time on task,
  5. Communicates high expectations,
  6. Develops reciprocity and cooperation among students, and
  7. Respects diverse talents and ways of learning.

Chickering, A. & Gamson, Z. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin (39) 7.

While instruments such as the Student Rating of Teaching Effectiveness (SRTE) provide a measure of student satisfaction with a course, the Seven Principles provide a useful framework to evaluate the effectiveness of online teaching and learning.

The Masters of Applied Statistics program as well as instructors teaching other online courses through the Department of Statistics will use a version of the Peer Review Guide for Online Teaching at Penn State (see Dutton Institute Site) that has been adapted to meet the needs of our online courses. The process involves three steps once the Director of Online Programs has assigned peer reviewers to review teaching in a specific course.

Peer Review Process

  1. Reviewee completes input form.

    The Peer Review instructor Input Form is completed by the instructor and sent to the reviewer in advance of the peer review. 
  2. Reviewer completes peer review guide for online teaching at Penn State.

    After reviewing the completed “Peer Review Instructor Input Form,” the peer reviewer uses the Peer Review Guide for Online Teaching to work through the online course, completing the following:

    1. Penn State Quality Assurance e-Learning Design Standards Rubric 
    2. 2 or 3 of the Seven Principles (The reviewer notes the instructor’s strengths and provides suggestions for improvement for selected Principles in the spaces provided.) 
       

      NOTE: Reviewers should feel free to ask questions of the instructor any time clarification or information is needed during the review process. Email and videoconference meetings can be used to communicate throughout the process.

  3. Reviewer submits summary.

    As noted in the Peer Review Guide above, both the peer reviewer and the course instructor discuss the instructor’s involvement in the course, using the notes from the "Peer Review Guide for Online Courses" as the basis for this conversation. The peer reviewer summarizes this conversation in a single document. This document, along with a copy of the completed “Peer Review Guide for Online Courses”, are submitted as attachments to a dropbox in the STAT Online Learning ANGEL Group where the Director of Online programs will access these.