There were originally six NS outcomes. The provisional “operationalized” set contains three. We generally accepted them (with two minor tweaks) as a workable starting point for assessing the Natural Science with Lab portion of the Common Course of Study. We believe that our courses offer much richer learning experiences than is reflected in these minimal outcomes, but these outcomes do represent one possible core of things that we agree are important. We decided it would be more productive to spend our effort discussing the implementation. Once a working assessment plan has been implemented, it should be relatively straightforward to consider additional SLOs.
There was some question whether all of our courses really meet all of the SLOs. Some objectives fit more naturally in some courses than in others. For example, questions of uncertainty can be quite subtle. On the other hand, the CCS outcomes should not be about sophisticated understanding, but should address a very basic competency level, suitable for a student who has taken a single course in a particular area.
The first tweak we made to the language was to replace “graphs, tables, and models” with “graphs, tables and/or models” in NS2. This was simply to avoid the combinatoric explosion of assessing each of “create,” “interpret,” and “evaluate” for each of the three products. The second tweak was to consider replacing “critically evaluate” with “evaluate” in NS2, since the word “critically” doesn’t really add anything. We anticipated further tweaks of language might be necessary to keep the SLOs aligned with those in other areas of the CCS, particularly STSC, but spent our time elsewhere.
In most cases, samples of student work on specific assignments would be evaluated.
The product would be typically be a student homework or test paper, or lab report, or other product. A random sample of student work would be assessed. (If the instructor is assessing work on a single test question, he or she could also possibly just evaluate the work of all students, and in some cases that might actually be simpler.)
Often, there are specific questions on exams or specific lab assignments that could directly address the SLOs. In some cases, a single question (e.g. “question 7 on Test 2”) might directly address a particular SLO. Examples included specific graphs on Biology or Physics tests. In other cases, a more extensive assignment, such as the Biol 102 invasive species lab project, would be a natural tool for assessment.
We considered two different models for the collection and evaluation of evidence. In one model, the instructor plays the central role, evaluating student work and passing a summary on to a divisional assessment committee. In the second, the instructor simply passes on a random sample of the student work, along with a brief description and answer key, to a divisional assessment committee.
Model 1: We briefly discussed the possibility of the instructor evaluating the evidence, and passing along a summary of the results to some college-wide assessment body. The instructor has to evaluate the work anyway in the normal course of grading. This is the norm in Engineering assessments. There was some concern about possible conflict of interest – would a faculty member be likely to judge his or her own course as anything other than successful? Also, there was some concern that assessing a course at the individual or departmental level might inadvertently be conflated with the evaluation of teaching for tenure and promotion.
Model 2: A second model is for the instructor to pass a random sample of student work (along with a brief explanation of the expected answer) on to some sort of divisional assessment committee. Student names and identifying course information would be removed to the extent possible (though we expect that in many cases it would be fairly easy to identify the course and instructor). We discussed whether faculty on such a committee could assess student work from another department, or whether specialized knowledge would be needed. Bearing in mind the very basic nature of the SLOs, we thought it likely that suitable questions could be designed with relatively modest effort on the part of individual faculty members.
Model 1: In this model, the instructor evaluates student work directly and can directly use the feedback to improve student learning.
Model 2: Results are anonymized, and overall student learning is assessed, but individual instructors or courses are not. There is no obvious direct feedback mechanism. Even if a particular course consistently fails to meet its objectives, there is no way to inform either the instructor (so the course can be improved) or CEP (so the NS designation can be removed). Of course the instructor still sees the student work as part of the normal grading for the course, so we expect that if students were consistently failing to meet the objectives, the faculty member would be aware of that.
We did imagine the committee could do two things to help improve student learning: First, we expect the committee would produce summary reports highlighting successful and unsuccessful experiences, and pass them back to instructors teaching NS courses. Second, the committee could help faculty develop more effective assessment questions based on successful examples in other similar courses.
We did not develop any complex multi-year cycle. We imagined assessing all three SLOs in the first year. In subsequent years, perhaps all three would be assessed, but with a rotating emphasis. Practically speaking, this could perhaps be done by having a larger sample for one SLO, but smaller samples for the other two.
Any such a divisional assessment committee would need long term continuity, not the rapid turnover we often get on regular elected faculty committees. The IRB was suggested as one possible model.
Assessment needs to not be a significant burden, or we run the risk of driving faculty away.
Perhaps a hybrid model could be implemented. Instructors would evaluate their own courses, but would pass up the statistical results and supporting samples of student work. The divisional assessment committees would not have to evaluate all the work, but could still coordinate and advise.
Some courses (which meet the NS requirements) are taught by visitors or adjuncts. We expect that particular care might be needed to ensure the assessment plan will work well in those cases.