Assessment of courses

The CyberCourse project (aka Cyco) improves university skills courses, like programming, writing, and statistics. Students learn how to do tasks independently. Key elements of effective Cycourses (courses built with Cyco) are:

  • High quality textbook replacements that emphasize problem solving, through patterns and big ideas.
  • Flipped, blended courses with many exercises.
  • Formative feedback from humans, for every exercise submission.
  • One-on-one help from experts for every student.

What about course assessment? Cycourses are assessment friendly, by nature. Course assessment can be done as a normal part of Cycourse operation. It doesn’t need to take extra money or time.

That’s quite a claim! Let’s see how it works.

Exercises

Students do many exercises, like this:

Exercise

Here’s a sample:

Two cows are hanging out. One says: “Mad cow disease is overrated.”

The other replies: “Quack.”

Each exercise has a rubric:

Rubric items

Graders use the rubrics to give students formative feedback. Here’s one student’s feedback for a dog joke exercise:

Feedback

A joke like this:

A three-legged dog hopped into a saloon in the Old West, and said, “I’m lookin’ for the man that shot my paw.”

Students get reports on their own performance. Instructors get reports on class performance. Now, what about course assessment?

Assessing course outcomes

Suppose faculty define a set of outcomes for a course. For example, for a programming course:

Write a program that reads numeric data from a file, and outputs descriptive statistics: mean, standard deviation, sample size, maximum, and minimum. The program should be able to handle missing data.

Perhaps the outcome could be less concrete:

Write a program that loops over data in a file. There should be a filter in the loop. The program should use accumulation variables in the loop. The program should output the accumulation variables, as well as computed functions of the accumulation variables.

However faculty choose to specify outcomes, they can create rubrics, with items like:

  • Code should be indented appropriately.
  • Variables should be named to reflect their purpose.
  • Comments should identify the main parts of the program.

Students could complete course assessment exercises at the end of the course. They could be a normal part of the students’ assessment, or separate items.

No new assessment tools are needed. The usual feedback mechanism suffices, because Cycourses are naturally assessment heavy. Why? Because frequent formative assessment helps with skill development.

Course outcome assessment could be built into a Cycourse from the very beginning. For the win!

Fine-grained assessment

But wait, there’s more! Remember that each exercise has a rubric, like this rubric for a cow joke exercise:

Rubric items

Rubric items have standardized feedback responses, like these for the Funny rubric item:

Rubric item feedback

Graders click on the appropriate phrases when evaluating student work. If the phrase is from the Good group, the student has complied with the rubric.

If the student doesn’t get it right, different reasons can be added to the Needs work and Poor sections. The grader can click on the appropriate phrase, telling students the specific mistake they made.

When a grader clicks, s/he is giving fine-grained judgment on whether a particular student met a particular rubric item on a particular exercise. To say it another way: a Cycourse has data on every rubric item, for every exercise, for every student. For every section, for every semester. That’s a lot of data!

Above, we discussed assessing outcomes at the end of semester. That can tell us whether students are performing as expected, but not why. Fine-grained rubric item analysis might help. It can show which rubric items students were having trouble with at different points in the semester, and what errors they are making. Course authors and instructors could use that information to improve their work. Fine-grained rubric item analysis gives specific guidance about what to improve.

The fine-grained data is gathered as part of the normal operation of a Cycourse. There are no extra costs, in money or time. Again, Cyco is assessment friendly.

Summary

Faculty love to do course assessment. Well, not really. Many see it as a waste of their time, and the students’.

With Cyco, course assessment is a natural part of every course. It doesn’t have to take any more work or time. Further, analysis of fine-grained rubric item response data can guide faculty to the specific parts of a course that should be improved.