Information technology tools and resources at the UW
UW Experience and Relationships Benefit the Canvas Gradebook
With years of experience designing and supporting Catalyst Web Tools, ACA user experience designers, engineers, and service managers are uniquely positioned to influence the evolution of the Canvas gradebook. Multiple cycles of user studies, research, close collaboration with the support team, and long-standing relationships with instructors and business units across campus have given the ACA team a deep understanding of the wide-ranging, grading needs at the UW. And although it has taken time for Instructure to respond to the feedback coming from the ACA team, that is beginning to change. A recent visit to UW by Instructure user experience designers is driving improvements to Canvas gradebook.
Building Expertise in Grading
How did the ACA team develop this expertise? Karin Roberts, the UW-IT Canvas service manager, recalls, “The faculty technology advisory board asked us to build an online gradebook, which we released in 2008. We began by doing user research with faculty to understand their grading needs and practices (such as scales, grading schemes, and workflow). We built a usable tool—Catalyst GradeBook–that met fundamental needs, and also allowed instructors to continue to use Excel, and import grades from scorepak and WebQ. It was widely used at UW, and then we learned more about grading requirements as we built GradePage for online grade submission.”
The contextual knowledge and relationships built over the years continue to provide real efficiencies for the UW. For example, Jason Civjan, UX Team Lead, points to engineer Jim Laney’s deep knowledge of UW-IT databases. “When we were first evaluating Canvas, Jim could tell us right away which data from Canvas our systems would be able to ingest, and which would require some kind of translation.” Years of research and designing grade tracking and input tools meant that the entire team could spot potential red flags in Canvas quickly.
It is this wealth of experience and relationships that enabled the UW Canvas team to provide feedback to Instructure since first piloting UW Canvas in 2011-12. It also positioned the UW as a key Instructure customer (its first institution of its size). Karin said, “We knew from the pilot that the Canvas gradebook had usability issues, that the functionality was limited and didn’t meet the varied grading needs at the UW. Despite raising those issues weekly with the account rep and in every product advisory group meeting, Instructure was slow to respond.”
But when Karin began coordinating feedback with a group of R1 peer institutions using Canvas, Instructure acknowledged that it was time to address the gradebook issues. They really took notice when Karin shared how a suboptimal gradebook is a major obstacle for instructors considering Canvas. The Canvas gradebook is often the last tool that instructors begin using, and some instructors are using Canvas without its gradebook. A recent UW Canvas survey showed that the two most frequently used tools in addition to Canvas are Catalyst GradeBook and Excel.
As a first step to improving the gradebook, Instructure created a new gradebook product team. Next, Instructure hired a user experience designer to lead the new product team. This in itself was a shift for Instructure; previously, marketing or subject matter experts had helmed product teams. Under the new leadership, the experience of the Canvas gradebook user is beginning to drive more decisions.
Canvas Visits the UW
The new Canvas gradebook team leads, Christi Wruck and Patrick Cox, came on board in September 2014. Karin organized a site visit, an opportunity for the gradebook team to see and hear how actual UW users interact with the grading system and to deepen their understanding of UW instructors’ needs and challenges. By early October, they were on the UW Seattle campus (the first higher ed institution they visited), interviewing instructors. Meeting users in person provides the larger perspective not available in user forums. The designers met with instructors of Anthropology, English, Law, Psychology, Chemistry, Biology, Public Affairs, Art, and Pharmacy. They heard different issues and requirements at each interview, illustrating the complex set of needs for grading across the UW.
To maximize the benefit of the interviews, Christi and Patrick first solicited complaints and pain points. “We want to let people vent first,” said Christi, “and to discover the specific problems instructors encounter.” They then move on to card sort exercises that help people evoke the challenges they encounter when using gradebook. The first set of cards features typical actions a Canvas user would want to accomplish, e.g., “enter grades.” Christi and Patrick asked instructors to organize the cards according to the frequency with which they perform these actions. A second set of cards features emotive words (e.g., confident, relaxed) and participants were asked to choose the three words that best describe how they want to feel when using Canvas. They then take pictures to capture the data.
Input from users is always on display, informing impromptu and formal conversations about features.
Instructure UX Designers capture the card sort exercises performed by instructors they’ve interviewed,
then post the pictures in a central location back at the home office.
Some of the primary challenges faced by UW instructors using Canvas gradebook are working with large classes, and collaborating with teaching assistants. In addition, UW schools use a range of grading scales: percentages, 4.0, pass/fail. The Canvas gradebook does not currently support this complexity, requiring UW-IT to create its own grading conversion tool. The gradebook also needs to support basic import and export from Excel sheets, which it does not currently provide.
The Road Ahead
Christi and Patrick are focused on improving the fundamentals. “Fundamental tasks need to be clean, clear, and fast. We want to improve performance so that getting data is quicker. The primary objective is to get data in and out quickly and easily.” They are now working on new designs based on interviews with users from the UW and elsewhere. “Once we have a version that can be tested, we will show that to users and get feedback.
“Our goal,” continued Patrick, “is to improve something in the gradebook in every release of Canvas. We want our users to see that we’re working on it. We don’t want to do a bunch of stuff behind the scenes for a year and then suddenly unveil a completely new version of gradebook. Our long term goal is to make the whole thing simpler, to surface the tasks that people do the most. It won’t be a rapid process of advancement, but all the changes will come from the interviews with users, supported by data points from forums and customer support.”
Karin is encouraged by Instructure’s efforts. “Christi and Patrick did a really good job of collecting information, and they have a good user experience process. Instructure is listening to us.” Improvements to the gradebook will not happen overnight. But changes are in the works, largely in part due to the UW Canvas team’s persistence in advocating for users.