Information technology tools and resources at the UW
Work in Progress: Canvas Gradebook
Since first piloting UW Canvas in 2011-12, the Canvas service team has communicated often with Instructure about issues with Canvas gradebook. Karin Roberts, UW Canvas Service Manager, said, “We knew that the Canvas gradebook had usability issues, that the functionality was limited and didn’t meet the varied grading needs at the UW.” UW schools use a range of grading scales: percentages, 4.0, pass/fail. “Despite raising those issues frequently with our account representative and in product advisory group meetings, Instructure was slow to prioritize improvements to the gradebook above implementing other features.” But after coordinating feedback with a group of R1 peer institutions using Canvas, Instructure acknowledged that it was time to address the gradebook issues. They took notice when Karin pointed out that a suboptimal gradebook is a major obstacle for instructors considering adopting Canvas.
Putting User Experience First
In response to the R1 group’s feedback, Instructure created a new gradebook product team. Next, Instructure hired a user experience designer to lead the new product team. This in itself was a shift for Instructure; previously, marketing or subject matter experts had helmed product teams.Under the new leadership, the experience of the Canvas gradebook user is beginning to drive more decisions.
Instructure Designers Visit UW Instructors
The new Canvas gradebook team leads, Christi Wruck and Patrick Cox, came on board in September 2014. By early October they were on the UW Seattle campus (the first higher ed institution they visited), interviewing instructors about their experience using the gradebook. Christi considers herself first and foremost an advocate for the user. “As product manager, I need to be the voice of customer.” In her new position, Christi wanted to get out and talk to Canvas users right away, to balance the input she was receiving from colleagues. “Everyone — e.g., sales, engineers — has an opinion about how to design features. There’s pressure from different parts of the organization [about feature decisions].” So Christi and Patrick do a lot of research. “We look at forums and client services, which provide lots of data points. But users who have important information to share don’t necessarily show up in chat rooms. So we like to go out and engage the user directly, to see how they are using the product in their specific context. This experience personalizes the user, and helps us understand use cases.”
Understanding the Larger Problems
Meeting users in person also provides the larger perspective not often present in user forums. Christi explains, “In forums, users will often say things like, ‘I want a button to do X.’ Those requests don’t necessarily get at the core problem. If we granted all those requests, the interface would quickly fill up with buttons. And it wouldn’t help us formulate a long-term plan for changing the product to better meet user needs. It’s our job to understand the larger problems, the challenges that instructors face.” Some of those challenges include working with large classes, and collaborating with teaching assistants. The gradebook also needs to support basic import and export from Excel sheets, which it does not currently provide.
Adapting Canvas gradebook to a complex grading environment is challenging in part because Canvas is designed for use in K-12 classrooms as well as higher ed. But of course grading needs in higher ed differ vastly from those in K-12. Higher ed instructors interact with Canvas gradebook quite differently. For K-12 teachers, the gradebook is the starting point; they may go there multiple times a day, entering information about levels of mastery rather than a grade. And, K-12 instructors have the same students all day long. Instructors at the UW Law School may go to the Canvas gradebook twice in a quarter; instructors in English may go to it several times a week.
Site visits stay with Christi and Patrick long after they leave campus. “We still talk about the UW visit. We remember their voices. It might be the most productive session we’ve ever had.” The designers heard different issues and requirements at each interview, illustrating the complex set of needs for grading across the UW. The designers met with instructors of Anthropology, English, Law, Psychology, Chemistry, Biology, Public Affairs, Art, and Pharmacy. Christi and Patrick described how invested the UW instructors were, and how willing to share the “real obstacles” they encounter using Canvas gradebook. “They want to help make the gradebook more usable for everyone.”
To maximize the benefit of the interviews, Christi and Patrick first solicit complaints and pain points. “We want to let people vent first,” said Christi, “and to discover the specific problems instructors encounter.” Once the venting is done, they move on to card sort exercises that help people evoke the challenges that they encounter when using gradebook. The first set of cards features typical actions a Canvas user would want to accomplish, e.g., “enter grades.” Just this one data point begins to illustrate the variety of grading behaviors at the UW.
Christi and Patrick asked instructors to organize the cards according to the frequency with which they perform these actions, and then priority. The next set of cards features emotive words (e.g., confident, relaxed) and participants were asked to choose the three words that best describe how they want to feel when using Canvas. They then take pictures to capture the data.
Input from users is always on display, informing impromptu and formal conversations about features.
Instructure UX Designers capture the card sort exercises performed by instructors they’ve interviewed,
then post the pictures in a central location back at the home office.
The Road Ahead
Christi and Patrick are focused on improving the fundamentals. “Fundamental tasks need to be clean, clear, and fast. We want to improve performance so that getting data is quicker. The primary objective is to get data in and out quickly and easily.”
“We also want to re-organize the user experience so that it’s easier and clearer to accomplish necessary tasks,” said Christi. “We plan to clean up a lot of the interface to make it more usable and to simplify user interactions. Doing that means figuring out what’s in the way. The features that people need may be there, but they aren’t obvious. Some features are buried. For example, grading schemes. Setting them up and applying them is not apparent or intuitive. We want to drive down the number of clicks it takes to do things.”
They are now working on new designs based on interviews with users, including those they spoke with at the UW. “Once we have a version that can be tested,” said Patrick, “we will show that to users and get feedback.”
“Our goal,” Patrick continued, “is to change something in the gradebook in every release of Canvas. We want our users to see that we’re working on it. We don’t want to do a bunch of stuff behind the scenes for a year and then suddenly unveil a completely new version of gradebook. Our long term goal is to make the whole thing simpler, to surface the tasks that people do the most. It won’t be a rapid process of advancement, but all the changes will come from the interviews with users, supported by data points from forums and customer support.”
Karin is encouraged by Instructure’s efforts. “Christi and Patrick did a really good job of collecting information, and they have a good user experience process. Instructure is listening to us.” Improvements to the gradebook will not happen overnight. But changes are in the works, largely in part due to the UW Canvas team’s persistence in advocating for users.