Skills Quiz
A closely monitored launch - using data to drive rapid design iterations of a novel feature, emphasizing speed to value. See it live
Background
edX, founded in 2012, is an online provider for Massive Open Online Courses (MOOCs) as well as degree-based programs. In addition, they've developed an open-source Learning Management System (LMS) for use by course creators around the world.
​
Part of the business model, known as edX Enterprise, was packaging the existing course catalog for use by third parties as an educational offering to their members.
​
Creating a skills architecture
In contrast to traditional edX learners, where course completion rates are a key indicator of success, enterprise customers measure success based on enrollment rate - what percentage of their learners enroll in at least one course. In order to both provide more granular data to these customers and encourage additional enrollments, edX partnered with the Burning Glass Institute to tag course content based on skills they identified. The hypothesis our product team had was that a skills-based architecture provides value for enterprise learners because it:
​
-
Is a tangible way to demonstrate knowledge for future job opportunities
-
Empowers organizations to upskill their workforce without additional staff
-
Can help learners advance their careers within an organization
To prove out this hypothesis, I worked with our product manager to design a feature, Skills Quiz. Our product hypothesis was:
"If enterprise learners could search for courses based on skills they're interested in, then learners would feel be able to find relevant courses more easily, resulting in increased enrollment rates."
Initial launch
I'd originally supported the design work for the launch of the feature, including wireframes and guiding developers who were less familiar with our design system (Paragon). Our goal was to leverage the existing design system as much as possible to avoid delays due to the approval process for new components.
​
The design concept was straightforward, using a quiz with conditional logic to generate a series of recommended courses based on your learning goals.
Once we'd made it through the initial launch, the product manager and I met weekly to review the enrollment data, comparing the learners who'd used the skills quiz to those that didn't in terms of conversions and learning time. We noticed there was a small difference between the two populations, but it was less significant than we expected. I recommend we deploy a HotJar intercept survey to gather feedback from our target users.
What we heard
01
Identifying as a student
We had a set of learners who identified as students and didn't see a clear option under "current role". This left them a bit confused as to how to proceed, since it also wasn't clear if this field was required or optional.
02
Not negative, but not positive
Overall, learners had a fairly neutral opinion of how effective the tool was in helping them find a course (based on NPS=6.7
03
It did help, if you made it through
Lastly, we found that users who made it through to the course recommendations had a higher conversion rate, but only a small percentage of users made it that far.
Next steps
Based on that feedback, I consolidated the experience from 2 screens, down to a single screen, while making small changes to address specific concerns:
-
Updated copy to better reflect what the tool was intended to accomplish
-
Added visual branding through using brand shape and imagery in line with brand guidelines
-
Added an option for students
Continued monitoring
With new designs shipped, we continued to monitor both qualitative and quantitative feedback channels to see our expected increase in conversion for learners.
​
After launch, we started to see a small improvement over our original target metric (baseline conversion, in Teal below)​​​​​​​
Next please!
I continued to iterate on the designs based on feedback we received from HotJar, input from other designers, and data we were seeing.
​
-
Pulled in additional brand elements such as color and shape to reinforce edX as the service provider
-
Made the student identification explicit
-
Reworked introductory copy
​
With this updated design, we started to see the conversion rate not only surpass the baseline search experience, but also surpass search-by-skills, a feature developed by another team.
Final designs
I did one final iteration to consolidate branding elements into a single focus area (the header) and reduce cognitive load. We successfully demonstrated that engaging users through a skills-based recommendation feature drove increased enrollments for the enterprise customers. This feature was brought to the non-enterprise experience based on our evidence.
