Project Background
Joe Pollock and his design team completed an analysis and evaluation of the usability of Papa John’s online ordering system. In order to complete the analysis and evaluation of the domain, we formed objectives along with critical questions that addressed the concerns of the stakeholders.
Objectives:
1. Discover and document any issues that users are experiencing with the interface.
2. Highlight areas within the website that need improvement.
2. Highlight areas within the website that need improvement.
Questions:
1. Are the expectations of ordering online being met?
2. Are users successfully completing their tasks with the current interface?
3. Do individuals complete tasks in a similar or different way?
2. Are users successfully completing their tasks with the current interface?
3. Do individuals complete tasks in a similar or different way?
Website Being Assessed:
Methodology
For this usability study, we recruited four participants. Each participant individually completed a moderated testing session. The testing sessions began with an icebreaker activity in order to help the participant get comfortable and think aloud. After the icebreaker activity, I guided each participant through a usability testing session that focused on three specific tasks designed to help evaluate the website papajohns.com. Each testing session was video recorded.
We asked participants to perform the following three tasks:
• Place three pizza orders varying in complexity
• Request discounts and coupons without signing up for an account
• Register a complaint with the corporate office
• Request discounts and coupons without signing up for an account
• Register a complaint with the corporate office
After we conducted the testing sessions, I viewed the session videos and collected data. After collecting the data and analyzing it, I synthesized it into tangible quantitative and qualitative metrics. I analyzed quantitative data using metrics and synthesized it via mathematical equations. I analyzed the qualitative data using an affinity map, and then synthesized using a matrix.
After synthesizing both quantitative and qualitative data, I identified key user issues with the interface and made product recommendations.
Test Participants
We recruited four participants for individual usability testing. Below is the demographic breakdown of the participants.
Usability Testing Sessions
Each testing session was video recorded and ranged in duration from 15-30 minutes. During the testing sessions, participants were asked to perform three separate tasks on the Paps John's website. Below are user quotes from each participant highlighting the usability sessions.
Participant 1
Participant 2
Participant 3
Participant 4
Synopsis of Findings
The usability testing completed shows that overall the website performs very well. All four participants could complete the vast majority of tasks with relative ease. There are a few minor areas of the website that need improvement, which I further outline below.
Each participant performed three tasks (in total) during their session. The average time of the sessions was under twenty minutes. The total success rate of the participants was over 90%. The average number of errors (per session) was less than three per participant. Below is a table illustrating the metrics from each session.
Testing Sessions: Quantitative Data
Analysis: Task 1
Task 1:
We are going to be looking at the papajohns.com website. Even if you are not a Papa John’s fan, imagine that the people you are with are fans of Papa John’s, and that is where you will be ordering the pizza. There are a bunch of people at this party, and you need to order 3 pizzas. They are:
• Mushroom pepperoni
• Half onion and half sausage with light sauce
• A specialty pizza because you want some variety
• Half onion and half sausage with light sauce
• A specialty pizza because you want some variety
Analysis:
All four participants could complete the task; the average time for completion was approximately five minutes. Below is a table illustrating the metrics from Task 1.
Task 1: Quantitative Data
Key findings from Task 1:
1. Users experienced confusion about where to begin their order.
2. Users felt overwhelmed by too many steps/choices presented at once.
3. Users found some fonts and graphics hard to see.
4. Users prefer not to be upsold.
2. Users felt overwhelmed by too many steps/choices presented at once.
3. Users found some fonts and graphics hard to see.
4. Users prefer not to be upsold.
Analysis: Task 2
Task 2:
You want to sign up for deals and coupons, but you don’t want to register. You just want to give them your email. Can you do this, and what do you think you will receive by email?
Analysis:
Three out of four participants could complete the task; the average time for completion was under three minutes. Below is a table illustrating the metrics from Task 2.
Task 2: Quantitative Data
Key Findings from Task 2:
1. Users experienced confusion about too many sign-up options.
2. Users had difficulty finding promotional offers by email.
3. Users were uncertain if they could sign up by email only.
4. Users generalized what “offers” they thought they would receive by email.
2. Users had difficulty finding promotional offers by email.
3. Users were uncertain if they could sign up by email only.
4. Users generalized what “offers” they thought they would receive by email.
Analysis: Task 3
Task 3:
Your pizza arrives, and it is terrible! Your driver was rude, and you are really upset about what just happened. You call the local store but get nowhere. You need to contact the corporate office. How would you do that?
Analysis:
All four participants could complete the task in less than four minutes with no errors. Below is a table illustrating the metrics of Task 3.
Task 3: Quantitative Data
Key findings for Task 3:
1. Users are expected to find the nomenclature “contact us” in the footer of the home page.
2. Users were not interested in filling out the feedback form.
3. The Snail Mail address is prominent, but users were not interested in writing a letter.
4. Users had difficulty finding a corporate phone number.
2. Users were not interested in filling out the feedback form.
3. The Snail Mail address is prominent, but users were not interested in writing a letter.
4. Users had difficulty finding a corporate phone number.
Qualitative Data Analysis
Post-testing sessions, I reviewed the recordings, deconstructed the data gathered during the sessions, and analyzed the findings through the use of an affinity map. I then summarized and aggregated the results into a matrix, which provided a synthesis of the data and applied a user success percentage.
Qualitative Data Matrix
Follow Up Questions and Product Recommendations
Because of the qualitative data gathered and the synthesis of the data via the research matrix, I could create a set of follow-up questions along with product recommendations to address the questions. I listed the follow-up questions and product recommendations below.
1. Is the corporate phone number easily accessible?
• Increase the visual contrast of the corporate phone number
• Place the number more prominently in the interface’s hierarchy
• Place the number more prominently in the interface’s hierarchy
2. Is the nomenclature of the website common/universal?
• Improve clarity of wording for common/universal fields
• Add a “Contact Us" link to the footer
• Add a “Contact Us" link to the footer
3. How are customers accessing product promotions?
• Improve the visual presence of promotional offers
• Have separate sign-up fields isolating email from text offers
• Have separate sign-up fields isolating email from text offers
4. Are customers confusing account sign-ups with email promotions?
• Place all sign-up and promotions into a single page
• Provide obvious distinctions/categories for sign-up offers
• Provide obvious distinctions/categories for sign-up offers
5. Is the frequency of upselling the product effective?
• Reduce upsell offers until the final checkout
• Do not upsell on every cart entry
• Do not upsell on every cart entry
6. Are there enough baseline pizzas offers with basic popular toppings?
• Place top baseline pizzas first in the hierarchy
• Move “Create Your Own” pizza lower in the hierarchy
• Move “Create Your Own” pizza lower in the hierarchy
7. Are users being given too many choices at once while selecting toppings?
• Reduce the cognitive load by minimizing choices
• Use progressive disclosure during the topping choices sequence
• Use progressive disclosure during the topping choices sequence
8. Do users understand how to start an order?
• Improve the home page visual hierarchy
• Enhance the visual clarity of the “Start Your Order” button
• Enhance the visual clarity of the “Start Your Order” button
Lessons Learned
This was my first project, moderating a Usability Test. The lesson learned from this project is I have great respect for anyone who can moderate a testing session artfully. Moderating a testing session, much like anything in life, takes practice. I feel that there is plenty of room for me to improve in the role of a moderator. That being said, the data I could gather because of the session was quantitive and invaluable to validating Papa John’s website as an effective tool for online ordering.
