Improving first-time user experiences for Amazon Alexa’s “headless” devices.
#usability-testing
#voice-ui
#cross-platform-ux
My Role
Project Team
Client
Timeline
Juliana L., Layal A. Lige Y.
Usability Researcher
10 Weeks
Amazon Alexa
The Problem
First-time setup and onboarding is one of the most critical moments for Amazon’s Alexa products, especially for screen-less (aka “headless”) Echo devices. With high expectations and excitement to interact with Alexa, any hiccups during the onboarding process can result in frustration, disappointment, and even an end to a user’s relationship with the Alexa product ecosystem.
The Scope
Aware that their device onboarding experiences are lengthy, the Amazon Alexa team has struggled to pinpoint exactly what aspects are causing unsuccessful user setup. Our team was approached with the request to conduct 1) an internal audit of the device onboarding process, and 2) usability testing of headless Echo devices.
The Outcome
Our 10-week study resulted in the development of a usability test kit, final research report, and actionable recommendations for the Amazon Alexa team to improve product onboarding experiences.
Stakeholder Engagement
Working with stakeholders to determine project direction.
Our team connected with the Alexa design organization to better understand their existing pain points and what they hope to uncover from their users’ interactions with their products. Working closely with a Lead Conversation Designer from the organization, we determined the largest areas of concern were:
#1
What causes first-time users to drop-off in the onboarding process?
#2
Why do users skip or reject steps during the initial setting selections and opt-ins?
Internal Audit & Research Questions
Understanding the mind of a headless device.
Focusing on the versatile 3rd-Generation Amazon Echo Dot, our study began with an internal audit to explore the “first-time user experience”, defined as the initial out-of-box experience (OOBE) and subsequent voice interaction with Alexa.
After gaining first-hand experience of the onboarding process, we developed an interaction map to better identify fail paths and areas of concern.
One major takeaway from our audit was that while the Echo device relies primarily on voice interactions, much of the onboarding actually occurs on the Alexa App.
#1
What causes first-time users to drop-off in the onboarding process?
Do first-time users encounter issues during the onboarding process? If so, what are those issues?
Will first-time users be able to complete setup and onboarding of their device without our assistance?
What do users like or dislike about the Echo device/Alexa during the first use?
Aligning with our stakeholder objectives, we translated our initial research questions into the following usability-oriented questions to further guide our methodology:
#2
Why do users skip or reject steps during the initial setting selections and opt-ins?
What opt-ins or additional steps, if any, will users accept to proceed with?
Will users attempt to explore Alexa’s other functionalities beyond basic commands introduced during onboarding?
How easily and successfully do users discover Alexa’s various functionalities?
Will users attempt to connect Alexa with other systems/applications during their first use?
Participant Recruitment
Defining the key characteristics and behaviors of target users.
Working with our stakeholders, we defined our primary target users as individuals who have either never used or owned an Amazon Echo device before but may have used one minimally in the past. We were also interested in gaining insights from secondary target users, and defined this subset of users as individuals who currently own an Amazon Echo device.
Utilizing an online screening questionnaire, we recruited a total of 8 participants: 4 primary (first-time) users and 4 secondary (existing) users, summarized by the following table:
Methodology
Developing user scenarios, identifying key tasks, and crafting a “natural” test environment.
Utilizing an in-person moderated, scenario-based, task-oriented approach, we observed how participants navigated the first-time user experience across three distinct scenarios:
1) Unboxing
Participants were asked to unbox the physical Echo device and power it on.
Methods & Rationale
2) Device Setup & Onboarding
Tasks included navigating to the Alexa app, pairing the device to WiFi, and setting up Voice ID.
To emulate real-life interactions, we transformed a University of Washington conference room to resemble an at-home living space. Participants experienced the usability test as if in their own homes, seated on a couch with the Echo Dot device on a coffee table. The room featured a mounted screen, shelving unit, and home-like decor for an authentic testing environment.
We utilized a 360° smart camera, microphone, and speaker paired with an overhead camera system to capture participants’ verbal and physical interactions across devices.
Each session included 1) a pre-test interview, 2) the usability test scenarios and tasks, and 3) a post-test interview & questionnaire.
1) The pre-test interview was intended to better contextualize participants’ behaviors, with open-ended questions expanding on their prior experience with Echo devices and other voice assistant technologies.
2) The usability tasks allowed us to directly observe participant behavior as they navigated between the Alexa App, the physical Echo Dot device, and their voice interactions. Participants were encouraged to follow a think aloud protocol detailing their thoughts and behaviors as they worked through each task.
3) The post-test interview allowed participants to reflect on their interaction experience and freely share their likes, dislikes, and desires with the Echo device & Alexa.
Through the post-test questionnaire, we were able to capture a standardized, quantitative assessment of participants’ interaction experience through the use of System Usability Scale (SUS) measurements. SUS scores were then calculated and compared to a benchmark score based on existing literature.
Test Environment & Equipment
3) Interaction & Exploration
Tasks included customizing Alexa’s voice, trying a new command, and integrating an external app.
Key Findings
5 focus areas to improve users’ onboarding experience with Amazon Alexa.
Leading the data analysis for my team, we created affinity maps and conducted thematic analyses to uncover 12 core usability issues summarized into 5 major focus areas for product improvement.
“There wasn’t a lot of instruction. It said to go to the ‘Help’ menu but there weren’t a lot of instructions on how, or the fact that it wasn’t connected was why it wasn’t able to be set up.” -P7
“The menu option is complicated with multiple layers to it. I don't think it's a straightforward process.” -P3
"Where am I now?... I don't know [where I am in the app]." -P6
We then utilized a 4-point severity scale to assign a priority level for each focus area, ranging from 1 (cosmetic, aesthetic fix only if time allows) to 4 (catastrophic, immediate fix necessary).
Focus Area 1: Lack of Information & Guidance
Most users expressed confusion and a lack of guidance during the onboarding process. Users had to figure out how to work around issues regarding setup and onboarding on their own.
3 out of 4 first-time participants did not find help information under the in-app “Help” section, despite being instructed by Alexa to do so.
6 out of 8 total participants explicitly made guesses as to what the Echo device’s ring light colors meant.
4 out of 8 total participants were unsure which port and button would even power on the device.
Focus Area 2: Confusing App IA
Severity: Level 3 - Major
All users, regardless of prior use experience, expressed some difficulty navigating through the Alexa app. Users stated the app's information architecture was overwhelming, confusing, and not intuitive to use.
5 out of 8 total participants expressed feeling overwhelmed or disoriented on the app.
4 out of 8 total participants expressed it was difficult to find specific information they wanted in the app.
Focus Area 3: Inconsistent Onboarding Flow
3 out of 8 total participants never received the in-app popup to pair their Echo device; of the remaining 5 participants, the popup appeared at different stages of the setup experience.
3 out of 4 first-time participants and 1 out of 4 existing-user participants were caught in the “Voice ID” loop and were unable to continue in the onboarding process without moderator intervention.
Severity: Level 3 - Major
Severity: Level 4 - Catastrophic
For first-time users especially, inconsistencies in the onboarding flow lead to high frustration and even complete hindrance from progressing further.
“Okay so my screen says it’s still preparing, but my Echo gave me a noise and a green light, so I feel like it should be almost done. But since there’s no update on the screen, I’m not totally sure...” -P1
“Voice ID feels like an important step so I’m going to keep trying...” -P2
Focus Area 4: Mismatched Cross-Device Experience
Severity: Level 3 - Major
8 out of 8 total participants expressed wanting more feedback and consistency in the cross-device experience.
4 out of 8 total participants expressed wanting more verbal instruction when being directed to use the Alexa app.
Severity: Level 2 - Minor (more testing recommended)
3 out of 4 existing-user participants asked a new question instead of giving a command during the “Try a new command” discovery task.
4 out of 4 first-time participants incorrectly attempted to customize their device’s voice thorough the “Communicate” tab of the Alexa app.
0 out of 8 total participants navigated to the “Things to Try” section of the Alexa app during the discovery task.
All users, regardless of prior use experience, faced issues due to poor integration between the physical Echo device and the in-app Alexa experience.
“It would be nice if the app had something to go along with what Alexa was saying during onboarding.” -P5
“I'm assuming [Alexa’s voice] will change because there's no save button, but I would've liked Alexa to speak to me in that voice now.” -P1
Focus Area 5: Complicated Information at the Expense of Discoverability
Based on findings within other focus areas, unclear product education, overwhelming information, and confusing app IA may be the reason why most users struggle with discovering new features and capabilities of their Echo devices.
“I think it was a little overwhelming just because when Alexa got set up, she was already prompting me with like things she could do” -P4
“If users do not actively look up [what Alexa can do], they just wouldn’t know.” -P3
Utilizing a benchmark average score of 68 based on existing literature, our participants scored Amazon Echo and Alexa as "below average" in terms of system usability. Our study garnered a SUS score of 54.7, falling around the 20th percentile (i.e. scoring below approximately 80% of other systems measured within the existing literature), further validating the need for product improvement to our stakeholders.
System Usability Score
Validating a need for product improvement.
Product Recommendations
Focus Area
Providing actionable recommendations to address usability issues.
From the findings outlined above, our team developed the following recommendations for improving the first time user experience for Alexa products:
(1) Lack of Information & Guidance
(4) Mismatched Cross-Device Experience
(5) Complicated Information at the Expense of Discoverability
(2) Confusing App IA
(5) Complicated Information at the Expense of Discoverability
Key Recommendations
Provide users with greater product education early on during the onboarding process so that they have an established language with Alexa products and can better troubleshoot errors as needed.
This should be supported by seamless integration between the Alexa device and Alexa App, as users expect their interactions and system feedback to match across devices.
Restructure the Alexa App’s information architecture to better meet user expectations and provide a more intuitive interaction experience. Simplifying the IA can allow users to more efficiently find what they need, in addition to creating a friendlier path for users to explore and get to know more of Alexa’s capabilities.
Recommended tests: card sorting, tree test, user interview to better understand user expectations and conceptualization of app.
(3) Inconsistent Onboarding Flow
For first-time users, employ a consistent onboarding interaction flow that prioritizes device-app pairing and WiFi connection before any other customizations, opt-ins, or setting selections. This should be communicated by both the physical device and in-app instruction.
For existing or returning users, the user path to pair a new device to their existing ecosystem should be made more salient in the Alexa App.
Reflection
Outcome & Impact
Limitations
Magic Wand
Our team presented and delivered a final research report containing full study documentation, key findings and recommendations, and a usability test kit to our project stakeholders. Overall, our work was received positively. Our stakeholders appreciated the insightfulness of our findings, and found our data helpful in drawing their attention to the usability concerns of the Alexa App. Having collaborated closely with the Lead Conversation Designer at Amazon Alexa throughout the duration of our project, we passed our deliverables onto them to disseminate within the greater Alexa design organization in consideration for future product improvements.
[Recruitment] Due to time and resource constraints, we had to ease our initial participant requirements for device ownership, exposure, and interaction history in order to qualify enough primary and secondary participants for our study. Additional constraints resulted in limited diversity in age, gender, education, experience with voice interaction technology, and confidence in technology learning among our participants.
[Data Measurement] While we had initially planned to measure more quantitative data during our usability study (i.e. task success rate, task failure rate, etc.), our pilot tests revealed large variation in what participants encountered during the setup and onboarding process. In addition to the wide range of interactions and features that participants could encounter during our discovery task, we decided to shift our attention to collecting and analyzing qualitative data given the subjective nature of participants' experiences. This change is largely reflected in our post-test interviews, as we opted for open-ended, free-response answers to questions that are traditionally measured using a Likert-scale rating.
If given a magic wand to conduct this study again with unlimited resources, I believe it would be incredibly valuable and insightful to test with participants of varying age groups, technology confidence/expertise, and native languages. As a smart assistant boasting a broad range of capabilities, I believe further studies exploring how Alexa products can better adapt to different user populations hold the potential to set it apart from other competitor products.
Should the onboarding flow be improved to be more consistent and integrated based on our aforementioned recommendations, I believe it would be valuable to repeat usability testing with headless Amazon Echo devices. With greater control and predictability over the onboarding flow, future usability tests can place equal measure on qualitative and quantitative data, with an emphasis on task success vs. failure rate, task completion time, and ease of use ratings.
Regarding the discoverability of Alexa’s features and capabilities, I would be interested in conducting a longitudinal study to explore how user learning may or may not develop over time. Our initial study findings could suggest that existing users may already be satisfied and/or content with how they have adapted Alexa into their routines; discoverability may not be high on their list of user needs or goals with Alexa. Thus, future studies could test to see the relationship between what commands users learn during onboarding and what they end up sticking with in the long term.