Home > User Experience > Evaluating the Usability of Apple CarPlay

Evaluating the Usability of Apple CarPlay

Evaluating the Usability of Apple CarPlay


Launched in 2016, Apple CarPlay is an in-vehicle infotainment system that incorporates iPhone’s features in a car’s built-in display providing a smarter and safer way for drivers to use these features on the move. With this study, we wanted to understand how new and experienced users make use of the interface in conjunction with Siri and what are the usability issues they face while doing so.

We created a Recruitment Questionnaire and sent out the survey to different student-led Facebook and Whatsapp groups. We wanted to make sure the user:

  • Is at least 18 years old,
  • Is an iPhone user, and
  • has a valid driving license

In total, we recruited 5 users ( 3 new — U1, U2, U3 — and 2 experienced — U4, U5 — ) who met the eligibility criteria.

Background research helped us identify Maps, Phone, Music, and Messaging as the four most commonly used functions on car infotainment systems. Therefore, we decided to test out these particular functions for our usability study.

Duration

30–40 minutes per session

Approach

We used the Think-Aloud technique along with direct observation to evaluate the usability of Apple CarPlay. In addition, we video-recorded the whole session to further analyze any issues that we might have missed out on during the live test.

Equipment

Volkswagen Jetta (2019) with Apple CarPlay installed

Methodology

  1. Introduction and Verbal Consent: We started out our tests by introducing the participants to the study and taking verbal consent to audio and video record the whole test.
  2. Pre-Test Interview: This was followed by a pre-test interview where we asked users questions about their past experiences with infotainment systems, the features they made use of and the problems they faced while doing so.
  3. Scenario-based Usability Test: This was followed by our scenario-based usability test where we asked the users to perform a set of tasks based on a given scenario. After each task, we asked the participant to self-measure the difficulty of a task on a 5-point Likert scale ranging from very easy to very difficult.
  4. Post-Test Questionnaire (SUS): After all the tasks were performed, we asked the users for final feedback and asked them to fill out 10 system usability related questions based on the SUS Questionnaire.

Scenarios and Tasks

We formed a total of 10 tasks — 2 to 3 tasks for each of the functions we were testing i.e Maps, Phone, Messaging, and Music. In order to make the test more realistic and to better engage participants, we made 5 scenarios in a continuous narration to incorporate these tasks.

The metrics used to analyze the test included both qualitative and quantitative measures.

After collecting the data we analyzed it using the following techniques:

Task Time & Completion Analysis

Time taken for each task and success/failure of the task performed was collected during the test sessions.

The results indicated Resetting Destination as the most time-consuming task leading to most task failures. Connecting with the Carplay, Setting up the navigation, Checking messages and Changing Songs were identified as other tasks of concern.

Task Difficulty Levels

After each task, we asked the participant to self-measure the ease of task on a 5-point Likert scale ranging from very easy to very difficult.

Connecting with the Carplay, Checking and replying messages as well as Resetting the destination were considered to be challenging. At least 40% of the participants found these tasks difficult or very difficult to complete.

SUS Score

After all the tasks were performed, we asked the users to fill out 10 system usability related questions based on the SUS Questionnaire. We calculated the SUS scores for each test participant and differentiated these scores based on users’ past experience with Carplay.

On average, new users gave a relatively low score of 65 to the Carplay system when compared to their experienced counterparts. It indicates a steep learning curve for new users and calls for a better onboarding process.

Affinity Analysis

We formalized the raw notes obtained from the usability tests, rewatched the sessions and created an affinity diagram to discover common insights and high-level findings.

A Snapshot of the affinity analysis diagram — created on Miro

In order to better categorize the issues we discovered, they were sorted into two categories:

  1. Issues of high severity
  2. Issues of low severity

Issues of High Severity

  1. Connecting phones to Apple Carplay is difficult for a new user

Apple Carplay requires users to connect their phone via USB and there is guiding text on the landing page prompting users to connect via USB once they turn on Apple Carplay.

Some of the users failed to connect their phones to Apple Carplay. They did not notice the text on the screen, and almost instantly started to try and connect via Bluetooth, spending a lot of time exploring different pages on the system and buttons on the steering wheel trying to locate the ‘Bluetooth’ button.

Use animations to make the ‘Connect via USB’ guidance more noticeable

2. Checking messages can be hard and distracting

40% of the participants found it very difficult to check the message while driving, mostly because in Apple Carplay, the text message is not displayed on the interface and it will only be read by Siri. This is different from most of the users’ conceptual model, who expected to read the text on the interface.

In addition, there is no way for users to go back to already read messages, which is also the reason why one user failed the task: the user didn’t pay attention to what Siri said when Siri was reading the message.

Due to these constraints, participants have to concentrate when Siri is reading the message to make sure they would not miss anything. Participants found this to be quite distracting.

Having both Siri reading the message and an option for users to see the content on the interface

Allow users to read the already read messages

3. Resetting destination while driving can be inconvenient

Resetting the destination was found to be the most time-consuming task for participants. Only one user was able to finish this task within 10 seconds because she used Apple Maps — which supports Siri.

However, for users who are used to navigation systems like Google Maps and Waze, they could not use voice input because the apps are not incorporated with Siri. This led them to input the address manually, which cost them considerable time.

Furthermore, as users cannot read already read messages they could not go back and copy-paste the address they received as a message, therefore two of the participants were not able to recall the address, which led to the failure of the task.

Incorporate Siri with other map apps

Let Siri automatically detect if a message contains an address and set the destination for users

4. Unintuitive interface design

Carplay’s interface provides a menu on the left showing a list of recently used apps. The one on the top is the app that is currently used by the user. It is an unclickable icon. The other two apps below the icon are recently used apps. However, during the tests, most of the users didn’t understand this design. They could not tell the difference between the icon and clickable buttons. Some participants tried to tap on the icon however nothing happened, which confused them.

The home button is located on the bottom left-hand corner. Two users had a hard time figuring out how to go back to the home screen because they did not notice the home button.

Clearly differentiate between clickable buttons and icons

Make the home button more prominent (such as in the center like many apple devices)

Issues of Low Severity

  1. Users want to get more control over Carplay notifications

Interacting with Carplay notifications led to a few unexpected usability issues.

One of the participants tried to swipe the notification to remove it but wasn’t able to do so as the notification system does not allow swiping off notifications.

2 of the participants were confused as the notifications disappeared automatically after a few seconds and they had to navigate to the desired app to see the notification again. This led to an increase in steps and time taken to complete a task.

Control over removing and viewing notifications

2. Need to make Siri more friendly and configurable for new users

We encountered a lot of issues surrounding Siri and how it should be made more friendly for new users.

2 of the 3 new users didn’t know with which apps Siri is compatible with.

In addition, all of the new participants didn’t know until the very end of the test on how to initiate voice control. Participants also talked about Siri not understanding foreign names and foreign accents.

Integrate Siri with other apps

More prominent Voice Control button on the Steering Wheel/Interface

Letting the users know if the app is Siri compatible or not

3. It takes time to find a particular song using Carplay

3 of the 5 participants took more than 40 seconds to find a particular song using Carplay. This was because most of the users used the Carplay interface to scroll through the list of songs — instead of using Siri — which was particularly time-consuming. This was again caused because participants were either using apps that were incompatible with Siri (such as Spotify) or just didn’t know how to initiate Voice Control.

4. Sometimes hardware can also be a barrier for users to complete tasks

Although this problem is not specific to Apple Carplay interface, it led to a lot of confusion among some of our participants. 2 of the 5 participants took more than 30 seconds to adjust the volume. This was because the users weren’t able to tell the difference between the two similar-looking adjusters as seen in the image above. It wasn’t clear which one of the two is a volume adjuster and which one is not. This was a clear problem with Volkswagen Jetta’s hardware.

Provide guidelines about how to better design hardware to incorporate Carplay

  • The driving experience was simulated and we did not make the users drive the car due to safety concerns. In the future, we can make the participant actually drive the car while using Carplay — under controlled conditions — and get even more accurate and relevant insights.
  • Our tasks considered typical/ideal driving situations. Experiences may vary because of Road condition, Traffic, Weather, and Urgency. This problem can also be tackled in the future by making the participant drive the car on real roads without simulating the driving experience.
  • Our study only targeted young, college-educated populations who are adept at using technology and newer interfaces (20–30-year-olds). Future studies could focus on a much more diverse population such as aging adults and/or disabled people. This would help us make Carplay more accessible to a wider range of populations.



Source link