The tools used by designers are studied by ergonomists and psychologist to scientifically prove the benefits bring by their use. This is the case for heuristic evaluation, usability testing, or card sorting.
Guerrilla testing has no scientific study to back it up, to my knowledge.
Two factors can explain why.
First, it’s a new tool.
Studies that back up other tools are 20 to 40 years old. Back in the day Ux designers didn’t exist or were rare.
People that conduct these studies were either developers, psychologist, or ergonomists that wanted to find how to create better software or analyze the mechanisms behind creativity.
Second, it might originate from marketing.
I didn’t find the origin of guerilla testing. It wasn’t created by a big name like Norman or Nielsen, it seems to have just popped up.
My theory is that having the first person available test an unfinished app is just a common practice for any designer. I like to have my girlfriend give me feedback, she always has great ideas and positive feedback motivates me to continue.
This common practice may have been given a name by the marketing department to give it some credibility.
Like the article used before said, “it’s great to convince stakeholders”.
If it is a creation from the marketing department, this would explain why no scientist studied it as it didn’t originate from scientific theories.
(and it would explain the badass name)
Guerrilla is still present in scientific papers but in the study methodology, not as the main subject.
One example can help us identify flaws in the use of guerrilla: researchers used guerilla testing to compare the Google website ranking to the perceived utility of each site about breast cancer.
In this study, guerrilla testing was used to select the participant and create a small sample. People found in the hallway were graduated or PhD students. They were asked to read websites about breast cancer and say how useful the websites were. The 200 first results of Google search were used.
The conclusion is that even a website in the low ranking of Google could be useful and even more useful than some of the first results.
This study can give us some ideas about what are the pro and cons of guerilla teasing.
People didn’t evaluate the website but its efficiency. Guerrilla testing could be used to evaluate how user perceive an app’s functionality performance.
Is the efficiency of web search perceived better on Google or Bing? Is it perceived easier to send a direct message to someone on Discord, Slack, or Messenger?
The hallway creates a bias. There might be a limit to the “random” factor. In this study, every tester was highly educated, the hallway was clearly inside a university. If the goal to have randomly selected people to test an interface is to get diversity, the places you go to recruit can break the random factor as similar people go to similar places.
New ways to use guerrilla testing
Guerrilla testing is a new way to analyze an interface and to get data to iterate. We can’t tell yet if it is better than classical user testing or heuristic analysis.
It is said to help detect major flaws, but as an expert aren’t you suited enough to detect them by yourself?
I think this tool has real potential but might not be used to its full extent.
It could be linked to an emotion-maps by getting the first impressions of anybody that open the app. Doing this could help Ui designers achieve a wow effect or get users in a good mood.
A user pleased by the look of an interface will be in a better state of mind to use it and more tolerant of imperfections encountered.
If the goal is to have a friction less workflow, having a lot of random people use your app without context can give you clues about what kind of users are at ease with your app and what kind isn’t.
This could even be linked to marketing by helping define the “ideal customer persona”.
Lastly, it could be used as preparatory work for bigger scale user testing. Maybe having first reactions from random people can help better define scenarios for classical user testing.
In the near future, scientists will conduct experiments to define the real value of guerrilla testing in a reproducible way, with data backing it up. Meanwhile, Ux designers will continue to use this method because it’s easy and cheap, but we must keep in mind that it can’t replace traditional tools.