NPS & on-site survey best practices
This is a chat with a senior researcher from another team on some best practices for survey questions on-site, esp for NPS. I found it super helpful so am writing it down here. (You won’t find NPS 101 in this article, such as the definition, purpose of NPS, etc, but more practical methods of crafting the NPS setup.)
NPS overall:
NPS is to measure the overall perception of your brand/company. It captures the longer-term relationship of customers with the company (e.g Quarterly, since it is not going to fluctuate much from time to time). In that case, it is best suited for being in a survey and sent to users via email.
NPS analysis:
Use the NPS results to compare to other companies, different sectors, industries to benchmark data.
Who should see the NPS survey or the on-site survey?
The more users the better. NPS can be sent to the overall population of your audience, or get representative samples to project the whole population. And if the NPS survey is situated on the website, it might as well be on the homepage and pops up right away when the users land on the page. It is not ideal to put the survey on an individual specific product/feature page, because you only get a particular segment of your audience to answer, and thus the answers might be biased. For a specific page survey, you can decide by page views of that page in a week/month cadence. Depending on your survey goal, you can select first-time users or users who come to this page with very high frequency, etc.
Registered users v.s unregistered users: onsite survey should be sent to unregistered users as well because you don’t have those users’ emails to send them surveys via emails. For registered users, you can send them both from emails or on-site. And you can segment unregistered users and registered users and compare the results.
NPS on a specific page:
If you want to test out users’ perception of a specific page, it is not NPS, but rather a different type of survey questions. Some use cases for page/feature-specific surveys on site are: whenever there’s a launch and something is noticeably different. You want the users to compare the new feature from the old pages.
What types of survey questions can you ask for on-site surveys?
One biggest consideration here is not to bombard the users with lots of survey questions. While in an email survey you can have a longer list of questions (usually because you send out at a lower frequency), for on-site questions each time you should only pop up one question (and potentially with a follow-up question). Some types of questions are: “Happy or not” question: a 3 point satisfaction scale (happy, mutual, negative) or smiley face, sad face, etc. When analyzing those survey question results, you can still use the NPS analysis methodology, e.g: People who choose “satisfied” are promoters. Don’t get a specific feature in the survey to ask the users. Remember, the more granular the question is, the more cognitive load will be added to the users.
How to decide to send out survey questions in email v.s on-site?
NPS is best suited to send out in email because you strip away unnecessary elements from the page to distract the users and force them to think in an abstract fashion about your brand/product in general. For other types of questions, if the users need some context and background to be able to answer, then it’s appropriate to put them on the website since it’s contextualized. Email survey is not good for testing out features because users are not interacting with it at the moment, but rather after the fact.
When should the on-site survey pop up?
To put it simply, it depends. Some features don’t need the users to do complex interaction or the effect is instant, so the survey can pop up sooner or instantly (e.g a change of button text, color, etc). On the other hand, some features require a lot of understanding upfront, for example, a new widget on the page. And for these pages, the pop-up should give users enough time to digest and interact before it pops up. Regarding #minutes/second, depending on the content, might need to test out and iterate to find the best timing.
Where should the on-site survey pop up?
Persistent footer, anchored to the bottom. This way it is not popping up to the center of the screen to disrupt the users’ workflow and still retain some visibility. The other advantage is, usually users tend to close pop-ups right up if it shows up in the center of the screen, so having it sticky to the bottom minimizes the real estate it uses.
What if I want to collect more data from the on-site survey?
As mentioned above, it is not ideal to add cognitive load to the users by asking them multiple questions at a time. But there are several ways to work around it if you indeed need to collect more data. Usually, after submitting a 3 point scale question (e.g “how satisfied are you with our product — “satisfied”, “neutral”, “not satisfied”), you can prompt a follow-up question: can you provide additional feedback? Comments or suggestions? Can you tell us more about why you made the selection? etc. The follow-up question should be optional though. The other way is to redirect the users to Qualtrics (or a third-party survey tool of your choice) and ask them to fill the survey there. The first question data will be sent to Qualtrics as well (might need some js scripts though).
Dismiss of survey on-site:
We should always keep the users on top of our minds. And thus we shouldn’t bother them too much/frequently ask for feedback. The on-site survey should always have a close button for the users to dismiss it easily. (instead of having questions centered and mandatory to answer to proceed) And once they dismiss it, the survey should not come back in a short amount of time (e.g in several hours, or days). Try to minimize the distraction of the survey on the customers from getting their jobs done (what they come to the website to do).
Should you modify the NPS score from 0–10?
You should not. Use 0–10 because it’s standard across industries so it will help you compare and benchmark the data with other companies/industries.
