Digital Transformation
cancel
Showing results for 
Search instead for 
Did you mean: 

I hereby challenge all survey software providers on the planet to do better than this...

mtfitzgerald

Blog graphic.PNG

NPS® surveys are by far the most common type of survey on the planet. The basic format has two questions:

  1. How likely are you to recommend [product or service] to a friend or professional colleague
  2. Why? 

People have become very good at getting scores for benchmark, relationship, product usage, and transactional surveys. However, answers to the second question are far more problematic. The lack of any decent software that provides usable results without human intervention is the main problem. Without it, people like me all over the planet tend to go through the list of responses until we (consciously or subconsciously) find things we agree with, saying “Ah ha! I knew it!” Because this is such an untrustworthy process, vendors tend to add lots of other questions, just to be sure. This has to stop. And it stops here.

Here is the challenge: The links below give you access to free software and a sample survey file. I hereby challenge all survey software providers on the planet to produce a better result on this benchmark file that you will see when you try our free software. Yes, you have to sign up to use it.

Links: The survey software is described and can be launched here. If you are not taken directly to the signup page when you try to launch it, please just click on the “Sign up” button at the top of the page.

The restaurant survey file is here. It is a fictitious survey of 303 people who have eaten at three different types of restaurants. Some very odd things seem to be happening at at least one of them. Will your software find out what the customer likes and concerns are, without any human intervention? Note that in addition to the NPS score and the “Why?” question, I have also included customer answers to the question “What would you like us to improve?”

 Notes

  • When you start, you must create an empty index by clicking on the appropriate button. Anything you upload will stay in the same index until you clear the index. If you do not clear the index, you file will still be there the next time you log on.
  • Click on the “i” button for information about the file format if you want to test with your own files.
  • Selecting the file to be uploaded and uploading it are two separate actions.
  • Click on any bar in the bar chart to see the actual survey responses.
  • If you want to play with the APIs the application uses, they are all free to use under the developer menu on www.havenondemand.com. 
  • I do have a separate project that produces richer results than what you will be able to see here, but I have not worked out to how to provide a free version. We also have further enhancements planned to the free software over the coming weeks.

Have fun.

Maurice

Maurice FitzGerald - Customer Experience and Strategy consultant
maurice.fitzgerald@hnetplus.ch
  • Big Data
0 Kudos
About the Author

mtfitzgerald

Maurice FitzGerald ecently retired from his position as VP of Customer Experience for HPE Software after a career in hardware, software and services at DEC, Compaq and HP.

Comments
mtfitzgerald

You need to sign up using the button at the top of the page before you can log in. Please let me know if that does not work for you.. You can email me at maurice.fitzgerald@hpe.com.

mtfitzgerald

Your first competitive comparison: Here is what Wordle (www.wordle.net) does with the answers to the "Why" question. I have not changed any of the default settings. This is not useful...

 wordle Why.png

mtfitzgerald

And here is sample output from what the Haven OnDemand application does. With this first all-in view, you can see what people are saying about why they do and don't like the restaurant, all mixed together. It becomes even more intersting when you select the Detractors.... Why not try it yourselves.

NPS1.PNG

mtfitzgerald

Another competitive view... Here is what you get with the Why answers using www.wordclouds.com, again with the default settings, and again useless, in my opinion.

wordcloud why.jpg

mtfitzgerald

WordyUP is qutie a bit more useful than Wordle and Wordclouds, though stil not all that useful because it works on single words, rather than phrases, and does not appear to use sentiment analysis. You can click on any word in the cloud to drill down further and to see actual survey responses.

 

WordyUP.PNG

mtfitzgerald

Forgot to mention that WordyUP worked find in Internet Explorer, but would nto work for me with Chrome.

mtfitzgerald

DataCracker is excellent software for analyzing numerical responses to surveys or those where you have to select among answers using a pull-down. It includes a word cloud chart, that suffers from the same defects of all other word clouds, in that it works on individual words, rather than phrases, and requires manual intervention to produce something even remotels useful.. It has a free version that limits analysis to 100 responses. Using my benchmark survey, here is what the word cloud chart looks like, without intervention. In short, not useful.

DataCracker.PNG

mtfitzgerald

Thanks to the many who have sent me messages directly about this. I am asking for permission to post from software companies who have sent me their results directly, rather than posting them here. In the meantime, we have added an additional algorithm to the software to ensure that every single survey response is used in some way. This is only on a dev instance at the moment. You will still be able to use a pull-down to select between only using the method that gave the bar chart above, or adding the additional topics that appear in the bar chart below. The catch, as you will see, is that not everything is really useful, while all the bar chart topics are useful above.

For the technically minded, the first version only includes topics where some sort of sentiment is expressed and detected. Sentiment is very common with the "Why" question, but not so common for the improvement suggestions. The new (not yet public) lets you compare both methods.

Restaurant why 2.PNG

mtfitzgerald

So... here is an update after loads of responses:

1. I may not have explained one aspect of the challenge well enough. There has to be no human intervention. Those of you who sent screen shots that looked good, but used software that requires human intervention to tag words or phrases in the responses need to challenge each other, but this is a different challenge.

2. The best I have seen so far is from Clarabridge. We disagree on which of our two solutions is the better, naturally.

3. Here is what we are imporoving with our nice free version after the main coder gets back from his honeymoon in New Zealand:

    • While the software should work in Mandarin and Russian, neither language can be saved in the required input format, which is a Microsoft .CSV file, with commas as separators. We will add XLS and XLSX as acceptable input file formats.
    • While the software works in French, Spanish and other languages that have accents, the UI does not, replacing accented characters by question marks.
    • We have done lots of testing on a new version that adds an additional algorithm to the sentiment-based one that is the core of the current version. The new version finds, for example, all twelve comments about ice cream in the ""Why" column of the sample survey about restaurants. We should be able to release it in early January.
    • There are three grammar-related issues that need some improvement. The first is the handling of contractions like "Let's" in English. We need to have that recognized as a single word, I think, even though it is really two. In any case, it should not come up a a concept. There are small issues in Spanish, French and Italian to do with the way the sentiment API handles some tenses. Finally, there is a tricky issue in the handling of pronouns. Take the following phrase: "The lasagne tasted great but it made my son sick." The sentiment analysis currently cannot work out what the "It" is that made my son sick. We need to see whether having pronouns make the algorithm go back to the prior noun gives a better result.

This is a fun topic so far. I have also had great feedback on the related post about using the same softwarwe to analyze employee surveys.

 

mtfitzgerald

A new version of the free NPS analysis sofrtware was released yestereday by my former employer. You can find it at https://netpromoter.havenapps.io/. It works much better and, in my view, breaks the tie with Clarabridge.

Notes:

- Russian and Chinese are on the language pull-down but .CSV is currently the only input format and does not support these languages.

- Accented characters do not display correctly.

- If you are using certain language versions of Excel, notably the French version, it puts semicolons in .CSV files instead of commas, and this will not work with the application. I use Excel in French and have to search and replace using Word.

mtfitzgerald

And here is a link to the restaurant survey file in .XLS format, which the applicaiton now accepts.

https://onedrive.live.com/redir?resid=1C8ADE8560CBD11B!59903&authkey=!AJMgMZXZmtVCaFg&ithint=file%2cxlsx

 

Labels
Events
28-30 November
Madrid, Spain
Discover 2017 Madrid
Join us for Hewlett Packard Enterprise Discover 2017 Madrid, taking place 28-30 November at the Feria de Madrid Convention Center
Read more
See posts for dates
Online
HPE Webinars - 2017
Find out about this year's live broadcasts and on-demand webinars.
Read more
View all