At SurveyGizmo we believe that numbers have no meaning without understanding the story behind them. We are not alone! From the SurveyGizmo Benchmark Guide Survey, we found that open-ended questions are being used frequently in online surveys. Sixty-four percent report using text box questions and 52 percent report using essay style questions in their online surveys.1
Open-ended questions provide commentary that goes hand-in-hand with your closed-ended questions. We recommend that you use these question-types to get context for your closed-ended response questions. You can add them as an “other” response in single or multi-select questions, as a text box asking for short response or an essay question, requesting a longer response from your respondents.
Note, that open-ended question types can be highly fatiguing, so please use these only when most necessary, and do not make them required in your surveys.
Building these question types into your survey is the easy part. Analyzing the data is where the challenge begins; it also delivers the rewards! There are several tools that claim to analyze open-ended data, but this market researcher thinks it is best if done manually, using a bucketing technique.
Here are some steps to follow when you manually bucket your data:
1. Read every response. This may take quite a long time, but it is worth it! You will really get to know the data. As you go through the responses, you will start to see some trends. Be sure to mark some quotes that resonate with you.
2. Put each comment into one or multiple buckets. Next, develop buckets or categories for the different trends that you see in the responses. Each response should go into at least one bucket. Sometimes there are multiple ideas expressed in a comment and therefore it may belong in multiple buckets. This process is often referred to as “multi-coding”.
3. Review and compress your buckets. After all the items are bucketed, review each bucket and check to see if some buckets may be able to be combined, and if others may need to be split into two. By doing this you will be able to identify clear trends, issues and new ideas suggested by your respondents.
4. Focus on the meaning and the context of responses. Once all the responses are in buckets and coded, you still need to make the data useful. You’ll want to couple your open-ended responses with relevant close-ended responses to get actionable data. Maybe you will find an “other” response that is a clear winner in a single-select question, or you will find out new information about why your customers choose your product or service. Note, that when using open-ended data, we tend to stay away from using percentages and use the data as directional with relevant terms. If there are a small number of respondents, you can mention the actual number of respondents that provided a particular bucket of ideas.
5. Use those buckets and quotes. Once you have analyzed your open-ended data and identified the major trends, your next step is to write a summary of what you have found. This can be descriptive text incorporating comments and quotes directly from the respondents. You can also use quotes:
- On your cover page to make the report seem approachable and add appeal to the reader
- To open up a live presentation to engage the audience
- To highlight each section or chapter of you written report
- When building personas
- In your team or department mission statement
We hope you use these techniques to analyze your next set of results. In fact, it is not too late to go back through your last data set and add context to your numbers.
1 Source: SurveyGizmo Market Research Benchmark Guide: 2012, Comparative Analysis on Survey Metrics, Techniques and Trends, Do you use open-ended text questions in your survey? Please select all that apply. n=876 Total Sample
