How to: Rating in Recollective

Written by Lara Aippersbach • Posted on September 16, 2021
Recollective How to Rating 5 Star Wooden Block Blog Cover

Back in May, we published a blog post called How to: Ranking in Recollective. Within that article, we provided an overview of the different options researchers have to set up a true ranking question within Recollective. We also provided guidance on how to select the best option for your study based on your research goals, target audience and data output needs. In this follow-up article, we will address some of those same topics and considerations but for rating questions.

As with rankings, there are a few different ways that you can approach a rating in Recollective and each method will produce a slightly different output to leverage when doing your analysis. In this article, we will cover three different Recollective Task types that you can use for rating and provide considerations to help you decide which one would work best in different use case scenarios.

More specifically, you will learn:

  • Difference between rating and ranking
  • Poll method
  • Grid method
  • Fill the Blanks method
  • How to choose the right method

Rating vs. Ranking

Similar to the last article, we'll begin by highlighting the distinction between a rating and a ranking. As you may recall, a rating question asks participants to assess a list of items/ attributes against a predefined scale. In contrast, a ranking question asks participants to assess a series of items/ attributes and place them in order of importance or preference relative to one another.

For instance, here is an example of a rating vs. ranking question:

  • Rating: Rate each of the following items on a scale from 1 to 5, where 1 is “Not at all satisfactory” and 5 is “Very satisfactory"
  • Ranking: Place the following items in order of preference from the most to least satisfactory

Let’s take a look at the tools we have in Recollective that can be used for setting up a rating. Specifically, the Task types that can be leveraged for a rating question include:

  • The Poll Task
  • The Fill the Blanks Task
  • The Grid Task

Poll Task

Poll Tasks are exactly what you’d expect from something thusly named; they allow you to ask a question and configure a series of single or multi-select options for participants to respond. The different use cases for a Poll Task are endless and one of those use cases is for rating. If you have a single item to rate (say, apples) and a single scale on which to rate it (say, deliciousness), you can easily set this up as a poll by setting each point on your scale as an individual poll choice. This would look something like this:

Poll Task Rating Apple Research Example

The key advantage to this setup is in the output: the Poll Task has one of the simplest, easy to understand and easy to analyze output of all the Task types available in Recollective. To start, you get a nice clean pie chart displaying the frequency of each poll choice:

Poll Task Output Pie Chart Format

You’ll also get an associated summary data table:

Poll Output Summary Chart

And, of course, the full, detailed data set of responses from participants:

Detailed Data Set All Participants

Easy to set up, easy to answer and easy to analyze! If your rating question can be made to fit into a Poll Task, it is a very good option that will help keep things as simple as possible for everyone involved. It is also worth noting that you can capture additional open-ended commentary within Recollective's Poll Task so that participants can explain their rating.

Grid Task

You may recognize this Task type from our Rankings in Recollective article, and if so, you can probably already imagine how it can be leveraged for ratings as well. To configure the Grid Task as a rating, you’ll want to use the columns for your rating scale and the rows to define the items to be rated. This works great for when you have multiple items (say, apples, bananas, and pears) to rate on a single scale (again, deliciousness):

Grid Task Single Card Scale

But this can also work if you have a single item (apples) to rate against different attributes (texture, favour, versatility, overall deliciousness) as long as the scales for each attribute have the same number of levels:

Rating Chart Fruit Preferences

While the output for this is a little more complicated than a poll, it is still relatively straightforward. Instead of a pie chart, you’ll get a distribution chart, which will show how many times a column was selected for each row.

Distribution C Hart Output Data Grid Task

It will also display two summary data tables - one for the columns that looks exactly like the one from the Poll Task and one for the rows that reflects the distribution chart:

Summary Chart from Grid Task

And, again, a response data table that breaks down the individual responses, participant by participant, row by row:

Individual Participants Summary

The Grid Task provides you with two options if you are looking to capture additional commentary to your rating. Similar to the poll, you have the option to add an additional text box to the end of the Task for a single follow up question. In addition, you can enable a function called “comment per row”, which reveals a text box associated with each row of the grid so that participants can leave an explanation for each individual rating.

Fill the Blanks Task

The third rating option is to use the Fill the Blanks Task (which is one of our most unique and versatile Task types!). It allows you to ask a series of questions on a single page using different types of “Blanks” that allow for different response types. The Blank options include Text, Photo, Poll, Scale and Allocation Scale. For the purposes of setting up a rating, we’re going to look at the Poll Blank and Scale Blank options.

Of the two, the Poll Blank is the simplest in that, aside from how it appears to participants (pictured below), it is identical to the Poll Task. The only difference is that it appears as a drop-down selection menu rather than radio buttons; the output charts and tables are otherwise exactly the same.

Fill in the Blanks Task Question Example

The Scale Blank, similar to the Grid Task, leverages the classic "slider scale" look:

Scale Blank Task Example

The Scale Blank, also like the Grid Task, is best suited for having multiple items—such as the different types of fruit in the image above or the different attributes on which to rate a single fruit in the image below:

Scale Blank Task Type Fruit Attributes

The Scale Blank output is almost identical to the output from the Grid Task, with one small addition: it includes a second chart which displays the average rating.

Scale Blank Data Output Example

So if the outputs are almost identical, you may be wondering what the advantage of the Fill the Blanks Task is. The core advantage is that the Fill the Blanks Task allows you to have multiple Blanks in each iteration of the Task. So, while the Poll Task can only have one poll question, and the Grid Task can only have one scale question, the Fill the Blanks Task can have as many different polls and scales as you want, on top of all the other Blank options available.

A secondary reason for why you might want to use a Fill the Blanks Task instead of a Poll or Grid is for the distinct look and feel of the Poll and Scale Blanks: the "slider scale" look as opposed to the the "table" look of the Grid, and the drop-down menu of poll options rather having them all displayed on the page.

Selecting your setup method

When selecting your method, you will first want to consider how many items you want your participants to rate and if they should be rated on a single scale or against multiple attributes. For the most part, rating questions will typically fall into one of the following categories:

Rate a single item on a single scale (ie: rate apples in terms of deliciousness)

When rating a single item on a single scale, using a Poll Task or Poll Blank in a Fill the Blanks Task is almost universally the best option; whether you use one or the other really depends if you want to have a single question displayed on the page or if you have a lot of additional questions that you want displayed as well.

Rate multiple items on a single scale (ie: rate apples, bananas, and pears in terms of deliciousness) OR rate a single item against multiple attributes (ie: rate apples in terms of texture, flavour, and versatility)

When rating a single item against multiple attributes, or rating multiple items on a single scale, you’ll need to use either a Grid Task or a Scale Blank. Aside from your personal preference around the appearance of each option, as well as the inherent versatility of the Fill the Blanks Task, the only other consideration to guide your choice between Grid Task and Scale Blank is around whether or not you need the “comment per row” function. This is a feature that adds a text box to every row of the Grid Task so that participants can explain their rating; while the option to add text boxes into a Fill the Blanks Task can somewhat emulate this feature, this workaround does not have the benefit of ensuring that explanations are linked to the appropriate rating on the back end, which could make analysis more challenging. Consequently, the Grid Task is the better option for soliciting multiple ratings and associated explanations simultaneously.

Rate multiple items against multiple attributes (ie rate apples, bananas, and pears in terms of texture, flavour, and versatility)

When rating multiple items against multiple attributes, assuming you want all these scales to be on the same page, the only option for configuring this is using multiple Scale Blanks in a Fill the Blanks Task. With this setup, you would create a separate Scale Blank for each item OR each scale—essentially, you’d convert your question into a single item on multiple scales or multiple items on a single scale, and have these combined by putting them next to each other in the Fill the Blanks Task. If it’s not imperative that all the scales be on the same page, then you could consider using multiple Grid Tasks to achieve the same effect.

As with the ranking method options, there is no wrong way to set up a rating in Recollective; ultimately, the “right” way is the one that meets the needs that are a priority to you.

As always, feel free to reach out to our team if you have any questions or need help determining which method could be right for you!

Want to chat about this topic?

Get in touch!