How to: Ranking in Recollective

Recollective is a highly customizable platform, which means there is almost always more than one way to accomplish your goals when setting up your research project. In this article, we’ll cover different ranking method options within Recollective and provide considerations to help you decide which one would work best in different use case scenarios.

Recollective is a highly customizable platform, which means there is almost always more than one way to accomplish your goals when setting up your research project. While there is rarely a right way of doing any particular thing, there is almost always a best way to set things up so that they meet specific research goals and data output requirements.

This holds true when it comes to collecting ranking data using Recollective. There are a few different ways that you can approach ranking on our platform and each method will produce a slightly different output to leverage when doing your analysis.

In this article, we will cover two different Recollective Task types that you can use for ranking and provide considerations to help you decide which one would work best in different use case scenarios.

More specifically, you will learn:

  • Difference between rating and ranking
  • Sort and Rank method
  • Grid method
  • How to choose the right method

Ranking vs. Rating

First, it's important to highlight the distinction between rating and ranking. A rating question asks participants to assess a list of items/ attributes against a predefined scale. Whereas, a ranking question asks participants to assess a series of items/ attributes and place them in order of importance or preference relative to one another.

For instance, here is an example of a rating vs. ranking question:

  • Rating: Rate each of the following items on a scale from 1 to 5, where 1 is “Not at all satisfactory” and 5 is “Very satisfactory".
  • Ranking: Place the following items in order of preference from the most to least satisfactory.

Although this article focuses specifically on ranking methods, we will follow up with a related blog on ratings in the near future!

Ranking Method 1: Sort and Rank

When it comes to ranking, the first option you could leverage in Recollective is the Sort & Rank Task. Sort & Rank is a Task type within Recollective that enables cards to be sorted into groups and then ranked within those groups, if desired. When designing a Sort & Rank question, the researcher is able to predefine cards based on the items or attributes that they are interested in having participants evaluate. They are then able to define the groups, or categories, that the participants will sort the cards into. Finally, the researcher can also enable the ranking feature which will prompt participants to rank the cards within each group based on labels defined by the researcher.

Within the Sort & Rank Task type you have the ability to create a single group ranking option for your participants or have them rank across groups. Let's explore what each of those options looks like in more detail…

Single Group Ranking

If you are looking to get a single ranking for all of your items or attributes the most intuitive way to do this is by leveraging the Sort & Rank Task type and creating a single Group. By taking this approach, you are able to get participants to place all of the Cards, or items, into a single group and then rank them against one another in order of their preference or importance. Below you will find an image that highlights a Single Ranking using the Sort and Rank Task in Recollective.

In this example, participants are instructed to move each of "Fruit" Cards into the "Fruit Ranking" Group and then reorder the items by dragging them up and down in order of preference from the "Most Preferred" to "Least Preferred".

The output of this ranking method gives you two key data sets:

1. Average ranking for each card across all participants:

2. Individual Ranking of each Card by participant:

Multi-Group Ranking

Alternatively, you could use that same Sort and Rank Task type but with multiple groups instead of one. When taking this approach, you would create one group for each ranking position and request that participants add exactly one card into each group:

In this example, participants are instructed to simply move each of "Fruit" Cards into the appropriate group based on their preference and are limited to placing only one card per group.

The output for this set up provides:

1. A distribution chart of card placement across groups:

2. An associated data table of how many times each card was placed into each ranking position:

3. And a table of individual entries per participant, though in this approach, instead of looking at the “Card position” column, you’ll look at the group name:

Of course, these two methods are specifically designed to achieve a true, simple ranking. The Sort and Rank Task type, however, is able to be configured for more advanced use cases that can even combine rating and ranking using a combination of multiple groups and rankings within groups. To learn more about these other options, see our Spotlight: Sort and Rank Task article.

Ranking Method 2: Grid Task Type

Another option within Recollective would be to use the Grid Task. The Grid Task allows you to configure a matrix of rows and columns where participants make a selection for each row based on the columns provided.

To set up a ranking question in a Grid Task, you’ll want to use the columns as your ranking levels and the rows for your items:

In this example, participants will need to select one column for each row.

The output for rankings in a Grid Task is a similar but simplified version of the output from the Sort and Rank: Multiple Groups option. It includes:

1. A distribution chart of placement across columns:

2. An improved, simplified data table (but without the “Rank” column from the Sort and Rank method):

3. And a table of individual responses:

Choosing your Method

While there are many elements to consider when deciding how to set up your ranking question, we suggest that you focus your attention on three key elements: the input, output and overall participant experience.

Input:

To start, you will need to determine how many items or attributes you need to be evaluated. Using Sort and Rank (with multi-groups) or a Grid work very well for a smaller list of items. If there is a larger group of items to rank, maintaining the 1 for 1 match of items to groups/columns can end up making the final task page look a little bit overwhelming. In those cases, you would be better served by having participants rank within the groups themselves.

Additionally, you will need to determine if just having the ranking data will be sufficient. If you are looking to gain additional qualitative insight on the ranking responses, you will want to select a Task type that also allows the collection of open-ended responses. While both Task types have the ability to add an Additional Commentary box at the bottom of the page, Grid Tasks have the additional benefit of being able to request a comment per row. Meaning, the Grid Task will allow you to capture open-ended responses on each selection or ranking and link those details back to the selections themselves for easy analysis.

Output:

Similarly, you will also want to consider how the data will be displayed on the backend. Using Sort and Rank (with one group) might be a better option for those intending to do more external data manipulation since you get the individual response data in numerical form, and you have an average ranking table. Whereas, Sort and Rank (with multiple groups) and a Grid have a more straightforward data output that is easy to analyze within the software—the Grid Task, of course, being the most straightforward of the two options.

Participant Experience:

And finally, the last thing you will want to consider is the overall participant experience when responding to the ranking questions. Sort and Rank Tasks typically come across as more fun, interactive, and innovative, while the Grid Tasks can feel more like a traditional survey question. That said, there are plenty of customization options you can leverage to make the Grid Task more fun and visually appealing—for one thing, it’s possible to add photos into the rows and columns of Grid Tasks; there are also text formatting options for adding colours and changing fonts to add to the overall appearance. No matter what option you choose, we always recommend previewing as a participant to ensure a seamless, engaging experience.

And at the end of the day, the choice is up to you! Each ranking option presents its own unique benefits, but they also share many similarities. So regardless of the option you choose, there's no wrong way of doing it. It's all about knowing the options available to you and picking the one that best serves your needs. Don't forget, too, that you can always export your data and manipulate it externally to get results that are in the exact format you're looking for.

As always, feel free to reach out to our team if you have any questions or need help determining which method could be right for you!

Lara Aippersbach
Senior Training and Policy Specialist

Let's research happy together