This is what Jim Perry, HT Morse Distinguished University Professor at the University of Minnesota, was interested in finding out. In this guest post, he describes how he used LabintheWild’s social intelligence test to see whether he could foresee group performance in his classes:
“We all are familiar with the concept of IQ. We each work in groups many times and we find group
performance to be highly variable. We might predict that group performance would be predictable
based on IQ of group members. In fact, that seems not to be the case. Recent work has shown that
there is a factor, known as c, that serves as a good predictor of a group’s ability to perform tasks. That
factor is not highly correlated with average or maximum IQ of group members. In contrast, it seems to
be well-reflected by the ability of the group members to read the emotions of other members. That
ability can be estimated by a simple, free, 10 minute test called Reading the mind in the eyes.”
“I was interested in knowing the degree to which student performance on various group tasks in classes could be predicted by student score on that “reading the mind” test. Therefore, I offered students in three classes a chance to participate. The approach was to correlate student score on that test with scores on things like draft and final papers. I invited 135 to participate by taking the test and sending me a screen shot of the result. Students who participated received a small amount of extra credit in the class, and we each thought this knowledge might be a benefit to a person in future group experiences (e.g., other classes) because a person might become more sensitive to other members’ emotions and therefore improve group performance.”
Here is what Professor Jim Perry found:
“Eighty four students chose to participate. Some people had multiple scores because they participated in different group efforts. Therefore, the final sample size was 281 observations. I used simple linear regression analysis to ask if a person’s score on the “reading the mind” test was a significant predictor of grade on group projects. The regression was not significant (p>0.05). In fact, r2 was 0.00.”
“In this study, the test score did not predict group performance. That might indeed be due to lack of such a relationship. Alternatively, it is possible that group performance was not adequately reflected in the grade assigned to the product of group performance. In academic settings, we use grades as a measure of performance but recognize that they are a weak measure. Other measures of group performance, such as completing an objective task might produce different results. Alternatively, it is possible that the variance among individuals as expressed by the test is not sufficiently large to capture the predictive capability within the population.”
“I am grateful to all the students who participated from my classes, and to Professor Katharina Reinecke.”
And we are grateful to Professor Jim Perry for testing this hypothesis! Thank you for this guest post.
LabintheWild turned two today! Thanks to all of you who have participated in our studies, sent us comments and suggestions, and have contributed to our experiments by spreading the word. We are excited to see how much LabintheWild has grown in its first two years!
During the summer, we launched an experiment to examine graph prediction tendencies amongst different cultures, particularly the contrast between Eastern and Western individuals. This study was based on “Culture, Change, and Prediction,” a 2001 paper published by Li-Jun Ji, Richard E. Nisbett and Yanjie Su. Participants in the study were shown a series of graphs, each with three points plotted and accompanying information, and they were then asked to predict the following two points on the path. According to the results in the original paper, Chinese students had a tendency to reverse the existing trend of the graph, while American students had a tendency to continue the trend, as shown in the figures below.
In our own study, we hoped to extend these results with other countries. After gathering data from about 600 participants, we focused on analyzing participants from the US, Romania, and several Asian countries.
We began by analyzing our data at the participant level. For each participant, we graphed both trials of each type of graph on one plot, to check for consistency amongst their responses.
After evaluating the American, Romanian, and Asian data sets against each other, we found that overall, there is a significant difference between the data points provided by US and Romanian participants. US Americans seem to envision graphs to follow a more predictable trend than Romanians do. However, we did not find a significant difference in the data when comparing the US to Asia or Asia to Romania.
When examining our existing data set in detail, we discovered a few compelling trends that could imply cultural differences in graph prediction trends. We are currently working on translating this study into different languages to be able to better simulate the original study by Ji et al.
How do you feel about Naver.com, a search engine shown in the screenshot below. Would you say it is badly designed?
Most Americans would feel alarmed at the sight of Naver.com. But many people prefer this design over the simplistic look of Google. In fact, Naver’s colorful design and high information density help make it the most popular search service in South Korea.
Research has repeatedly found that people’s visual preferences vary widely. This is just as true in the online world as it is in the offline world. Challenging the notion of a one-size-fits-all universal design, we find that some people like colorful, complex websites, while others prefer simple and clean designs, and still others prefer something in between.
You might argue that it doesn’t matter what a site looks like. But it does matter! If we find a website appealing, we are more likely to buy one of its products , and we perceive it as more trustworthy [4, 5]. A more fitting design can even increase our satisfaction with the site, and our work efficiency .
But who likes what? Answering this question is difficult. Our online preferences seem to depend on individual and demographic differences, such as personality, gender, or age [2, 3, 6, 7].
To find out how our visual preferences differ, we launched a study on LabintheWild. We asked people to rate a set of websites based on visual appeal. Over the course of a year, we collected approximately 2.4 million ratings from almost 40 thousand people around the world. We used this data to study how people from different countries, educational backgrounds, ages, and genders respond to a website’s colorfulness and visual complexity.
And here is what we found:
Website preferred by male participants more than by females
Website preferred by female participants more than by males
We also looked at how aesthetic preferences vary across different countries. Prior to this analysis, my stereotypical understanding was that people from Germany and Switzerland prefer simple and colorless websites, while people from China and other Asian countries prefer colorful and complex ones. But I had to correct my stereotypes, at least to some extent. For example, we found that people in Finland, Russia, and Poland liked websites with even fewer colors than the Germans and the Swiss. Similarly, people from Malaysia, Chile, and Macedonia preferred much more colorful sites than the Chinese, whose preferences actually don’t seem to be that different from those of people in the U.S.
You can have a look at the country overview below to see more details about each country’s preference for levels of colorfulness (red lines) and visual complexity (blue lines).
We also found that countries close to each other seem to often share similar preferences. For example, the neighboring countries Finland and Russia preferred the lowest visual complexity and colorfulness of all countries, followed by Sweden. Participants from Macedonia, Serbia, and Bosnia and Herzegovina—all countries that were part of former Yugoslavia—had very similar preferences for highly colorful websites. In addition, the Northern European countries in our dataset (Denmark, Switzerland, France, Germany, Sweden, Austria) preferred a lowerer colorfulness than Southern European countries, such as Italy, Spain, Greece, or Romania. Northern European countries also preferred a lower colorfulness than Asian countries, such as China, Singapore, and Malaysia. Hong Kong and Japan preferred a lower colorfulness than other Asian countries, but higher than Northern European countries. All of the English-speaking countries (Australia, New Zealand, Canada, United States, Ireland, and the United Kingdom) also preferred a higher colorfulness than Northern European countries.
These results could suggest that countries with a regular exchange of cultural values, perhaps due to migration, share similar website preferences. We are looking forward to investigating this further.
Thanks to all our participants for taking part in this study! We cannot thank you enough for your contribution to science and to making it a little less WEIRD!
If you are interested in more details about the study design and analysis, you can download the paper here (see also ).
1. Bloch, P. Seeking the Ideal Form: Product Design and Consumer Response. The Journal of Marketing 59, 3 (1995), 16–29.
2. Cyr, D., Head, M., and Larios, H. Colour Appeal in Website Design Within and Across Cultures: A Multi-method Evaluation. Int. Journal of Human-Computer Studies 68, 1-2 (2010), 1–21.
3. Hsiu-Feng, W. Picture Perfect: Girls’ and Boys’ Preferences Towards Visual Complexity in Children’s Websites. Computers in Human Behavior (2013).
4. Lindgaard, G., Dudek, C., Sen, D., Sumegi, L., and Noonan, P. An Exploration of Relations Between Visual Appeal, Trustworthiness and Perceived Usability of Homepages. ACM ToCHI 18, 1 (2011).
5. Lindgaard, G., Fernandes, G., Dudek, C., and Brown, J. Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression! Behaviour & Information Technology 25, 2 (2006), 115–126.
6. Moss, G., and Gunn, R. Gender Differences in Website Production and Preference Aesthetics: Preliminary Implications for ICT Education and Beyond. Behaviour & Information Technology 28, 5 (2000), 447–460.
7. Reinecke, K., and Bernstein, A. Improving Performance, Perceived Usability, and Aesthetics with Culturally Adaptive User Interfaces. ACM ToCHI 18, 2 (2011).
8. Reinecke, K. and Gajos, K., “Quantifying Visual Preferences Around the World”, Human Factors in Computing Systems (CHI), (2014).
LabintheWild has joined forces with TestMyBrain and GamesWithWords with the goal of Making Science Less WEIRD. Together, we are hosting a panel at the SXSWedu Conference and Festival in Austin, TX titled “Taking Research Into the Wild”. It is scheduled for Tuesday, March 4, at 9am. Come and see us there!
In the spring we conducted an experiment, in which participants clicked on several dozen dots (like the ones below) and at the end of the experiment our system made a prediction of each participant’s age based on their performance.
We were overwhelmed by the amount of interest this experiment received, by how many people were sharing it, and how many newspapers around the world reported on it. In fact, even our servers were sometimes overwhelmed: On several occasions I had to turn back to the office to restart the server after it had silently crashed underneath my desk.
When we started analyzing the data in early June, only a few months after launching the experiment, more than 500,000 people from 218 countries and territories participated. The map below shows where our participants came from (darker color means more participants).
If you were one of the participants, we thank you! With your help, we were able to gain new insights into how clicking performance differs depending on age, gender and country. Over the next several weeks, I plan to write up some of the most interesting results and share them with you.
Today I will start by telling you how the overall clicking performance changes with age. This is only one of many measures that we recorded from the experiment to “guess” people’s age, but arguably a very important one. And if you are really interested in how exactly we measured the performance, we have the geeky details at the bottom of this post.
But let’s talk about the results. The graph below shows the average performance per age between 5 and 85. The error bars show the 95% confidence intervals – the smaller the error bars, the more reliable the result.
And here we are already with some bad news for you. If you are over 17 years of age, you have seen your best times, at least in the mouse-clicking world. On average, 17-year olds are the most efficient clickers. Starting in late twenties, we all begin to very steadily get a little slower every year.
If you are under 17 years of age, the news is great. Children and adolescents get faster very quickly from year to year. This is actually very important for those of us building software for children because it means that even in a single classroom we are likely to see children with very different clicking speeds. Finally, you can see that the clicking efficiency of young children (5 through 7 year old) is very similar to the clicking efficiency of our participants aged 80 and older.
Some technical details
How exactly do we measure the clicking performance? You might have noticed that it is much easier (and therefore faster) to click on big things than on small things. It is, of course, also faster to click on things that are nearby than on those that are far away. Because our experiment was designed to adapt to the size of the screen on which it was displayed, everybody saw a slightly different version of the experiment: for some, all the dots were close together, while for others the dots were further apart.
To make it possible to compare how different people did, even though each completed a slightly different task, we used the concept of Throughput (sometimes also called Index of Performance) developed in 1954 by a psychologist named Paul Fitts. Paul Fitts developed an equation for measuring a difficulty of a clicking task that depended on how far one had to move and how big the target was. Of course, there were no personal computers in 1954 — what Paul Fitts was studying was how quickly and accurately people could point with their fingers, but, as it turns out, his findings apply to mouse clicking as well. His throughput measure is simply the difficulty of the task divided by the time it takes a person to complete it. Throughput is measured in bits per second. We realize that saying that a person has a “throughput” and measuring it in bits per second is very geeky, but it works. Think of it as speed: the higher the better.
A vast amount of research findings out there relies on studies that were conducted with American undergraduates. In fact, most of what we know about people’s perception, cognitive skills, or their behavior is derived from such studies. Yep, we know pretty well how American undergraduate students work.
But we have only found out recently that American undergrads are not necessarily representative of the world’s population. In fact, they are quite a WEIRD species: Western, Educated, Industrialized, Rich, and Democratic. (See Henrich et al.’s paper (PDF) for some excellent examples on how WEIRD some of us are.)
We totally agree that science is a little bit WEIRD. This is also why we have proposed a panel presentation at the SXSW conference next year to show how much better we can do by collecting larger, and more diverse samples on LabintheWild.org, TestMyBrain.org, and GamesWithWords.org.
In order to be selected, however, we need your vote!
Thanks for your support and for helping science become a little less WEIRD. :)
We design studies for LabintheWild based on this central idea: we trust you. Whether you are a new arrival or a long-time participant, we trust your responses to be an honest representation of your beliefs and perceptions of the Web. We also hope that you trust us - to use your information carefully and to provide a fun experience where you can learn more about yourself.
But what about other websites? Trust is a decisive factor for engagement when users go online, so it’s important to understand what in a website makes users decide - even within the first few moments of visiting that site - whether to stay or leave in search of greener (and perhaps safer) pastures.
With this question in mind, the LabintheWild team is delighted to announce our new Trustworthiness Test, where you can use your intuition to rate websites based on their trustworthiness. We’re interested in testing out claims from previous studies that we can make judgments about websites within milliseconds (!) of seeing them for the first time. We are also interested in investigating what people perceive as (un)trustworthy and how this differs around the world. At the end of the test, you’ll see how well your intuition of trustworthiness matches reality. Click here to take the test, and stay tuned for our findings!
This just in: a new study that focuses on how you use information to predict the future. Our LabintheWild summer team - Michelle, Willy, Rishav, Jon, Dianna, Katharina, and Krzysztof - has been hard at work over the past few weeks, and we’re very excited to present our new Graph Prediction study, which analyzes the way you make predictions and compares your tendencies to others around the world. Participate now!
LabintheWild has a new design! Check out the Wild Facts on our homepage to see how much we’ve grown over the past few months - it’s incredible that we now have participants from more than 200 countries and regions all over the world! With such a growing number of participants, we want to make sure that everyone can stay updated on the latest and greatest from LabintheWild. You can now subscribe or follow us on Facebook, Twitter, or Tumblr to hear about new studies and results!
Internet users make lasting judgments about a website’s appeal within a split second of seeing it for the first time. This first impression is influential enough to later affect our opinions of a site’s usability and trustworthiness.
Because the first impression counts, we developed an approach to predict whether users will find a website appealing. Given a website’s screenshot, we are now able to
In the future, we would like to enhance this approach to work on a more individual level, for example to enable a comparison of the preferences in various cultures. But of course, we need your help. Please participate in our ongoing study!
Interesting in reading more? Have a look at our recently published paper:
Katharina Reinecke, Tom Yeh, Luke Miratrix, Rahmatri Mardiko, Yuechen Zhao, Jenny Liu, and Krzysztof Z. Gajos. Predicting Users’ First Impressions of Website Aesthetics With a Quantification of Perceived Visual Complexity and Colorfulness, in Proceedings of Human Factors in Computing Systems (CHI), 2013. Download paper
Check out our new Facebook page! We’ll make sure to provide updates there if a new experiment comes online or if we have results for our existing experiments. Thanks to all of you who have already participated!
Many people have asked me why LabintheWild is mainly about exploring how we are different and similar in our cultural preferences. I guess one of the reasons is that I have always liked to travel and see how people around the world differ from each other. What I find fascinating, and what my research has shown, is that these differences also translate to the online world.
I first noticed this around 7 years ago when I lived in Rwanda for a while to develop an e-learning platform for agricultural advisors in the country. Being completely ignorant about cultural preferences, I designed the user interface the way I thought was nice and intuitive. I guess my German background must have convinced me that “nice” means plain, very few colors, and very little information. Whatever it was, my Rwandan co-workers didn’t like it at all. It took me a while to understand that they preferred more colors, a higher complexity of the user interface, and a linear structure that would lead them through each lesson step by step. But why? Why doesn’t everyone in the world have the same preferences? One reason might be that everything in Rwanda is a little bit more colorful. At least in comparison to Northern Germany! Their clothes, the nature, … the colors make everything look a little more cheerful.
In an attempt to explain some more preferences, I also visited a few local schools and noticed that in those schools, the education was mostly teacher-centered and students focused on following instructions. Could this maybe explain why my co-workers favored more guidance in the e-learning platform?
Well, years have past and answering these questions has proved to be a little more complicated than I originally thought and has evolved to be my central research interest. What I have found so far is that online cultural preferences are quite complex—but they give a unique and fascinating view into who we are and how we experience today’s global village called the Internet.
We’ve just launched LabintheWild with three exciting experiments revolving around culture and how it influences people’s interaction with computers. Enjoy learning about your own cultural particularities, how other cultures see the world, and make sure to check back to read about our first results for these studies.
Copyright © by LabintheWild.org