The Divnich Debrief: Tainted Reviews a 'Very Scary Prospect'

Last week, the researchers at EEDAR released a new study in collaboration with SMU Guildhall, which revealed just how powerful professional game reviewers can be in the industry, in terms of their direct impact on purchasing decisions and word-of-mouth marketing among friends and family. For this edition of The Divnich Debrief with EEDAR analyst Jesse Divnich, we decided to follow-up with Jesse to get some more insight into the study.

IndustryGamers: Most would say it's obvious that there's a correlation between review scores and sales, so why do an in-depth study?

Jesse Divnich: You are right, but every time some new report, article or comment from a prominent professional goes near the subject suggesting a correlation exists, it always sparks a debate and that has caused many qualms on the subject. In fact, most publishers don’t even speak in terms of “achieving” certain review scores publicly; instead they strive for “high quality” just to avoid the entire “Metacritic/GameRankings” subject. Internally we felt this subject has been beaten to death, but the industry continues to debate it so we wanted to be the ones to take the final swing at this dead horse to end the debates once and for all – not to mention that because of the way the study was designed, we can actually conclude that there exists in the video game industry a causal relationship between reviews from respected, professional outlets and sales performance; much stronger than a correlative relationship.

IG: Were you surprised by the results?

JD: Myself, I’ve been reading psychology text books (for fun) for the last 10 years and Geoffrey Zatkin, Co-Founder and COO of EEDAR, has a degree in Psychology, so we were both well educated on Anchoring and the natural ability of people to rely too heavily on pre-exposed information. So no, we were not too surprised. 

I will say that reading about it and witnessing it are two different experiences. It was just amazing to see these participants all play the same game for the same amount of time, giving them ample enough time to form their own opinion, and yet this one little variable, review scores/review comments, made their actions drastically different – so drastically different that the group exposed to high reviews gave Plants vs. Zombies a review score 20% higher than the group exposed to low reviews, and the high review group were twice as likely to take a copy of the game over the $10 cash payment offered for participating.

IG: The study also measured the ability for review scores to influence word-of-mouth. What's the takeaway from this part of the study?

JD: Actually, we were quite surprised by this part. The study found that the group that was exposed to high review scores was 40% more likely to recommend the game to a friend compared to the low review score group. This suggests that high critical acclaim from credible media outlets can continue to influence purchases long past the original consumer who might have leveraged professional review information to make a purchasing decision. 

IG: What does this mean to professional media outlets and the reach of their audience?

JD: It is pretty fascinating because the study would suggest that the reach of professional review outlets is much larger than their direct audience. Adam Sessler from G4TV made an interesting self-observation about his role as a reviewer. He always felt his ability to educate consumers was no bigger than his direct audience; however, we can now see the breadth of his reach, and other review outlets are much larger.

To put it into perspective: Over 3 million people have purchased Red Dead Redemption. No doubt a significant portion of game sales came from personal recommendations (compared with professional reviews). The study suggests, however, that if we peel back the layers, those recommendations likely came from consumers who bought the game because of positive professional reviews in the first place.

Originally we assumed the influence of reviews ended after a consumer made a purchase because at that point the consumer would be able to conduct their own unbiased and independent self-evaluation. If the game lived up to the hype, the consumer would recommend it to friends; if not, they wouldn’t. But the study suggests the chain-of-events are not independent from each other. Just one consumer reading a professional review could impact whether another entire group of people, who never heard of Metacritic, G4TV, etc., ended up purchasing a game. The study certainly reinforces the assumptions that credible media outlets have an incredible amount of power in our industry.

IG: Are you afraid this study could increase the pressure on media outlets to possibly score games a certain way? 

JD: It’s a very scary prospect. We have seen some articles reporting on our study remark how media outlets can “dupe” consumers; however, we need to keep in mind that one of the primary reasons anchoring is so effective with video game consumers is because of the strong trust between gamers and credible professional critics. We believe the causal link between professional review scores and sales is contingent on the process remaining unbiased. If media outlets or publishers tamper with the process, it will only deteriorate that trust between credible outlets and consumers, and if that happens we are no different than the movie industry which is filled with “reviewers” who would rather see their quotes and name on a movie poster than properly educating consumers.

That is the entire reason EEDAR and SMU Guildhall made this report free for anyone to enjoy, despite the large costs both parties occurred. We felt that the results were so powerful, that if left in the hands of just a select few it could potentially unbalance this ecosystem within the video game industry.