User Reviews Revisited
Since that post, we’ve continued to listen to feedback from both players and developers. It’s clear to us that players value reviews highly, and want us to ensure they’re accurate and trustworthy. Developers understand that they’re valuable to players, but want to feel like they’re being treated fairly. We’ve also spent a bunch of time building analysis tools to help us better understand what’s happening in the reviews across all titles on Steam. With that feedback and data in hand, we think we’re ready to make another change.
That change can be described easily: we’re going to identify off-topic review bombs, and remove them from the Review Score.
But while easy to say, it raises a bunch of questions, so let’s dig into the details. First, what do we mean by an off-topic review bomb? As we defined back in our original post, a review bomb is where players post a large number of reviews in a short period of time, aimed at lowering the Review Score of a game. We define an off-topic review bomb as one where the focus of those reviews is on a topic that we consider unrelated to the likelihood that future purchasers will be happy if they buy the game, and hence not something that should be added to the Review Score.
Obviously, there’s a grey area here, because there’s a wide range of things that players care about. So how will we identify these off-topic review bombs? The first step is a tool we’ve built that identifies any anomalous review activity on all games on Steam in as close to real-time as possible. It doesn’t know why a given game is receiving anomalous review activity, and it doesn’t even try to figure that out. Instead, it notifies a team of people at Valve, who’ll then go and investigate. We’ve already run our tool across the entire history of reviews on Steam, identifying many reasons why games have seen periods of anomalous review activity, and off-topic review bombs appear to only be a small number of them.
Once our team has identified that the anomalous activity is an off-topic review bomb, we’ll mark the time period it encompasses and notify the developer. The reviews within that time period will then be removed from the Review Score calculation. As before, the reviews themselves are left untouched – if you want to dig into them to see if they’re relevant to you, you’ll still be able to do so. To help you do that, we’ve made it clear when you’re looking at a store page where we’ve removed some reviews by default, and we’ve further improved the UI around anomalous review periods.
Finally, we’ve also enabled you to opt out of this entirely, if that’s your preference – there’s now a checkbox in your Steam Store options where you can choose to have off-topic review bombs still included in all the Review Scores you see.
While we’re working on some other features around User Reviews, we thought this one was worth shipping by itself. As always, if you have thoughts or concerns, feel free to voice them in the comments below.
Q: I care about some things that I worry other players don’t, like DRM or EULA changes. Review bombs have been about them in the past. Do you consider them unrelated or off-topic?
A: We had long debates about these two, and others like them. They’re technically not a part of the game, but they are an issue for some players. In the end, we’ve decided to define them as off-topic review bombs. Our reasoning is that the “general” Steam player doesn’t care as much about them, so the Review Score is more accurate if it doesn’t contain them. In addition, we believe that players who do care about topics like DRM are often willing to dig a little deeper into games before purchasing – which is why we still keep all the reviews within the review bombs. It only takes a minute to dig into those reviews to see if the issue is something you care about.
Q: So if I post a review inside in the period of an off-topic review bomb, my review won’t be included in the Review Score?
A: Unfortunately, this is correct. We’ve tested our process of identifying off-topic review bombs on the entire history of reviews on Steam, and in doing so, we’ve found that while we can look through reviews and community discussions to determine what’s behind the review bomb, it isn’t feasible for us to read every single review. But as we mentioned back in our first User Review post, our data shows us that review bombs tend to be temporary distortions, so we believe the Review Score will still be accurate, and other players will still be able to find and read your review within the period.
