If you’re a regular reader of SoundGuys, you’ve probably noticed that we like to keep things as direct and unimpeachable as possible. However, we’re a collection of human beings—and we often screw up. Every so often I go through the way we do things to see if there’s any improvements that we can make, and if so: implement them. However, I thought it would be helpful for superfans and casual Googlers alike to see how we handle our work with things go awry.
Whether you’re a veteran review-junkie or someone just looking for the best product to buy, it can be very comforting to see a review with high ratings on Amazon or in print. Surely, you might say, that high score means this is the best product for me to buy! Unfortunately, that’s not always the case (and don’t call me Shirley).
The thing is: data can only tell you so much, and there’s often a piece of the puzzle missing when objective data is used to determine the “best” product in any category. You can’t just rely on a number to tell you everything you need to know about any product out there, simply because the only thing that matters is how you’ll like something. That’s why we spend so much time on our reviews to contextualize everything we’ve found with a set of headphones, earphones, or speakers.
Editor’s note: we have published our audit for 2021, which can be found here.
Scoring audit: our scores diverge from user scores
After scraping all scores for all reviews on our site, we wanted to see how well we approximated value judgments for different aspects of products’ performance. I personally was looking for a variance of under 1.5 points in either direction, and we hit that—with one notable exception.
Users rate isolation/attenuation much higher than we do
I’m not surprised by this result, as it’s one of those things that we fundamentally disagree with our readers on, and probably always will. The way we rate noise attenuation is built to reward both headphones and active noise cancelers on the same scale, so that way you can directly compare the effectiveness of all headphones to one another.
Because many ANC headsets offer better noise attenuation by a couple orders of magnitude, making a fair scoring scheme is difficult, and it doesn’t fit the mental models of many people who are only comparing headphones of a certain type. While many people will rate headphones that block out a bunch of noise highly, it might not be taking how well ANC headphones perform at the same task into account. Consequently, we couldn’t make an isolation/attenuation score for each product category because we then couldn’t directly compare products across categories.
Users rate sound quality lower than we do
The other anomaly in our scoring is that our users are pretty much universally more negative on sound quality than we are. While that may be because the very nature of human ears means people disagree on a lot, I wasn’t expecting the near-universal dip in scores compared to ours.
While it’s not a huge difference, it’s still a difference—and that’s worth exploring. So I plotted all the counts of user scores for sound quality and:
A normal distribution (like the one above) around 7.5 is something you’d expect with complete coverage of a single category of headphones, but it’s a little strange that it worked out this way. I say this because the total breadth of coverage on SoundGuys tends to avoid both the upper and lower extremes of the headphones available, so our review selection is a little more curated. In that case, you wouldn’t necessarily expect scores to look like a nice, neat bell curve. If you were to try to review the top options of any industry, the scoring distribution would probably look a less pretty, like the SoundGuys house scoring:
While that doesn’t tell us anything out of the gate, it’s possible that we should re-examine how we score sound quality, because even though most of the scores are well separated: they gravitate more toward the top of the scale than I’d like to see. Though it’s possible that most headphones nowadays are approaching the point where their sound quality is a settled issue—I don’t believe it. Not yet, anyway. We’re going to look to dial in these scores a bit more, but the good news is we have all of the test data on hand to do this. The big thing we have to do is make new scoring algorithms.
Proposed remedy: revisit scores for sound quality. Expand testing.
Be sure to check back as we explore new methods of scoring and analysis. While it’s tempting to acknowledge the difference and move on, we’re going to test a few things behind the scenes to hopefully bridge the gap between our readers and our own analysis. We feel like any excuse to improve our data collection is worth it.
An ideal gaming mic is not an ideal microphone
A while back we had a couple manufacturers cry foul when we rated some microphones lower than others because they attempted to reject low-end noise by being less sensitive to lower frequency sounds altogether. At the time we saw no reason to change our methodology, but after reviewing a large number of gaming headsets, we’re forced to conclude that we were wrong—so we’re changing the way we score microphones.
Think about it this way: with a gaming headset, the microphone is going to be pretty close to your mouth, right? Well, that means it’s susceptible to something called the “proximity effect.” In short, the proximity effect is what makes many radio DJs and podcasters sound like they have a boomy, bassy voice on the air. When you use a directional mic and get too close to it, you’ll notice that sounds under 200Hz can get up to a 30dB bump in emphasis. Great for recording kick drums, not so much for recording voices. By allowing that kind of emphasis, you can unintentionally make your voice tougher to understand over voice chat, which needs sounds in the highs to stand out for your brain to differentiate many consonant sounds.
You can probably see where this is going, but I’ll break it down anyway. In order to combat this, most audio professionals will de-emphasize low-frequency sounds to avoid the negative consequences of the proximity effect, and take all the work of equalizing gamers’ voices away so nobody has to think about it. In our view: this is the correct move, and we’ll be taking steps to address this in our scoring.
Normally we fight to keep all scores comparable to one another, but this is a situation where that isn’t the best idea. Namely, the results people get from normal use of gaming headset mics will be different than what they want them to be without low-end de-emphasis, so comparing a gaming headset mic against dedicated studio options is unfair.
Proposed remedy: create new scoring target for gaming headsets
In the near future, you’re going to see the scores of microphones change a bit, namely to account for gaming headsets and their varying low-end attenuation. It’s for the better that we move to meet the needs of consumers, so we’re going to do exactly that.
I do periodic audits of our methods and content, and this one was brewing for some time. Most of our slowness on some of these issues stems from needing a large corpus of data to analyze, and our readers have been very generous with their own opinions and assessments! We’re very appreciative of our readers’ help in grappling with a useful scoring system.
If you want to see changes, feel free to reach out on our contact page, or even just leave a question on our FAQ at the bottom of every article we have. While I can’t promise we’ll ever be perfect, I can promise that we’ll always try to do better than we did previously… and that’ll be true for as long as we’re around.