Headphone reviews are in a bad way for the most part. Compared to other gadgets, this product category is probably the most rife with compromised ethics, bad testing methodology, and simply untrue statements.
Though it’s a relative rarity to see a headphone site provide any sort of standardized testing, this particular site has made the leap to provide a more objective picture of what’s going on with the products you buy. You may have noticed oddly-specific charts and tables showing up in our reviews of late, so let me explain why they’re there.
Reviews often don’t tell you what you want to know
Much like any other study, approaching the problem of “which headphones do I buy” requires a certain philosophy to guide you. Because your life will dictate different needs than others, there is no one “best” set of headphones—only what might meet your needs. Before you read a review, you should be looking to answer a few specific questions:
- Do the headphones have the features do you need? (Price, features, do you need an amp, etc.)
- What can you do without?
- What kind of sound do you like?
- Are there any unsavory surprises waiting for you?
While you can read about the first two just about anywhere, it’s that last one that most sites trip up on because it’s very difficult—and very expensive—to do it right. For this reason, a figurative monsoon of content that lacks credibility or substance clogs forums, Amazon pages, and blogs that review without any sort of independent verification of fact.
Reading these reviews is not only a chore, but often worse than useless. I say “worse” because many straight-up lie. They could be written by a marketer for all you know! This happens all the time.
On top of that, the way that many people describe sound is also misleading. Many of these reviews will contain overly-flowery prose rife with unquantifiable adjectives. The surest sign that a review is a waste of your time is if you’re reading lots of descriptions of sound that can’t be supported with numbers. Because audio is something we know a lot about, it doesn’t help to explain in terms that aren’t math-based. While that can be pretty boring, it’s much more helpful to actually address the issue in a way that isn’t subjective.
Objective testing helps us as writers lay all the cards on the table so you’re not making any purchasing decisions based on faith. How does a product sound? Is there a flaw? What exactly does it block out in a noisy area? Nothing’s left to the imagination: you’ll have all the information you need.
How to separate facts from fake news
Like any other online reviews, it’s very difficult to find a source you can trust right away. However, I’ll give you a few questions to ask yourself that works for any product category you could ever come across.
1. Who’s writing this review?
If we’re trying to establish what’s useful to know and what’s not, starting with strongly ethical writers dedicated to finding and communicating truth is essential. Amazon reviewers could be anyone, from just some guy to an intern at a marketing department trying to drive that product rating up. You want someone who not only knows a lot about your product category, but will also be honest with you—an expert without an ulterior motive.
For us here at SoundGuys, we wear our qualifications on our sleeves. We have several years’ experience in objectively testing audio products like headphones, and we’ve been around long enough to see where things might be headed. On top of that, we have a strict ethics policy that prevents any of our writers from giving any product preferential treatment, and we also look to hire trained journalists to test products. Should we get something wrong: we own it, and fix the issue rather than risk readers getting misinformation. Our singular goal is to arm you with the best information possible.
You can look for similarly-minded reviewers by reading the ethics pages of their sites. If they don’t have an ethics page or clearly detail how they test products… you want to move on.
2. What information is conveyed? Is opinion minimized?
Though it can be tough to tell the difference between information and opinion, the litmus test most people use is simply “Does the reviewer show their work?” For headphone critics, that can be anything from showing charts and graphs to simply describing tests they performed and how to do it yourself. Without any understanding of how something is assessed, it’s almost impossible to translate the reviewer’s experience to your own. This is especially true for comparisons; if you’re not familiar with the headphones the reviewer is comparing your unit to, what have you learned?
While numbers and technical explanations are about as interesting as watching paint dry, they offer an extremely valuable insight into how well a product actually works, especially when you have an expert to contextualize the results. They’ll tell you if you need an amp, what kind of sound certain headphones have, or if a certain feature is a gimmick (or actually useful).
Look for an overabundance of “I” statements like “I feel,” or “I think.” It’s a good clue that the information being presented to you wasn’t arrived at by any sort of objective process, but rather by a subjective experience.
3. Separate fact from feelings
Finally, if a review is confusing, that may be by design. Remember those unquantifiable adjectives I mentioned earlier? Often people will search for words to act as filler when they can’t demonstrate a concept to a reader. Marketers also sometimes do this, but in a different way. If you read copy that contains a lot of flowery, nice-sounding words that sound like they came out of a wine review, you should pay close attention to how each of those words like “brilliant,” “balanced,” “deep,” and “rich” make you feel. These words aren’t there to convey information, but to make you have emotion about a product.
Needless to say, most people shouldn’t be dealing with strong emotions when deciding what products to buy. It’s a pair of headphones, not a damned wedding venue. If you want the best information possible, we need to deal in facts and figures instead of emotions. It’s easy to feel strongly about something, but it’s critical to explore issues as they are—not how we might feel they should be.
There’s a lot to unpack when discussing the philosophy behind online headphone reviews, but for the most part all you need to know is that it follows a straightforward disagreement: subjectivists vs. objectivists. Objectivists believe that you can learn everything you need to know from scientifically-collected data, and subjectivists believe that you should just trust your ears. Because the extremes of each camp disregard each others’ arguments completely, scores of heated arguments rage online.
It's easy to feel strongly about something, but it's critical to explore issues as they are—not how we might feel they should be.
Obviously, SoundGuys falls somewhere in between these two camps, because we currently can’t gather and analyze data on every reader’s ears. Since that’s currently impossible, we trudge on with the best available methods available to us. While there are a lot of purely-subjective reviews out there that use feeling more than fact, we caution you to not put stock into those descriptions. They don’t tell you as much as you think they might be telling you.
Holding reviewers accountable
You may still run into situations where your experience differs from that of the reviewer. That’s why we have a Frequently Asked Questions section below every review and best list, as it keeps an open line of communication between you and the author. Additionally, you can reach out to us on Twitter or email. While you may not receive a response, it’s important to put your questions out there. How else are publishers going to know what to address if they’ve missed something? Always speak up if you can—you never know what kind of change you may spark!
A disciplined approach and a healthy curiosity will bring you a deep appreciation for any subject—and that’s why we test the products we review. We want to discover and report the truth about everything we cover.