Whether you’re an audiophile or just someone looking for your first set of headphones, finding information you can trust is difficult. With so many conflicting opinions, marketing, and—let’s be frank—lies surrounding personal audio products online: how is anyone supposed to know what’s right or wrong? We believe it takes a dram of experience, and a gallon of testing.
SoundGuys is a collection of journalists dedicated to cutting out as much BS as possible, while still giving you the information you need in order to maximize your happiness with your personal audio products. In our quest to demystify the world of music, we regularly post education features—but we also test the headphones that come our way with a standardized process. In order to make sure nobody misses out on what our reporting means, we want to take a minute to go over each of our tests.
Editor’s note: this article was updated on May 14, 2021, to expand upon the frequency response, noise cancelling, and microphone testing sections.
In order to gather information about how each product works, we subject them to a battery of testing we’ve created for our readers. Objective testing refers to the scientifically verifiable performance of the product using standardized tests in ideal conditions. While it is certainly extremely important, we here at SoundGuys don’t subscribe to the false dichotomy of the objectivist vs. subjectivist debate in audio reporting.
For our headphone testing, we use a Bruel & Kjaer 5128 head simulator hooked up to a Listen Inc. SoundConnect 2 and AudioConnect to power the ear simulators and analyze the signal. The measurements are then fed to our test computer, and we use a program called SoundCheck (also by Listen Inc.) to record, analyze, and export all the data. Because this hardware and software combo is what’s used on some manufacturing lines, we’re able to test and quantify products much in the same way that they would in the factory. This way, we can troubleshoot different possibilities, and even detect if something’s broken or not before publish.
When anyone mentions “frequency response,” they’re talking about how each note in your music is emphasized. Most headphones don’t play each sound back as loud as the others, and that’s a very deliberate choice. While studio headphones try to make every note exactly as loud as all the other ones, most consumer headphones will do things like boost bass, and some high harmonics. You can read more about it here.
We calibrate our Bruel & Kjaer 5128 test fixture for each sample, and apply correction curves to account for how people hear the world around them. Then we take the data from that test and put it into a chart that’s a little easier to read than a spreadsheet with 57,000 data points. Here’s what that looks like:
The pink line represents our house curve, what we posit as the platonic ideal, and the cyan line represents the product’s normalized frequency response. This chart will give you a general idea what the headphones sound like—and also give you an idea how you might want to equalize them once you buy.
For all you nerdy audiophiles out there, we use a psychoacoustic smoothing algorithm to solve for math outliers in our charts. All that means is that we correct for errors based on how humans actually hear instead of posting a jagged and confusing line. While this won’t satisfy hardcore enthusiasts, it removes a lot of the uncertainty and errors that are necessarily created by using a non-human fixture.
Not everyone likes studio sound, and some people can't stand consumer headphones. It's just a fact of life.
While there are many competing targets, there is no one true ideal for everyone. Every single person on Earth hears slightly differently. Not everyone likes studio sound, and some people can’t stand consumer headphones. It’s just a fact of life. That’s why we also listen to the headphones, and point to actual songs to contextualize what you’d hear if you bought them.
Isolation and active noise cancelling
Because lots of people like to take their headphones out in the world, we also feel that it’s important to test and show how each set of headphones or in-ears blocks sound. Not every noise is blocked out equally, and reading advertisements can’t tell you how well a set of headphones cancels out sound. Using the same scale as the frequency response chart, we can explore how different headphones act.
To test how well a set of headphones or in-ears keeps sound from reaching your ear, we play a loud sample of pink noise over a speaker 1m away with the headphones off our robot ear unit—and record. Then we put the headphones on and record again. After subtracting one curve from the other, we can display the data in a way that makes sense.
In the example above, we can see that the Bose QuietComfort 35 II noise cancellation combats low noises (0-1000Hz) like engine rumbles, car noise, and voices. The pink line represents isolation performance which takes care of higher-pitched sounds (1000Hz and up), think of nails going across a chalkboard. The Bose QC 35 II is great for travel and surprisingly better than the newer Bose Noise Cancelling Headphones 700.
Note: the ANC score on each review reflects objective performance, not comparative, so it reflects a product’s gross attenuation. This can make things like versus articles a bit odd: when we compare the Jabra Elite 75t to the Apple AirPods Pro, the ANC scores alone appear to indicate that the Elite 75t has more effective noise cancelling than the AirPods Pro, but that’s not the case. Again, the ANC score accounts for the gross attenuation, and combines passive isolation with noise cancelling performance.
Microphone testing is fairly straightforward at SoundGuys. In studio, we use a combination of frequency response testing and practical use to determine how each microphone performs. We use a Listen Inc. SCM-3 microphone with the substitution method to determine the performance of each device we test.
The microphone score follows how our readers rate a corresponding sample. This may draw some ire, but we feel it’s important for the mic performance to be assessed in the appropriate context. In the case of true wireless earbuds, this means the mics are generally used for hands-free calls and even conference calls, not for live musical performances.
Some people like Bluetooth headphones, and that means having to deal with charging a battery. Because we have testing fixtures with human-like ears, we can tell you how long you can expect your cans to keep making music.
We're trying to meet a normal use case—and an independently verifiable performance metric.
First, we find out what settings to bring each set to 75dB(SPL), a safe and enjoyable listening volume. Then we set a playlist to repeat infinitely, and a recorded waveform tells us exactly how long the headphones or in-ears lasted. Often, this is different than what manufacturers tell you, but it’s meant to reflect your average consumer’s use more than just some nebulous specification.
Wired headphones are increasingly rare nowadays, but sometimes they simply require too much power to be used on a smartphone. In the event that happens, we will let you know. By using a little bit of math and the specifications page of any set of headphones, we can tell you if the power output of your smartphone is enough. If you don’t see a section talking about power, assume that everything is okay and you shouldn’t run into volume problems.
In the world of audio, objective testing isn’t enough because there are so many factors that can’t be controlled for. Tests regarding certain things are generally impossible to perform given the insane range of head sizes, ear shapes, and noise-induced hearing loss of individuals across the globe. That’s why we take each audio product we review and give it an extended spin in a normal home, with normal people—just like you.
Sometimes we find some faults, other times we don’t, but we always do our due diligence in figuring out what each product is like. Sometimes our experiences will differ from yours, and that’s okay! Let us know in the comments if that happens, and we will periodically update our articles if we’ve missed something important.
Not all headphones are made to last, and it’s usually quite apparent from the materials they’re made of. We try to list as much as we can about the construction of the cans, whether they have removable cables, and other concerns about durability.
Obviously, no two heads are identical, but we can figure out when headphones aren’t going to be right for you by actually trying them out. We check for heat buildup, how much force is put on your head, and many other considerations. We never review a set of headphones without using them for hours on end.
Sometimes headphones aren’t amazing, but they satisfy a need extremely well. Not only so we identify the main target for every model of headphones we review, but we also weigh its value. That could mean how much you’re getting for how little money spent, or it offers a truly unique feature not seen anywhere else.
Je ne sais quois
If there’s a feature or product that is so different from what we cover, we will do our best to develop and add testing for it. However, that’s not always possible. In the event that happens, we will usually put it into a real world situation, and kick the tires there.
We’re constantly re-evaluating how we do things here at SoundGuys, and are always looking to upgrade our equipment, coverage, and education resources. We don’t chase numbers, we chase the truth. You can read our ethics statement for more.