Whether you’re an audiophile or just someone looking for your first set of headphones, finding information you can trust is difficult. With so many conflicting opinions, marketing, and—let’s be frank—lies surrounding personal audio products online: how is anyone supposed to know what’s right or wrong? We believe it takes a dram of experience, and a gallon of testing.
SoundGuys is a collection of journalists who have dedicated ourselves to cutting out as much BS as possible, while still giving you the information you need in order to maximize your happiness with your personal audio products. In our quest to demystify the world of music, we regularly post education features—but we also test the headphones that come our way with a standardized process. In order to make sure nobody misses out on what our reporting means, we want to take a minute to go over each of our tests.
Editor’s note: this article was updated on February 12, 2021 to reflect equipment changes.
In order to gather information about how each product works, we subject them to a battery of testing we’ve created for our readers. Objective testing refers to the scientifically verifiable performance of the product using standardized tests in ideal conditions. While it is certainly extremely important, we here at SoundGuys don’t subscribe to the false dichotomy of the objectivist vs. subjectivist debate in audio reporting.
For our headphone testing, we use a Bruel & Kjaer 5128 head simulator hooked up to a Listen Inc. SoundConnect 2 and AudioConnect to power the ear simulators and analyze the signal. The measurements are then fed to our test computer, and we use a program called SoundCheck (also by Listen Inc.) to record, analyze, and export all the data. Because this hardware and software combo is what’s used on some manufacturing lines, we’re able to test and quantify products much in the same way that they would in the factory. This way, we can troubleshoot different possibilities, and even detect if something’s broken or not before publish.
When anyone mentions “frequency response,” they’re talking about how each note in your music is emphasized. Most headphones don’t play each sound back as loud as the others, and that’s a very deliberate choice. While studio headphones try to make every note exactly as loud as all the other ones, most consumer headphones will do things like boost bass, and some high harmonics. You can read more about it here.
We have a test fixture that we calibrate for each sample, and apply correction curves to account for how people hear the world around them. Then we take the data from that test and put it into a chart that’s a little easier to read than a spreadsheet with 57,000 data points. Here’s what that looks like:how you might want to equalize them once you buy.
For all you nerdy audiophiles out there, we use a psychoacoustic smoothing algorithm to solve for math outliers in our charts. All that means is that we correct for errors based on how humans actually hear instead of posting a jagged and confusing line. While this won’t satisfy hardcore enthusiasts, it removes a lot of the uncertainty and errors that are necessarily created by using a non-human fixture.
Not everyone likes studio sound, and some people can't stand consumer headphones. It's just a fact of life.
While there are many competing targets, there is no one true ideal for everyone. Every single person on Earth hears slightly differently. Not everyone likes studio sound, and some people can’t stand consumer headphones. It’s just a fact of life. That’s why we also listen to the headphones, and point to actual songs to contextualize what you’d hear if you bought them.
Isolation and Active Noise Canceling
Because lots of people like to take their headphones out in the world, we also feel that it’s important to test and show how each set of headphones or in-ears blocks sound. Not every noise is blocked out equally, and reading advertisements can’t tell you how well a set of headphones cancels out sound. Using the same scale as the frequency response chart, we can explore how different headphones act.
To test how well a set of headphones or in-ears keeps sound from reaching your ear, we play a loud sample of pink noise over a speaker 1m away with the headphones off our robot ear unit—and record. Then we put the headphones on and record again. After subtracting one curve from the other, we can display the data in a way that makes sense.
In the example above, we can see that low noises like engine rumbles, car noise, and voices aren’t blocked out (0-1000Hz), but sounds 3,000Hz and up like airplane engines, and nails going across a chalkboard would be (green, cyan). Probably not the best option to take on the subway. But something that blocks out more low end sound would be! While we rate each set on how much noise is blocked out or attenuated, every review will discuss what noises you can expect to be free from.
Microphone testing is fairly straightforward at SoundGuys. Our writers use a combination of frequency response testing and practical use to determine how each microphone performs. We use a Listen Inc. SCM-3 microphone with the substitution method to determine the performance of each device we test.
Each microphone is judged against its intended use-case, so there’s no unfair penalization occurring. For example, an instrument microphone will be judged against the whole audible range of frequencies, and a set of headphones will have their mics judged solely by the range of notes and harmonics a human vocal tract could produce (50-4,000Hz).
Some people like Bluetooth headphones, and that means having to deal with charging a battery. Because we have testing fixtures with human-like ears, we can tell you how long you can expect your cans to keep making music.
We're trying to meet a normal use case—and an independently verifiable performance metric.
First, we find out what settings to bring each set to 75dB(SPL), a safe and enjoyable listening volume. Then we set a playlist to repeat infinitely, and a recorded waveform tells us exactly how long the headphones or in-ears lasted. Often, this is different than what manufacturers tell you, but it’s meant to reflect your average consumer’s use more than just some nebulous specification.
Wired headphones are increasingly rare nowadays, but sometimes they simply require too much power to be used on a smartphone. In the event that happens, we will let you know. By using a little bit of math and the specifications page of any set of headphones, we can tell you if the power output of your smartphone is enough. If you don’t see a section talking about power, assume that everything is okay and you shouldn’t run into volume problems.
In the world of audio, objective testing isn’t enough because there’s so many factors that can’t be controlled for. Tests regarding certain things are generally impossible to perform given the insane range of head sizes, ear shapes, and noise-induced hearing loss of individuals across the globe. That’s why we take each audio product we review and give it an extended spin in a normal home, with normal people—just like you.
Sometimes we find some faults, other times we don’t, but we always do our due diligence in figuring out what each product is like. Sometimes our experiences will differ from yours, and that’s okay! Let us know in the comments if that happens, and we will periodically update our articles if we’ve missed something important.
Not all headphones are made to last, and it’s usually quite apparent from the materials they’re made of. We try to list as much as we can about the construction of the cans, whether they have removable cables, and other concerns about durability.
Obviously, no two heads are identical, but we can figure out when headphones aren’t going to be right for you by actually trying them out. We check for heat buildup, how much force is put on your head, and many other considerations. We never review a set of headphones without using them for hours on end.
Sometimes headphones aren’t amazing, but they satisfy a need extremely well. Not only so we identify the main target for every model of headphones we review, but we also weigh its value. That could mean how much you’re getting for how little money spent, or it offers a truly unique feature not seen anywhere else.
Je ne sais quois
If there’s a feature or product that is so different from what we cover, we will do our best to develop and add testing for it. However, that’s not always possible. In the event that happens, we will usually put it into a real world situation, and kick the tires there.
We’re constantly re-evaluating how we do things here at SoundGuys, and are always looking to upgrade our equipment, coverage, and education resources. We don’t chase numbers, we chase the truth. You can read our ethics statement for more.