We're grading firmware updates, but we need your help

In the last two years, we’ve seen manufacturers struggle a bit with firmware updates. Because headphones are now commonly tied to apps and smartphones, it’s not uncommon to see firmware updates that significantly impact function without much warning. This has led to some enshittification, as well as some abrupt updates that cause a furor with users rightfully upset by the products they own no longer working the way they used to.

As we like to try to give you as much information as possible, we sometimes find that re-testing everything every time a firmware update is a lot of effort for little return. So instead, we’re going to cover updates by quickly grading their effects when possible. This will also help us determine when to re-test, and when to simply note the update without breaking our backs to get new data when it isn’t necessary.

Our current grading criteria:

To make this as fair — and informative — as possible, here’s how we’re proposing to judge firmware updates when we cover them. We will look at five metrics, using modified Likert scales not only to assign grades to these firmware updates but also to poll readers about their feelings toward the updates. Conveniently, a 5-point scale roughly translates to typical American report card grades. Here’s how that looks:

Bugs

Does this update fix issues or introduce new ones?

1. Causes major issues.
2. Causes minor issues.
3. Doesn’t affect anything at all.
4. Fixes minor issues.
5. Fixes major issues.

Performance

Does this make the product perform better or worse?

1. Strongly reduces performance.
2. Reduces performance.
3. Neither reduces nor enhances performance.
4. Enhances performance.
5. Strongly enhances performance.

Added effort

Is the total mental and physical effort (interaction cost) required to perform basic operations greater or less than that of the previous firmware?

1. Significantly increases interaction cost
2. Somewhat increases interaction cost
3. Neither increases nor decreases interaction cost
4. Somewhat decreases interaction cost
5. Significantly decreases interaction cost

Informativeness

Do the update notes effectively communicate everything it does to the user? Is the user aware of the consequences of installing?

1. Does not describe updates.
2. Somewhat describes user updates.
3. Describes specific updates, without explanations.
4. Describes specific updates, with adequate explanations.
5. Describes specific updates, with excellent explanations.

Consistency

Does this update meet user expectations on how a product should be used/behave? (Does the way you use the product stay the same/does it not require you to relearn anything?)

1. Strongly disagree.
2. Disagree.
3. Neither agree nor disagree.
4. Agree.
5. Strongly agree.

An example

Earlier this year, Nothing released a firmware update that was aimed at fixing a serious bug introduced by a previous update. Here’s how the grading would look for this:

• Bugs: 5 (A). Fixed a major issue with the DSP, causing an unintended tuning change for the worse.
• Performance: 5 (A).  Performance increased due to a bug fix.
• Added effort: 3 (C). Did not change the effort required by the user to operate the headphones.
• Informativeness: 3 (C). The changelog is direct, but scant on details.
• Consistency: 5 (A). Operation of the headphones is unaffected, but no new features have been added.

While the mean score would net this update a 4 (or “B”), it is a strongly positive update. We had briefly considered a scale using the midpoint as neutral, and the extremes as double-minus or double-plus, but we felt that it would be more confusing than helpful. We also want to encourage headphone companies to be more complete with their update notes so that users can understand exactly what’s going on BEFORE installing an update that may break certain features. Additionally, doling out “C” grades for doing nothing creates an incentive — however small — to truly improve user interaction or bugs instead of just claiming something’s better. It also makes an “A” grade difficult unless a company puts a massive effort into its overhaul.

If we were to grade Bose’s recent QuietComfort Ultra (2nd Gen) update, it would score… lower, using these grading criteria.

We need your feedback

Of course, these are merely our best first effort, but we need you to help us learn if this is adequate or not. Though we have access to UX experts, we’d love to hear your thoughts on what we should use to assess firmware updates. What things stick in your craw about updates? Is there a category we should add to our grades? Feel free to comment or open the discussion up online somewhere. Just be sure to let us know that you’re posting about it so we can monitor the discussion for good ideas.