Speaker evaluation joins DXOMARK’s testing suite

DXOMARK is proud to announce today the expansion of its sound-quality testing to speakers, in particular wireless speakers, with its newest protocol — DXOMARK Speaker. It is the audio team’s first foray into a new category of audio product testing since DXOMARK started evaluating smartphone audio playback and recording a year ago.

Ever since Amazon released its Echo device four years ago, the world of wireless speakers has grown rapidly. New devices are constantly being released and more brands are entering this very exciting market. These standalone speakers go by various names like home speaker and smart speaker. But the main thing they all have in common is that their playback relies on wireless connectivity.

In addition to playing music from a smartphone, tablet, or computer, many of them also can “interact” with the user by responding to voice commands or answering users’ random questions. (DXOMARK’s protocol, however, does not yet test this aspect.)

A unique testing method

Our Speaker protocol tests the important attributes in sound quality such as timbre, spatial, dynamics, volume, and artifacts. We also take into account the speaker’s playback ability to adapt to its environment, such as the intensity of reverberation in a kitchen as opposed to a living room. What makes our evaluation unique is that we also test speakers in eight real-life use cases under various conditions. These use cases take into account the room the speaker is placed in, and how it is placed in the room—whether it is centered, close to a wall, or in a corner. Other factors include the room’s ambient sound, the content you plan on listening to —a movie, podcast, or music —as well as how loud you like the content. And last but not least, where the listener is in relation to the speaker. (Read the specifics about how we test speakers by clicking here.)

DXOMARK also developed its own proprietary music to test speaker sound quality. These challenging music clips of various genres condense many audio cues in short tracks. You can read all about what went into the creation of these sound tracks, the composers and musicians behind them, and even listen to clips of the wide variety of music genres made for the speaker protocol in our article, “Introducing DXOMARK’s new testing playlist.”

For the debut of our Speaker rankings, we carefully chose a variety of 12 devices from 11 brands from different regions. We chose some devices that are globally available such as the Amazon Echo, as well as some that are available only in certain parts of the world, such as the Xiaomi XiaoAI speakers, Baidu Xiaodu Smart Speaker, and the TMall Genie X5, which can only be purchased in Asia. (Another speaker, the Yandex Station, is available mainly in Russia.)

To further add perspective to our first selection, we divided the speakers into two price categories that we call Essential (< 200€) and Advanced (200€ to 499€). Soon we will add a Premium category for speakers priced above 499€. Our segmentation is primarily based on manufacturers’ suggested retail prices at time of release in Europe (prices from other regions are converted to euros for consistent comparisons).

Reviewing the results

Regardless of the price category a speaker falls in, we evaluate all devices using the same DXOMARK Speaker protocol. The results are in, so let’s take a look at what our testers found.

Essential segment (< 200€)

Advanced segment (200€ to 499€)

With an overall DXOMARK Speaker score of 147, the Harman Kardon Citation 200, in our Advanced category, took the No. 1 spot in our first ranking of wireless speakers, with the Google Home Max closely behind at 145. In the Essential category, the Amazon Echo Studio claimed the top spot with an overall score of 124.

When it came to the eight use cases we use to evaluate speakers—Cooking, Bathroom, Relaxing, Party, Bedroom, Gatherings, Watching TV, and Outdoor—two speakers consistently came out on top: the Harman Kardon Citation 200 and the Google Home Max. These two speakers ended up testing as most suitable in five out of eight use cases: Relaxing, Bathroom, Party, Watching TV, and Bedroom. They were also the brands for which music had the best rendered timbre at low volume.

For gatherings, the Bose Home Speaker 500 was best for its omnidirectivity and for maintaining sound properties. Huawei’s Sound X also performed well in this category. The Apple HomePod came out on top in the cooking use case, helped by the device’s extensive algorithms that correct the difficult listening conditions present in a kitchen.

Only two battery-powered devices, the Harman Kardon Citation 200 and the Xiaomi XiaoAI Art Battery, received bonus points relative to the sound quality produced when running only on battery, such as in an outdoor scenario.

Some speakers performed better with certain music genres. For example, the Google Home Max and Sonos One performed well with classical music, while the Bose Home Speaker 500 and the Amazon Echo Studio stood out when playing jazz tunes. For rock music, the Harman Kardon Citation 200 and the Google Home Max outperformed the other speakers tested. Curiously, the two Xiaomi speakers in the Essential category performed on opposite sides of the spectrum when it came to music for meditation, with the Xiaomi Art’s playback topping the list in this genre.

When looking at the scores from a regional point of view, we found that Chinese-brand speakers performed well for their price segment, in particular the two Xiaomi speakers and the TMall Genie X5. Both these brands’ speakers were in our Essential category, and we were impressed with their performance relative to their price category.

Another observation from our testers was that while all speakers aimed to achieve a high degree of wideness, some advanced speakers seemed to have an unusually limited spatial reach, as if directing the sound toward one listener instead of a room full of listeners. The Harman Kardon, the Sonos, and the Google stood out in that regard, providing perhaps some insight into the kinds of listeners and use cases the brands wanted to focus on.

While none of these speakers can compete with the perceived wideness of the traditional two-speaker hi-fi system, most bring innovative solutions to reproduce the stereo soundstage. Only the Amazon Echo Studio and the Google Home Max achieved perceived wideness, but did so in very different ways.

(Note that speakers from the Chinese brands as well as the Amazon Echo Studio do not allow users to play their local content via wifi; uses can play only locally saved music via a Bluetooth connection or a streaming service.)

So whether you want to blast music in your living room, listen to a recipe while preparing a meal in the kitchen, or fall asleep to soothing music, check our wireless speaker reviews and let DXOMARK guide you to a device that perfectly suits your needs.

Videos