What’s new in DXOMARK’s protocol updates

At DXOMARK, the various smartphone test protocols are updated periodically not only to keep up with the rapid pace of innovation in smartphone technology but also to make them even more relevant to smartphone users by ensuring current user behavior is reflected in the testing methodologies. DXOMARK frequently undertakes surveys and organizing consumer panels to ensure that we fully understand what smartphone users expect from their devices in terms of performance and usability. The results feed directly into our methodology updates.

This latest round of updates brings new elements to all protocols, but mainly to Camera and Audio, which both include new measurements to adjust to the evolution of both technology and device usage. In addition to the new measurements, the following improvements have been made:

• Increased transparency: For all scores, the high-level formula will be available to consult in the appropriate How we test sections on the DXOMARK website.
• Improved readability of the scores: We have modified the score scales in order to make the scores more comparable across protocols. For example, in the past, an overall score of 80 would have been low for Camera but comparatively high for Audio or Display. Scores have now been reformatted in order to ensure that they represent a similar performance level across all test protocols.
• Geometric averages are now used for all protocols bringing increased consistency: This measure is applied to all protocols and rewards a homogenous performance across all sub-tests of a protocol and penalizes devices that are very good in one key test area but perform poorly in another.

New for Camera

For Camera, we have added a range of new use case scores, providing neutral assessments that help consumers make smartphone purchasing decisions based on their device usage. These use case scores are computed from a subset of our measurements and cover Photo, Video, Bokeh and Preview. They are published next to the overall score, which they complement. We are updating our test protocols on a regular basis to reflect changes in user behavior and since our last release two years ago, some habits have changed and new trends have increased. This is reflected in this latest version of our Camera protocol, in which the impact of low light performance, the friends and family use case, zoom and close-up on the overall score have increased.

 Use case scores

Under the new protocol results, in addition to the usual attribute scores, results are presented based on use cases that were prioritized by the participants in our consumer studies. In our tests, we cover many more use cases, but we have selected the most important ones to be presented in detail within our test results:

• Friends & Family: This is a very typical use case and represents more than 50% of the photos and videos captured by the users who took part in the YouGov-DXOMARK survey undertaken in December 2021*. For this use case, smartphone photographers expect realistic and pleasant skin tones, capture at the right moment, natural contrast and face detail, as well as freezing of motion among other image quality characteristics. This sub-score is computed from all data collected for Photo, Video, Preview, and Bokeh tests.
• Low light: This is a very challenging use case and even the users of Ultra-Premium phones don’t find it easy to achieve great image quality in low light. According to the YouGov-DXOMARK survey, it is also a major source of dissatisfaction among users. Only 50% of users are satisfied with the quality of images and video clips that they capture in low light. Low-light images should deliver natural exposure and colors as well as a good trade-off between detail retention and image noise reduction. This trade-off is even more difficult to achieve when recording video footage than with still images. The camera should also be capable of freezing motion in the scene with only minimal motion blur.
  For version 5 of the DXOMARK Camera test protocol, we have expanded and enhanced the low-light analysis. Thanks to a dedicated focus group on the topic, we have been able to take consumer preference into account even more. A lot of effort went into defining what constitutes an acceptable low-light image result. By doing this, we have been able to improve all our lab metrics to match 2022 consumer expectations even better. In addition, Night is not an attribute under the Photo score anymore and has been integrated into the low-light score.
• Outdoor and Indoor: Even though low light is currently the most critical differentiator between smartphone cameras, all newly created test results will also feature scores for Indoor and Outdoor use cases.

New protocol and extension

Zoom score extension for Wide and Tele

Zoom performance is a key differentiator for smartphones in the Premium and Ultra-Premium segments ($600+), with most devices in these categories featuring dedicated tele zoom and ultra-wide cameras. Given that according to the YouGov-DXOMARK survey* 30% of users use the zoom function of their smartphones, it’s no surprise that manufacturers are putting a lot of work into improving their devices’ zoom capabilities. The good news for budget-conscious consumers is that the zoom technologies used in today’s top-end devices will no doubt trickle down into lower price segments over time.

A dedicated Zoom score was first introduced to the DXOMARK Camera test protocol with version 4. In addition to lab tests, the DXOMARK engineers perform in-depth perceptual evaluations across more than 35 real-life scenes, covering landscape, cityscape, architecture, interior design, portrait, and still life. Version 5 now brings the following enhancements and additions:

• New video zoom evaluation in the lab: DXOMARK is now evaluating video zoom quality from ultra-wide to tele
• Low light and very low light: Zoom tests are now also undertaken at a very low light level of 5 lux
• On-screen zoom buttons tested: We are now testing the image quality provided with the zoom buttons of the device’s camera app UI
• Enhanced lab testing: We have increased the number of zoom measurements in our perceptual scenes. They now include measurements (target exposure and white balance) and AI metrics (detail preservation and noise) on the DXOMARK Chart (DMC), repeatability across a series of shots (including defocusing of the camera after each shot), new video measurements (pinch-zooming before and during recording) as well as perceptual evaluation of zoom smoothness. Zoom smoothness is now also evaluated for two different pinch-zoom actions (from ultra-wide to 72mm and 144mm equivalent respectively).

Close-up

Close-up or macro photography of objects and details that are not visible to the human eye has been around as a photographic discipline for a long time, but in the past this type of photography typically required interchangeable lens cameras with dedicated macro lenses. More recently we are seeing more smartphone cameras with close-up functionality, and the fact that the feature is already used by 7% of users* indicates that interest in macro is growing, which is why close-up testing has been added to the DXOMARK Camera protocol.

Close-up testing is added as a bonus to the Photo score. This means a device can score a bonus if it comes with this feature, but no points will be deducted from the score of devices without a close-up feature. New close-up related tests include the following:

• Document scanning scene: An official document, such as a passport or national identity card is photographed in three different ways to cover most scanning scenarios. It is first photographed at full size and at a high level of magnification. The objective of this test is to check whether the camera can capture enough small detail in two ways, by moving the camera lens very close to the document, or by staying at the original distance and using the zoom in order to achieve maximum detail.
• Three additional close-up scenes: If the camera captures a sufficiently high level of detail in at least one of the two small detail document scanning tests described above, the DXOMARK testers shoot a total of three additional scenes that are subsequently evaluated for image quality attributes, with a focus on detail preservation. These additional tests focus on still-life macro photography.

Friends & Family use case: Measurements in HDR conditions and for moving subjects

Smartphone users take photographs of friends and family in all kinds of conditions, including difficult backlighting and other high-contrast conditions, with or without subjects in motion. With version 5 of the Camera test protocol, we have expanded our HDR camera testing to include a range of new measurements, including the following:

• Portrait HDR lab setup: The HDR portrait setup has been drastically improved by adding a new AI measurement for face detail as well as objective measurements for dynamic range and contrast. As usual, the objective measurements are complemented by perceptual evaluation. These new additions allow for a more exhaustive and complete evaluation of texture in portraits across various light conditions.
• Evaluation of scenes with motion: Freezing motion in a scene can be a difficult task for a camera in any light condition, but it is particularly tricky in high-contrast scenes where complex algorithms, such as HDR or motion detection, kick in. To better evaluate this kind of scene, we have added a motion blur measurement to our lab testing. Overall testing has been expanded to a total of 24 light conditions.

New Camera version 5 results

The new version of the Camera protocol does not completely reshuffle our current rankings. Camera V5 is simply a natural and more profound evolution of our Camera version 4 protocol. A smartphone camera that earned a high rank under V4 will still be well-ranked under the new protocol. But to reflect the latest in-depth additions to our testing, some adjustments had to be made, as seen here:

To give you a better idea of what to expect from the protocol updates described above, in this section we give you an overview of the test new test results for two popular devices: the Apple iPhone 13 Pro Max and the Samsung Galaxy S22 Ultra (Snapdragon). We also explain how the update affects the version 5 overall scores.

Apple iPhone 13 Pro Max

Close-up: The previous generation of the top-of-the-range iPhone is still one of the best devices we have for close-up photography to date. Image results are very well balanced, with pretty high levels of detail. Some artifacts are visible but all within acceptable limits.

Friends and family: It’s also one of the best performers in the friends and family use case. The new expanded analysis offers a lot of additional information, and our testers found the iPhone 13 Pro Max freezes motion nicely, controls low contrast issues well and delivers good skin tone rendering in all light conditions.

Low light: The iPhone does not do as well for the low light use case as it does for family and friends, blocking the Apple flagship from taking the top spot in the new Camera version 5 ranking.

Samsung Galaxy S22 Ultra (Snapdragon)

Close-up: The Samsung Galaxy S22 Ultra is a good device for this category. Both the Exynos and Snapdragon versions capture high-quality close-up images that are quite similar to the iPhone 13 Pro Max output, but with less detail.

Friends and family: This use case score is not as high as on some other top-tier phones. Our testers measured longer shutter delays in various  HDR conditions and in low light. This could be problematic for photographers who rely on instant capture.

Zoom: The new low light elements in our Zoom testing reveal interesting strong points of the Samsung. This is true for zooming into both directions, to ultra-wide and tele. In our testing, the S22 Ultra demonstrated great robustness and one of the most future-proof designs, earning it extra points for the Zoom and overall scores.

Image quality aspects making a stronger impact in Camera version 5

Contrast: It’s important for contrast to be well preserved, especially on portraits and backlit scenes. Consumer feedback studies on the topic show that consumers do not like low contrast on portrait shots. With the current evolution of the HDR technology, contrast becomes even more important as image quality is increasing overall. With version 5 of the DXOMARK Camera test protocol, we expand contrast evaluation with new measurements (local contrast indicator) and new perceptual analysis guidelines based on consumer feedback. As a result, the scores of some devices, for example, the Xiaomi Mi11 Ultra and 12S Ultra or the Huawei P50 Pro, will be slightly impacted by low contrast rendering.

Motion in the scene: Capturing the right instant and freezing motion in the scene are important capabilities in photography. Our updated testing shows that some devices are clearly better than others in this respect. For example, the Honor Magic4 Ultimate does very well in this respect under indoor lighting, while other phones, such as the Huawei P50 Pro or iPhone 13 Pro Max struggle more with detail rendering on moving subjects.

Low light and night “atmosphere”: Exposure is a crucial element of low-light photography. Our focus group results show that viewers do not like low light or night pictures that are overexposed and therefore do not transmit a low-light atmosphere. We have therefore updated our evaluation guidelines for target exposure in extreme low-light conditions. In testing, some devices deliver a bright target exposure in low light and very low light conditions, which goes against users’ preferences. These devices, for example, the Huawei P50 Pro, Xiaomi 12S Ultra and Apple iPhone 13 Pro Max, will lose some points with the updated scoring system.

New for Audio

Tests and measurements

The overall structure of the DXOMARK Audio score remains unchanged, with sub-scores for Playback and Recording. However, two new tests, Wind Noise and Audio Zoom, have been added to evaluate the performance of more advanced smartphone audio features.

Wind noise

Wind noise, which can ruin an otherwise perfectly fine audio recording, has been identified as the most frustrating issue in smartphone audio recording. A brand new setup in the DXOMARK Audio lab measures the impact of wind noise on various audio recordings, at a range of angles and wind speeds.

Within the Audio score structure, wind noise measurements can add a bonus or malus to the Friends & Family videos, Selfie videos and Memos use case scores. Within the test results that are published on the DXOMARK website, consumers will be able to compare wind noise track recordings from multiple devices.

Audio zoom

Audio zoom is a smartphone audio feature that is becoming more popular. It allows users to record sound that is coming from a specific direction and attenuates the incoming sounds from other directions. This is usually controlled dynamically, for example in conjunction with the camera’s tele zoom.

Audio zoom testing includes both objective and perceptual measurements and is undertaken on a brand new setup that allows for wide, tele, and super tele zooming. In the test results, the audio zoom measurements feed into the directivity measurements.

Custom audio tracks for precise evaluation

Most testers and reviewers of audio equipment use testing playlists to listen for specific audio cues. These custom-created sets of songs help highlight audio attributes, such as dynamics, distortion, tonal balance, low-end extension and spatial wideness, when played back on speakers or headphones. In 2020, the DXOMARK Audio team commissioned several musicians to compose custom tracks that are perfectly tailored for our wireless speaker testing, containing all the sound attributes required by the DXOMARK Audio protocol. The 10 tracks that were composed are now being used for the evaluation of the smartphone audio, allowing for even more precise and comprehensive testing.

 Audio version 2 results

Like for the other protocols, the new Audio score formula rewards devices with a balanced performance and stretches the scale for better differentiation between performance levels. The best-performing devices for audio zoom and wind noise receive bonus points.

In the updated audio ranking, the Black Shark 5 Pro remains the king in every way. It performs especially well in the new wind noise evaluation. Asus and Google devices move up in the ranking, thanks to overall good performance, so does the iPhone 13 Pro Max, thanks to a great audio zoom feature. The Samsung Galaxy S22 Ultra provides excellent wind noise reduction and therefore gains some spots in the ranking as well.

Refreshing the other DXOMARK test protocols

Even though the Camera and Audio test protocols are the ones that feature new or updated tests and other modifications, this latest update also affects all other DXOMARK smartphone test protocols — Selfie, Display, and Battery. For the first time, the general formulas for all protocols are now available on the DXOMARK website; score scales are now comparable between test protocols; and a new geometric score formula rewards a homogenous performance across all sub-tests. This is not where it ends, though. At DXOMARK, we will continue to optimize our testing methodologies for new technology developments and changes in user behavior, so stay tuned for future updates.

 

*YouGov RealTime survey conducted on behalf of DXOMARK from December 17 to 23, 2021, among 2,000 people per country, representative of the national population aged 18 and over (France, Great Britain, USA), the urban population for India and the online population for China, using the quota method.