DXOMARK Camera version 4: What does it mean for the rankings?

The latest version 4 of the DXOMARK Camera test protocol comes with two major updates: we added tests for image quality in the camera preview, which gives you a very first impression of image quality and can have a big impact on image results as well, and what we call trustability. Trustability is designed to make our Camera test protocol more exhaustive and challenging. It brings more use cases and an updated scoring system for increased user relevance. Another more minor change features an improved testing methodology for autofocus that includes new objective lab tests in low light, for handheld shooting, and under HDR conditions. You can find in-depth information about the changes in DXOMARK Camera version 4 in this article.

In addition, we have made changes to the way our sub-scores are presented. The new preview score has been added to the Photo category. We have also removed the zoom and wide sub-scores from Photo to create a new Zoom sub-score that is on the same level as Photo and Video. The previous zoom attribute has been renamed “tele” to reflect that it is looking at telephoto zoom (zooming in), but not at wide-angle zoom (zooming out from the standard focal length). You can see the new DXOMARK Camera score structure here:

What does all this mean for the DXOMARK Camera ranking? Let’s have a closer look.

For the launch of DXOMARK Camera version 4, we have updated the reviews of ten devices that we had already tested under the previous version of the test protocol:

• Xiaomi Mi 10 Pro
• Samsung Galaxy A50
• Samsung Galaxy Note 10+ 5G
• Sony Xperia 1
• Oppo Find X2 Pro
• Apple iPhone 11 Pro Max
• Huawei P40 Pro
• Google Pixel 4
• Samsung Galaxy S20 Ultra
• Xiaomi Mi 10 Ultra 

We have also published a review for the Oppo Find X2 Neo which had not previously been published.

We are looking to retest and update scores for most devices in our ranking, but for now you’ll find both on our dxomark.com website — devices that were tested under Camera version 4 and devices tested under the previous protocol. Please keep in mind that the two scores are not directly comparable, but the score and rank of a device tested under the old test protocol will still give you a very good indication of its camera performance compared to the competition.

Overall score

This graph shows the ranking of the eleven (re)tested devices, and we can see that the devices that were at the top under the old test protocol, the Xiaomi Mi 10 Ultra and the Huawei P40 Pro, still occupy the number one and two spots, thanks to their most consistent performance overall.

This graph shows the changes in overall score between version 3 and version 4 of the test protocol. The Apple iPhone 11 Pro Max has made the biggest jump forward, thanks to the best image preview we have tested to date and an very consistent performance across all test conditions for Photo and Video alike, earning it points for trustability.

Photo score

The situation is similar for the Photo score, with the Apple iPhone 11 Pro Max again making the biggest gains.

The changes for Photo can be generally explained by three main factors:

• The restructured scoring system (see above: zoom and wide are now combined in the Zoom sub-score, which is on par now with Photo and Video) means that devices with a very good tele-zooms lose some points in the Photo score.
• Preview is a completely new attribute and has been included in the Photo score.
• The newly introduced Trustability concept is better at revealing flaws, and camera failures have a more significant impact on the scores now than before.

Trustability for Photo brings a large range of new test scenes that help test the consistency of image quality across a wide variety of shooting situations. The new scenes include more portraits, cover all types of skin tones, and put more emphasis on face exposure as well as skin tones. Group portraits with subjects in different planes should have good sharpness on all faces.

We have also included more low-light and high dynamic range scenes, including some with motion that can reveal fusion artifacts such as ghosting or halos. For example, the Oppo Find X2 Pro and the Samsung Galaxy S20 Ultra lost a significant number of points for these artifacts under our new test protocol. The Apple iPhone 11 Pro Max, on the other hand, is very good at freezing the motion, but on the downside shows strong noise on moving subjects in such scenes.

New autofocus tests under HDR conditions have also resulted in some changes to Photo scores. For example, the Apple iPhone 11 Pro Max autofocus locks on consistently without delay under HDR lighting. In contrast, the Huawei P40 Pro autofocus is slower to react under HDR lighting than in low-contrast conditions.

Preview

As mentioned above, Preview is a brand-new attribute in DXOMARK Camera v4 and feeds into the Photo sub-score. Among the devices we have tested for the launch of the new protocol, the Apple iPhone 11 Pro Max is the clear winner in this category. HDR processing and bokeh simulation are not rendered perfectly in the preview image, but are still very close to what we can see in the captured image. Zooming is very smooth, too.

The Xiaomi Mi10 Ultra and the Samsung Galaxy S20 Ultra lose points for not offering a live HDR preview, but they perform well in zoom preview. Google’s Pixel 4 does not apply any bokeh simulation at all to its preview images and therefore finds itself in the bottom half of our new ranking. The Huawei P40 Pro’s performance in Preview is a little disappointing, too: the differences between preview and captured image are significant and the preview image is quite unstable when zooming.

At the bottom end of the scale, the Sony Xperia 1 and the Samsung Galaxy A50 are limited by the lack of zoom range and the image quality of the output photo. The preview score takes into account both the image quality of the preview and the similarity between preview and final rendering. So if, as in the case of the Sony and the Samsung, the preview image is close to the final image quality, but final image quality is low, the result is a low preview score.

Zoom score

The Zoom sub-score is new in DXOMARK Camera version 4. As mentioned earlier in this article, the previous zoom and wide scores were extracted from the Photo sub-score to form the new Zoom sub-score that covers both ultra-wide and tele-zoom performance.

The new Zoom ranking is topped by two devices from Chinese manufacturer Xiaomi. Both the Mi 10 Ultra and Mi 10 Pro offer excellent image quality for tele-zoom and ultra-wide, but the Ultra model has the edge, thanks to its better ultra-wide camera. The Huawei P40 Pro and Oppo Find X2 Pro also benefit from very good tele-zoom performance. In contrast, the Samsung S20 Ultra achieves its high ranking thanks to one of the best ultra-wide cameras we have tested to date.

Video score

When retested with our updated test protocol, the Huawei P40 Pro and Xiaomi Mi 10 Pro are still the top devices for video, thanks to very good video output across all quality attributes and test situations. The Mi 10 Ultra follows right in the top duo’s slipstream. It is capable of producing very smooth 4K clips at a 60 fps frame rate but can sometimes struggle with focus on faces.

The Apple iPhone 11 Pro Max follows closely with high-quality videos but is slightly penalized for sometimes unstable exposure and high noise levels when shooting in low light. The Google Pixel 4 still does pretty well, especially for a device that has been tested at 1080p Full HD mode, thanks to its adaptive frame rates and overall well-balanced image quality. Both the Sony Xperia 1 and Samsung Galaxy A50 suffer from autofocus issues and relatively low levels of detail in the tested Full HD mode (note that we previously tested devices sing the original video settings).

This graph shows the changes in Video scores between version 3 and the new version 4 of our test protocol. Just like for Photo, the Apple iPhone Pro Max is the biggest winner, thanks to an overall very consistent and balanced video performance with only some minor exposure instabilities. The Huawei P40 Pro also performed very well and increased its Video score based on very good sub-scores in all areas, particularly excellent rendering of fine detail. Noise and artifacts in low-light recordings were the only real points of criticism.

In contrast, the Xiaomi Mi 10 Ultra and the Oppo Find X2 lost some ground. The Xiaomi is capable of capturing high levels of detail and very smooth footage, thanks to 60 fps recording, but our updated test protocol revealed focus failures on faces and a limited dynamic range. The Oppo’s video clips offer stable exposure in most conditions and a good stabilization system, but skin tones can be unnatural and faces can be underexposed in backlit conditions. We also observed some focus failures and oscillations.

We hope you found this overview of the new DXOMARK Camera version 4 results useful. For the full picture, head over to our smartphone section to see all the reviews, and make sure you don’t miss any of our upcoming review updates.