DxOMark Mobile test protocol and scores

The DxOMark Mobile test protocol takes a different approach to our sensor and lens test protocols. Rather than looking at a discrete component (sensor, lens) the Mobile protocol assesses the performance of the imaging pipeline in its entirety, including lens, sensor, camera control, and image processing. This means that still image quality evaluation in the Mobile protocol is based on the camera’s JPG output rather than on RAW image data as for sensors and lenses. The protocol also covers the test device’s performance in video mode, and scoring is calculated from both objective measurements and from perceptual evaluation of more than 50 challenging and realistic indoor and outdoor scenes by a panel of imaging experts.

Testing for most image quality criteria is undertaken with the smartphone camera in its default (auto) mode. We intervene manually only to activate the bokeh (portrait) mode, the flash, and the zoom feature. In video mode we use the resolution setting and frame rate that provides the best video quality. For example, a smartphone might offer a 4K video mode but use 1080p Full-HD by default. In these circumstances we will select 4K resolution manually.

DxOMark Mobile: What’s new for 2017?

With the mobile sector being a major driver of innovation in imaging, and smartphone camera technology rapidly evolving since the original DxOMark Mobile test protocol launched in 2012, the time has come for an update. The new version of DxOMark mobile takes into account the latest developments in mobile imaging technology and adds the following components:

• The new zoom test analyzes image output at three subject distances and in several test situations (outdoors, indoors, and in the lab). Key criteria for judging the results are resolution and image detail, noise, color reproduction, and image artifacts.
• A new bokeh test analyzes the camera’s ability to create a pleasing background blur while keeping the image’s main subject in focus. The test evaluates the strength, smoothness, and transition of the blur effect. It also looks at distortion on portrait subjects, artifacts, and the image processing’s depth estimation capabilities.
• When testing the autofocus and sharpness, the updated test protocol now takes into account motion in the scene and the speed of image capture. Sharpness on moving subjects with a moving photographer is measured in a simulation setup in the lab. A new autofocus test evaluates not only accuracy but also speed and texture, and we now undertake noise tests shooting handheld as well as shooting with the device mounted on a tripod.
• For the updated low-light analysis, we take still image samples at very low light levels between 1 and 20 Lux. For our video quality analysis, we record all low-light footage in between 5 and 20 Lux. Overall light levels for most photo tests now range from 1 lux to 1000 lux, and color temperature from 2300 to 6504 Kelvin.
• We have expanded our DxOMark Mobile video analysis to include objective measurements in addition to perceptual evaluation for most test criteria. For many tests — for example, white balance, exposure, and noise, we now also measure how quickly and smoothly the camera adapts to changing light conditions during recording.

DxOMark Mobile Scores

The DxOMark Mobile Photo and Video sub-scores and the overall score are generated from a combination of objective lab measurements and perceptual analyses. The following test criteria feed into the Photo sub-score:

• Exposure and contrast, including dynamic range, exposure repeatability, and contrast
• Color, including saturation and hue, white balance, white balance repeatability, and color shading
• Texture and noise
• Autofocus, including AF speed and repeatability
• Artifacts, including softness in the frame, distortion, vignetting, chromatic aberrations, ringing, flare, ghosting, aliasing, moiré patterns, and more.
• Flash
• Zoom at several subject distances
• Bokeh

The Video sub-score is calculated from the following criteria sub-scores:

• Exposure
• Color
• Texture and noise
• Autofocus
• Artifacts
• Stabilization

Scores are presented in the format as shown below.

For a better idea of how the updated protocol impacts device scoring in practice, please have a look at our 2012 vs. 2017 score comparisons for a range of devices (including the Google Pixel, Nokia 808 and iPhone 7 Plus) that we tested using both test protocols.