Expanding the DXOMARK Camera protocol: The Night score

Photographing a city or capturing a flash portrait at night can result in great images, but also poses some unique challenges to the photographer. Shooting in low light traditionally required a camera with a large sensor, long shutter times, and a tripod, or high ISO settings and ideally some manual fine-tuning of camera settings for optimal results.

Smartphone cameras were far from the ideal tool for night photography for a long time. Small image sensors meant the cameras had limited light-gathering capabilities, resulting in often very noisy and/or blurry night shots. However, over several generations, the low-light capabilities of smartphone cameras have increased dramatically. Thanks to a combination of advances in sensor design, faster lenses, and computational imaging methods, current high-end phones can achieve more than acceptable results in low-light conditions.

To reflect and evaluate those technological advancements, we decided to add a Night score to our DXOMARK smartphone camera test protocol. We were already testing cameras in low-light conditions and with flash; however, most of our previous low-light tests were lab-based. Now we test using a variety of real-life low-light scenes that cover such use cases as night cityscapes, low-light portraits, low-light street photography, flash shots, and shots in extreme low light. If available, we also test the device’s night mode. Read on to find out more.

Night test conditions

To evaluate the low-light capabilities of smartphone cameras, we use a variety of test scenes designed to cover most typical low light use cases. We capture all scenes with flash on, flash off, and flash auto settings. We use two landscape scenes to test the dedicated night mode if available, and in addition, we test the flash in our current lab setup as before. You can see below a selection of typical use cases that we cover for our Night test protocol. (Please note that these images are illustrations for use cases and not our actual test scenes.)

Night image quality evaluation

We analyze all the same image quality attributes for low-light images as we do for our general image quality assessment — exposure, color, texture, noise, artifacts, and autofocus performance. However, we have to evaluate some of these attributes slightly differently in a low-light context.

Exposure

We would expect a modified exposure strategy in very low light in order to transmit the mood of a dimly-lit room or outdoor scene. Target exposure should be fairly dark to create an image that is close to the real-life viewing experience — for example, the camera’s auto exposure system should render night skies quite dark and not brighten them too much in order to maintain the scene’s atmosphere.

Dynamic range is even more important In many types of night scenes than in day scenes — for example, in nocturnal cityscapes. The contrasts between the illuminated parts of the scene and the dark areas is very high, which can easily result in highlight and/or shadow clipping. Cameras with a very wide dynamic range will be able to achieve more natural-looking results in night scenes that include brightly illuminated elements.

In the image below, the Google Pixel 3 has handled the typical city night scene well. It did not brighten the sky compared to the original scene, and exposure on the building is very good, with only minor areas of highlight clipping in brightly-lit areas.

Smartphone cameras tend to select very slow shutter speeds in very dark conditions. This strategy can work well for static scenes, such as city or landscape images, thanks partly to efficient stabilization systems. However, moving subjects in the scene will almost inevitably be blurry. In our new protocol, we evaluate how well a camera deals with motion in the scene — that is, if it can detect it and increase the shutter speed accordingly to avoid it.

In very low light, smartphone cameras often underexpose the image—sometimes quite —because it has to maintain a balance between exposure on the one hand and noise and other artifacts on the other hand. A dark image can always be brightened, but that comes at a cost. Lifting the exposure will reveal luminance and chroma noise and also make other artifacts more visible, as can be seen in the two samples below. The image on the left, straight out of the camera, is noticeably underexposed and very dark. On the right, we can see the same image after we have altered the tone curve in post-production in order to brighten the exposure.

Different manufacturers have different views on exactly where the sweet spot is, but we consider in our testing that faces should at least be recognizable, even in underexposed images. A noisy but recognizable face will score higher than an image that is too dark to see anything. This said, noise and artifacts have to be kept within acceptable limits so as not to lose any points.

Noise versus detail

Once exposure has been dialed in, the camera has to find a good balance between noise and detail. The levels of image noise in an image depend on many factors, including the pixel size of the image sensor and the image signal processor (ISP), but tuning the image pipeline is very important, too.

For each scene, the camera needs to find a combination of ISO and exposure time that offers an acceptable trade-off between image noise and motion blur. It’s possible to apply noise reduction after capture, but this will have a detrimental effect on textures and fine detail. Noise and texture performance, as well as tuning strategies, vary a lot between manufacturers, as you can see in these four sample images below.

Overall, it is fair to say that for current smartphone camera technology, the level of detail in low-light images is still quite low when compared to pictures taken in bright conditions. Low-contrast detail in particular, and fine textures such as distant vegetation, are often completely blurred away by noise reduction.

Color

Completely neutral colors are considered a good result in daylight scenes. However, a completely neutral color balance in night scenes often looks unnatural, as artificial illumination tends to produce slight but noticeable color casts. Slightly warm color casts tend to work well for low-light scenes under most light conditions, and it’s also important to maintain good color saturation in low-light conditions. At the other end of the spectrum, many devices produce night images that are too yellow, which is also not acceptable.

In the example below, you can see that the color response of the three different smartphone cameras is very different. In this complex scene, the models in the foreground are illuminated by sodium lamps. In the background we can see light from white Xenon lamps and also from some red and blue fluorescent lamps.

The Google Pixel 3 used its dedicated Night mode to achieve the most pleasing result. It goes for a color rendering compromise that leaves the skin tones slightly yellow, with the street having pronounced blue and pinkish casts. The other devices opted for a more neutral color on the street, which resulted in very yellow skin tones. It’s fair to say that in this situation it makes more sense to prioritize skin tones over the street.

Flash

As ever, we undertake most of our testing using a device’s default camera settings. One exception was the Flash test, where we manually forced the flash on. We now integrate the previous “Flash-on” test results into the Night score, and in addition, we test camera performance for “Flash-auto” settings.

When testing flash in auto mode, we don’t evaluate the actual activation of the flash—what counts is the end result. When the flash fires, the foreground subject should be well-exposed and show good detail, but the background is likely to be heavily underexposed. If the camera doesn’t use the LED flash illuminate the subject, the subject may be slightly underexposed or show less detail, but more background detail should be visible. It’s up to the manufacturer to find the best compromise and determine which option is best for which type of scene. Best results should show good exposure, color, and detail in both foreground and background.

In the scene below, you can see that the Google Pixel 3 is capable of producing a good image with both flash-on and flash-off settings. In the flash-off image, for which we used Night mode, it rendered the subject slightly darker than in the flash-on image, and there’s more emphasis on the background. Both images work in their own way, however.

Scores

The Night score replaces the previous Flash score in our scoring system, as we have now integrated flash-on into our Night testing. To create our Night score, our image quality experts evaluate all image attributes (color, texture, noise, etc.) for all test images. The results are then synthesized into sub-scores for each shooting mode (on/off/auto flash and night mode).

Given that Night mode is not available on all models yet, we decided to make it count like a bonus. If Night mode results are actually better than images captured in standard mode, it can improve the overall Night score, but a bad performance in Night mode, or the lack of a Night Mode, won’t drag the Night score down (though we will certainly talk about the mode’s performance).

Finally, we use a weighting system to compute the overall Night score of the device, which is a sub-score of the DXOMARK Camera score, just like Zoom and Bokeh.