Smartphone portrait photography and skin-tone rendering study: Results and trends

In the summer of 2023, DXOMARK experts conducted their largest study of smartphone portrait photography, focusing on everyday life moments. The study:
● focused on portraits (all varieties of pictures featuring individuals);
● captured 405 scenes with 83 regular consumers as models;
● consisted of a user panel of these models, 30 professional photographers, and 10 DXOMARK image quality experts.

Our goal was to measure user preferences on people’s pictures and identify emerging trends in smartphone portrait photography. In the study, individuals representing a wide range of skin tones were used, which led to a compelling question: Does the perceived quality of images remain consistent across different skin tones?

Read on to learn about our main findings.

Three devices, three renderings

As part of their methodology, DXOMARK experts included in their study the rendering of three premium flagship devices released in late 2022 and 2023, along with those of a professional photographer using a Digital Single-lens Reflex (DSLR) camera. Participants were then asked to identify which image they would not want to post on social media, as a criterion to highlight their level of satisfaction.

The goal of this study was to identify trends in user preferences by obtaining rendering preferences.

The first key finding of the survey, which explained the subsequent results, was the noticeable differences in overall rendering between the three devices. This suggests that each manufacturer has its own distinct visual “signature.”

The manufacturer’s technical choices

We observed significant differences between the photos produced by the devices, even in very basic use cases. This resulted in different Satisfaction Indexes. This was true for both typical outdoor and indoor scenes.

“Satisfaction levels vary widely between the photos, hence the importance of studying trends in terms of user preferences. It also underscores the challenge manufacturers face in creating a unique style while ensuring they deliver a rendering that appeals to the majority of users.”
Hervé Macudzinski, Image Science Director, DXOMARK

 

Here, we observed notable differences in overall brightness, skin color, color rendering, and face exposure. Even in less technically demanding scenes, the manufacturer’s signature clearly influenced the results. Each device also scored a unique Satisfaction Index, underlining their distinct characteristics.

The verdict

The best smartphones are failing to meet user expectations for portrait pictures.

A total of 1,620 photos were taken for this study, and each photo was assigned a Satisfaction Index. A score of 70 or more guarantees a high JOD score and a low rejection rate, indicating that the photo is generally satisfactory to panelists. An index below 70/100 indicates that the photo may not meet user expectations.

The overall Satisfaction Index for all the portrait pictures reviewed was 61.

Interestingly, users had higher expectations for portrait photos in all conditions:

Indoors: of the following indoor shots, respondents preferred the brighter picture taken with Device A (after the photographer’s rendering.) Here, the Satisfaction Index was at 60.

At night: users expect people to stand out. The satisfaction index was 57.

Low-light conditions are not ideal for photography. Even for professional photographers, it was a significant challenge to meet the expectations of the panelists.

Exposure has the greatest impact on satisfaction

The portraits taken with a professional camera received an impressive overall average score of 77. By comparing the professional camera results to the smartphone results, we were able to better understand the trends in user preferences and identify areas of dissatisfaction.

Respondents often expressed dissatisfaction with the overall color rendering and incorrect exposure of faces.

Here are some other takeaways:
● users have strong expectations on level of brightness, skin color, and overall color rendering (as shown in the SDR scenes);
● users strongly penalized underexposure of the face, which had a significant impact on overall satisfaction (as shown in the indoor scenes);
● the most saturated or brightest image was not necessarily the preferred option.

In low-light situations, users expect the resulting photos to be similar to typical indoor photos, often unaware of the impact of low light on photography. This contributes to a greater disparity in user perception.

However, exposure remains a top priority for users and has a significant impact on their satisfaction.

When shooting at night, users want to maintain the ambiance of the scene while ensuring the subject’s face is properly exposed. This can be a challenge even with a professional camera, especially when the subject has a darker skin tone.

“People demonstrate remarkably specific preferences and a keen eye for details. In that context, standard consumer insights (ranking, device comparisons, etc.) are not enough to understand them.”
Hervé Macudzinski, Image Science Director, DXOMARK

The perceived quality of images varies across all skin tones pictured

Photos of people are captured in various conditions, from indoors to outdoors, day or night, sunny or backlit. Despite its popularity, this type of photography is technically challenging for smartphone cameras.
Through our rigorous scientific approach, this new survey provides insight into the factors that influence respondents’ choices. One of these factors is the rendering of the model’s skin tones, revealing image quality issues.

Satisfaction varies depending on age

A total of 123 panelists participated in the survey, divided into subgroups based on gender and age. This allowed us to gain first consumer insights.

Younger consumers (under 40) were more selective when rating portrait pictures than those over 40. They also had a lower overall Satisfaction Index. In particular, there were significant differences in scenes with higher technical complexity, such as low-light, HDR, and backlit scenes.

This suggests that younger people are more sensitive to image quality issues than older people. Face exposure emerged as an area of particular concern for young people.

Given their discerning nature and high demands, satisfying the younger demographic is a key for manufacturers.

Satisfaction varies depending on gender

We also noticed a significant discrepancy in the Satisfaction Index between male and female panelists.

In all conditions, women had higher expectations of image quality than men. The more challenging the conditions in a given scene, the more degraded the image quality, and the women were more adept at recognizing this.

This difference in expectations between genders proved to be the most substantial compared to other subgroups, such as age, cultural heritage, or skin tone.

Satisfaction of every respondent varies depending on the skin tone of the model

A total of 83 models participated in the study, representing a wide range of skin tones.

As previously discussed in our methodology article, we used the Fitzpatrick scale, a widely used classification system for categorizing different skin tones. However, it is important to note the limitations of this scale, as it may not encompass the full spectrum of skin tones.

Still, the survey results clearly show that the presence of people with darker skin tones in photographs consistently correlates with lower levels of satisfaction. Crucially, this finding is not an issue of representation, as it applies to all respondents, including models, photographers and DXOMARK experts. Everyone believes that these pictures are less effective and that smartphones deliver less favorable renderings when the skin tone deviates from white.

Hence, the problem lies in the inadequate rendering of darker skin tones. Other factors contributing to lower satisfaction include incorrect white balance and poor overall exposure for the same scene with a darker-skinned model.
The satisfaction scores declined as skin tone darkens, suggesting that the problem is not exclusively related to darker skin tones, but to skin that is perceived as “not fair”. While light-skinned models are consistently rendered with similar image quality across devices, rendering challenges arise with any non fair skin tone.

Tuning issues are more visible on deeper skin tones

The two photos on the left were taken with the same device. We can see that with a darker skin tone in the same scene, more minor issues were detected that affected the overall Satisfaction Index. An example of this is the underexposure observed in the second picture.

However, the device used for the two photos on the right delivered equal satisfaction for both skin tones. The device on the left failed to provide satisfactory results for the darker skin tone. This is due to lower exposure settings and a lack of adaptation to the skin tone of the person in the scene.

These examples highlight the challenge for smartphones, which must adapt and use different tuning/settings to achieve optimal renderings for all individuals.

A reminder

This survey was not designed to assess device quality, but rather user rendering preferences. The preferred rendering was not always the most “natural” or the one that “accurately rendered” skin tone.

Room for improvement

Satisfaction with outdoor portrait photography was high, yet not flawless. As previously explained, it depends on different criteria, like lighting condition, of course, but also tuning choices by manufacturers.

To provide a basis for comparison, we included pictures taken and edited by professional photographers using DSLRs in the survey. These images represent what an “ideal target image” might look like and what would be considered perfect from the photographer’s point of view. The average Satisfaction Index for the professional pictures was 74. The lowest index, observed in low-light conditions, was 71.

Of the smartphones assessed, only one device achieved an overall score of 71, with a high Satisfaction Index in all lighting conditions. The other two devices received significantly lower scores from our respondents.

Is the smartphone camera just a tool for capturing memories? Far from it. With its embedded technology, such as computational imaging capabilities, the smartphone camera plays the role of the photographer by making decisions on behalf of the user.

Through technological advancements over the years, smartphone cameras have made significant progress in bridging the gap with DSLRs in many ways.
The results indicate that there is a significant need for improvement in low-light, night, and backlit portrait photography, as users were highly dissatisfied with the results. For example, when shooting at night, users are unwilling to compromise between capturing the ambience of the nighttime setting and ensuring that the subject’s face is well exposed.

Photographer satisfaction: A guide to tomorrow’s consumer demands?

Thirty photographers and 10 DXOMARK image quality experts participated in this survey. Although one smartphone received high satisfaction scores, these participants were able to distinguish its photos from those of the photographer rendering.

The photographers’ Satisfaction Index was significantly lower, because they knew exactly what they were looking for in different situations, leading them to be more demanding and reject more pictures than consumers. Also, they have the ability to detect subtle issues, which is different from their expectations as a signature and aesthetic goal.

The professionals had high expectations for all types of scenes and lighting conditions. And their top reasons for rejecting photos were exposure and color rendering.

The disparity between smartphones and photographer renderings was even more pronounced when it came to certain lighting conditions, such as low light and night photography. We found that the more challenging the conditions, the greater the preference for professional rendering.

In summary, the general preference for photographer rendering provides valuable insight into the ideal target rendering that manufacturers should strive to achieve. This knowledge can guide their efforts to meet and exceed consumer expectations in the future.