A brief introduction to how we test doorbell cameras

Doorbell cameras are becoming the staple of every connected home because they have become the first line of defense in a home security system. Whether it’s seeing who is at the door when nobody is in the house, or accepting a package delivery remotely, doorbell cameras have become convenient as well as a necessary component in any home surveillance system. The use of doorbell cameras is also an area of home security that is expected to continue to grow.

Strengthened by its expertise in camera image quality evaluation since 2003 (DSLRs, smartphones among others), DXOMARK is now extending its expertise to home surveillance cameras and smart doorbell cameras. The image quality of this new breed of connected devices is of particular interest to DXOMARK. Unlike smartphone or laptop cameras, where the focus is to produce images that are generally pleasing to the eye, the tiny sensors in security and doorbell cameras must focus on producing accurate facial details that would make it possible to identify the person in front of the camera.

DXOMARK has developed a new protocol that adapts its already stringent image-quality testing methods to evaluate doorbell cameras. It is the latest to join the company’s extensive family of protocols in smartphone camera, audio, display, and battery as well as video conferencing, wireless speakers, camera sensors, and lenses.
Currently focusing only on image quality, the setup is built based on the same unique combination of lab measurements and tests in real-life/natural scenes that have made DXOMARK the standard of image quality in the smartphone industry.

Test setup

Use cases are the basis of any DXOMARK protocol because they determine the types of lab setups, the real-life scenes to be shot, and the list of quality attributes to evaluate.

The scenes that we reproduce in the laboratory and outdoors are :
• An outdoor daylight scene to test the intrinsic quality of the camera (combination of lab and outdoor measurements)
• A backlit scene with a forefront in the shadow and a background in the sun, to evaluate the HDR capability of the camera (combination of lab and outdoor measurements)
• A night scene, to either test the infrared (IR) mode or to test under a floodlight-helped color mode

Field of view

The field of view is the cornerstone specification sheet of a doorbell or a security camera. Unlike most consumer cameras, security cameras claim their ability to cover an angle of up to 180°. However, in the specific context of doorbells, there isn’t an industry-level consensus yet on the right approach to field of view. Should the video format, for example, be square, or portrait?

Because of the diversity of the existing solutions, DXOMARK’s protocol tests each doorbell camera at its default field-of-view setting, respecting each manufacturer’s choices. However, we evaluate image quality attributes on every aspect of the image, including distortion artifacts. A doorbell model that claims to have a very wide viewing field but delivers a poor quality image would be impacted more in our evaluations than a device that had a more conservative field of view but guaranteed a higher-quality image — one that would allow recognition of the person in any part of the image.

The evaluation: Image quality attributes

We perform two kinds of evaluations on videos: objective and perceptual. Objective tests focus on testing the standards established by the industries, such as white balance and texture level, while perceptual evaluations are the quantification of qualitative parameters. Perceptual evaluation takes its roots in long-established scientific methods, also described in standards, like the International Telecommunications Union’s subjective video quality assessment methods for multimedia applications (ITU) ITU-T P.910.  Both objective measurements and perceptual evaluations are necessary to assess fully the image quality of a video, as they complement each other and lead to a more enriched assessment.

DXOMARK has developed its doorbell camera protocol around the most important image quality attributes for a security camera: Exposure, Details (texture and noise), and Artifacts, keeping in mind that the main purpose of a doorbell camera is to recognize the face of a person at the door, whether it is a friend or an intruder.

Exposure

Good exposure is crucial to properly identify a face. The DXOMARK doorbell testing protocol evaluates four parameters related to exposure.
• Exposure on the person, which is the amount of light on the individual that allows for their correct identification; when the identification is guaranteed, we then use our expertise from the consumer camera world to evaluate whether or not the representation is also pleasing to the viewer.
• The dynamic range of the camera, i.e., its ability to correctly render the dark areas of the scene as well as the bright areas. In particular, we test here the HDR performance of the doorbell camera.
• The contrast, which is the gradient of differences between dark and light areas on the video. Contrast is especially key in the context of HDR pictures, as a poorly balanced HDR setting can lead to unnatural images, which can in return make the identification of the person difficult for the viewer.
• Exposure adaptation, which is the ability of the camera to adjust, in real-time, the exposure when the lighting conditions suddenly change.

Details, Texture & Noise

Related to the need to identify the person at your door, the measurements related to  details, texture, and noise explore all the elements of the picture related to the clarity of the image:
• Although details can be roughly estimated with a resolution chart, they must also pass real-life scene tests, such as the ability to read the logo on a shirt or even a car license plate.
• Texture evaluates the way the camera depicts the details of facial characteristics such as a beard or skin, as well as surrounding areas such as grass and bushes. DXOMARK has developed a specific perceptual evaluation on realistic mannequins to complement the tests of natural scenes.
• Finally, noise assesses the graininess of the overall image. It is worth noting that here, again, the goal is to identify people, so having a low level of noise is not as important as having a high level of detail.

Artifacts

In this attribute evaluation, we chase mostly three types of artifacts: compression, color fringing, and distortion.
• Compression appears when the doorbell needs to transmit the video to a server, in real-time, forcing a rather high compression level. When the scene changes rapidly, the ISP sometimes fails at keeping up with the pixel count, and a phenomenon known as blocking appears. This is particularly true of battery-based systems.
• Color fringing is a type of chromatic aberration often caused by the failure of the camera at focusing all colors on one point; this is often seen at the edge of a subject, separating the foreground from the background.
• Distortion is very often present on wide-angle lenses of cameras. What matters in the context of doorbells and security cameras is whether the distortion could impede the possibility of identifying the person being filmed. The DXOMARK protocol measures not only the objective distortion but also the impact in real scenes, through its perceptual analysis methodology.

Conclusion

DXOMARK has put several doorbell cameras through its rigorous testing protocol. They cover all price points and regions. All the leaders of the market are represented, including Google Nest, Ring, and Arlo. The results are still being tallied, and we plan to publish the results very soon on dxomark.com!

---