How We Test Smartphones
See How We Test Other Products:
These days, you can do almost anything from your smartphone, whether it’s texting, banking, catching a ride across town, pursuing Pokémon, or picking out paint colors for your new kitchen. Even though we love our phones, the technology improves so quickly that we often want to buy a new one every 1-2 years.
If you’re a rabid brand loyalist, then there’s no question that you’ll buy the latest version of your favorite phone, but what if you don’t have a preference for either Android or iOS? Here at Reviewed, we can help: we put the latest smartphones through a battery of objective tests that tells us everything we need to know about the smartphone performance, including the most robust and technically advanced smartphone camera analysis currently available. Additionally, we also use the phone on a day-to-day basis, so that we can tell our readers all the minute details that can make or break a smartphone experience. For more information on how we test smartphones, read on!
Setup and Updating
Prior to testing, all phones are unboxed, fully charged, and updated to the newest publicly available version of the operating system and primary applications. For both Android and iOS, the only apps we install are those we use for analysis, benchmarks, and transferring of files to and from the device.
For nearly all of our tests, we use the default settings that ship with the phone. We may adjust some settings to improve performance if they are obvious to the end user (such as screen brightness), but in all cases, we strive to set a level playing field among all phones by only activating options that a typical consumer would use.
One of the biggest selling points of smartphones is that they take pictures that you can send to your friends and family in the blink of an eye. Because consumers care deeply about the image quality of their smartphone’s primary camera, we go to great lengths to test every smartphone camera as completely as possible.
All of the smartphone camera testing takes place in our camera lab, where we are able to maintain very specific temperature, lighting, and imaging conditions. Our light-controlled lab is lined in light-blocking fabric in order to eliminate reflections, and we use the same lighting and test equipment for smartphones as we do for professional imaging equipment, with the exception of some necessary modifications.
All tests are conducted with the smartphone on a tripod for stability purposes. We typically utilize the default camera app and analyze JPEGs at the highest available quality settings, with exposure adjusted and checked with our image analysis software, Imatest, between each camera test. For certain tests, we also capture RAW image data, though this usually necessitates the usage of a third-party application.
Noise & Noise Reduction
To see how noisy the sensor in the smartphone camera is, we use the smartphone to take a picture of the X-Rite ColorChecker Classic chart, which is comprised of a grid of 24 colored squares: 18 known color values across the spectrum, and 6 monochrome shades. This chart is evenly lit at 1500 lux, using a standard D50 4700 K illuminant.
All digital cameras work by converting incoming light into a charge via the photoelectric effect, reading that charge, and using that information to recreate the scene as a digital image. For a variety of reasons---the inherent randomness of photons, inefficiencies in sensor design, and ambient factors like temperature---this process has a variance that is known as “noise”. These variations from pixel to pixel create a visible grain effect, and when amplified to make up for a lack of light (i.e. when shooting at higher ISO speeds), these differences can become dramatic, drowning out image detail.
To assess how a camera will perform in a typical low light scenario, we capture the ColorChecker chart in low light (~60 lux), using the default camera app at the highest quality JPEG setting. We then process this image with Imatest’s ColorChecker module, which quantifies the average noise level as a percentage. We may also capture other images where RAW or ISO control is available.
All of today’s smartphones rely heavily on a software process called “noise reduction” to improve image quality. This process involves taking a captured RAW image from the sensor, converting it into a JPEG, and applying filters and adjustments to reduce visible noise, while also attempting to preserve fine detail and textural resolution.
To test the noise reduction algorithm, we capture an image of our “spilled coins” chart, which is comprised of a number of occluding circles, of various sizes and colors, which have been distributed randomly. This chart is both scale-invariant and random enough that all noise reduction algorithms should treat it the same way they would any real-world scenario, regardless of the properties of each camera system.
For the most part, noise and noise reduction are in balance with one another. Smartphones with aggressive, heavy-handed noise reduction earn high marks for keeping noise to a minimum, but lose a great deal of points for failing to preserve fine detail. These phones tend to be among the worst performers. Higher quality phones tend to take a lighter touch with noise reduction, leaving in some image noise while preserving the majority of the fine detail in the image. Of course, there are a select few phones that have high quality noise reduction algorithms that are both aggressive about removing noise and still manage to preserve fine details. High-end cameras with large image sensors can produce nearly noise-free images without the need for noise reduction, which would be considered ideal.
Low quality noise reduction algorithms result in a lower noise reduction score (loss of detail and edges due to pixel averaging) and a higher noise score (low amounts of total image noise). Higher quality noise reduction algorithms result in a higher noise reduction score (preserving edging and fine details) and a lower noise score (edging and fine details can register as “noise”, in addition to actual noise that was not averaged). If a camera can produce a noise-free image in low light without the need for noise reduction, then it will earn top marks in both tests.
Because people take pictures under a number of different ambient lighting conditions, we also test how well the smartphone camera can reproduce the colors of the ColorChecker chart in two specific light conditions: daylight (5500 K) and white fluorescent (3700 K). Image exposure is adjusted between the two different illuminants.
Using Imatest, we analyze the captured images to ensure accurate exposure and white balance. Color temperature errors are recorded in kelvins. Smaller errors (<100 kelvins) are considered negligible, but the greater the error, the lower the white balance score will be.
To test how well the combination of the camera sensor and lens can resolve fine detail across an entire image, we take a picture of our custom-made SFRplus 11”x17”, 5x9 grid chart. This chart is printed on LVT film and backlit with the D50 4700K standard illuminant. To get the best possible result, we pay close attention to the orientation of the smartphone, and adjust it as necessary until the entire chart is in focus.
The SFRPlus chart uses a number of slanted-edge targets of medium contrast. The images are captured at an exposure level that ensures that the dark slanted targets and the light background are neither black nor white. Imatest then analyzes the pixel levels between the dark and light areas; the sharper the image, the fewer pixels required to transition from dark to light. The distribution of targets means we get this information across the entire frame.
Most cameras enhance the perceived sharpness of an image by overshooting these luminance changes, so a dark gray box on a light gray background will have a white “halo” effect around it; our eyes will discern that the edges of the box appear to be slightly sharper as a result. While this can improve image quality, it does skew sharpness measurements, and that is taken into account in our final sharpness measurements.
The transition from dark to light (and thus the sharpness of the target) is expressed with a formula called the Modular Transfer Function, or MTF. For scoring, we look at the average resolution as calculated at MTF50, or the function’s midpoint. The average is weighted across the frame to lend more weight to the center of the frame.
These days, smartphones cameras have to not only shoot high quality images, they also have to record high quality (most likely 4K) video. We test video sharpness so that our readers will know that they’re getting awesome video, even when the subject is moving rapidly across the frame.
For video sharpness, we record a few seconds of the CamAlign Mega Trumpet chart, as provided by DSC Labs, in bright light conditions. This chart shows lines with varying thicknesses and distances from one another. As the video is recording, we rotate the smartphone position slightly in both the horizontal and vertical directions. The idea is that, while the camera is moving horizontally or vertically, it will be more difficult for the camera to resolve very thin lines that are packed closely together. By watching the video, we can determine the maximum video sharpness, where the video transitions from actually seeing distinct black lines to recording lines undulating in a blurred, dark grey area.
A higher video sharpness in both the horizontal and vertical directions means that the image sensor and video recorder can pick out very fine detail, even on a small scale, resulting in a higher video sharpness score. A lower video sharpness in both the horizontal and vertical directions means that the image sensor and video recorder cannot resolve very fine detail while recording motion, and will result in lower video quality and a lower video sharpness score.
We record in the highest available resolution, with the highest quality setting possible, in the default camera application. There is a significant difference between 4K and 1080p cameras, so that is often the primary differentiator for this score.
In addition to being our cameras and our camcorders, we also demand that our smartphones act as pocket-sized televisions. With the uptick in streaming and on demand content services, more people are spending time staring at their smartphone screens. Our display tests will tell you everything you need to know about your smartphone viewing experience.
If a phone has multiple display modes, we will test them all (within reason), and report the scores from the display mode that scored the highest overall.
After measuring the size of the screen, we calculate the horizontal and vertical pixels per inch (PPI) using the known screen resolution. Higher horizontal and vertical PPI values mean that both the image in general and the details in particular will be crisp and clear, and will result in a higher resolution score. Lower PPI values means that it will be difficult to pick out fine details, resulting in a lower resolution score.
Contrast is the ratio between the lowest possible luminance (the dimmest rendering of black) and the highest possible luminance (the brightest rendering of white). Using our Konica Minolta CS-200 Luminance Meter, we measure the maximum and minimum screen luminance for a screen rendering a pure white image and a pitch black image, respectively. With those maximum and minimum luminance values, we calculate the contrast ratio for the smartphone’s display.
Large contrast values mean that the display can render very bright whites and/or very deep black shades, which makes for a viewing experience that more closely replicates that of the real world, and results in a higher contrast score. Smaller contrast values involve some combination of dimmer white shades and brighter black shades, which can make images looked washed out and poorly defined. This results in lower contrast scores.
Battery & Benchmark Testing
A smartphone can have an amazing display and a fantastic camera, but that doesn’t matter if the phone’s battery and operating system can’t support extended use. Our tests reveal which phones can keep up with a constant usage demand, and which phones will need to stay close to an outlet.
Prior to battery and benchmark testing, each phone is charged up to 100%, and the display is set to 200 nits.
We use the battery test from Primate Lab’s Geekbench 3.4 program to assess the battery life of each smartphone. The Geekbench battery test runs a number of operations and calculations that require a lot of power. Geekbench runs until the phone’s battery is completely run down, and then reports a battery score. That score is directly related to the number of hours it took to draw down the smartphone’s battery to 0%. For reference, battery test results from users with different phones around the world are reported on Primate Lab’s website.
Higher Geekbench battery test scores mean that the smartphone will be able to last for a long time without being recharged. This results in a higher battery score. Lower Geekbench battery test scores mean that a phone may need to be recharged frequently, much to the user’s chagrin. This results in lower battery scores.
Benchmark tests comprise a series of scripts that ask the smartphone to perform very specific actions aimed at drawing processing power from the CPU and/or the GPU. Benchmark tests differ from the battery test in the types of tasks the phone is asked to complete. Our two benchmark tests, which include Geekbench and 3DMark, test everything from the user experience (i.e. how well it runs multiple programs simultaneously), to rendering high quality video with a lot of quick cuts and rapid motion, and much more.
High scores on benchmark tests means that phone can take whatever punishment (not counting physical) the user can dish out, such as playing games or watching movies for hours at a time, while also checking email and saving pictures from Facebook to your phone. Obviously, people prefer a versatile, robust phone, so high marks on the benchmark tests will result in a high performance score. Phones that cannot switch rapidly between multiple tasks and do a poor job rendering complex video content will have lower benchmark test scores, and will result in a lower performance score.
Once all of the performance testing is done, we also assess the phone’s hardware and features. Some hardware aspects we look for include the amount of RAM built into the phone, if the battery is removable, and if it is capable of charging wirelessly. Some of the features we look for include an accelerometer, a headphone jack, a fingerprint scanner, and whether the phone is waterproof or not.
People generally prefer to have their phones chock full of neat hardware and features, as it makes for a more flexible and fun user experience; consequently, this will result in higher spec-based scores. Smartphones with minimal hardware and feature add-ons may be desirable for customers looking for a simpler, more streamlined user experience, but at the end of the day, the absence of those features means that the smartphone just cannot do as much as its tech- and feature-heavy counterparts. This will result in a lower spec-based score.
While we test the smartphones extensively, there is still important information that cannot be quantified by objective tests. After all of the testing is said and done, we actually use the phone for a few days to see how it feels on a personal level. We answer questions like: How rapid is the feedback from the buttons/touchscreen? How customizable is the smartphone? How well do the pre-loaded apps actually work? Does the phone feel cheap, or is it made of nicer, stronger materials? Strong positive answers to these questions will result in high subjective scores, while phones that actively annoy us will have lower subjective scores.
By combining our extensive objective testing and our more subjective, experiential assessments, we can make solid recommendations (or warnings) to our readers concerning the latest smartphones available on the market today. To see our thoughts on the newest smartphones, check out our expanding library of smartphone reviews. If you want to see our top ranking smartphones, check out our Best Right Now articles, like the Best Smartphones under $500 and the Best Android Smartphones.