How We Test Smartphones
See How We Test Other Products:
These days, you can do almost anything from your smartphone, whether it’s texting, banking, catching a ride across town, pursuing Pokémon, or picking out paint colors for your new kitchen. Even though we love our phones, the technology improves so quickly that we often want to buy a new one every 1-2 years.
If you’re a rabid brand loyalist, then there’s no question that you’ll buy the latest version of your favorite phone, but what if you don’t have a preference for either Android or iOS X? Here at Reviewed.com, we can help: we put the latest smartphones through a battery of objective tests that tells us everything we need to know about the smartphone performance. Additionally, we also use the phone on a day-to-day basis, so that we can tell our readers all of the little details that can make or break a smartphone experience. For more information on how we test smartphones, read on!
Setup and Updating
Once the battery is installed and fully charged, we immediately ask the operating system to identify any updates. This is particularly important for the smartphone camera, as a significant portion of the product’s score depends on the image processing software. Operating system updates are also vital because part of the objective testing includes benchmark and battery tests. Having all of the relevant updates ensures that we are testing each smartphone at its best.
Additionally, we turn off any possible automatic settings that affect the smartphone camera, the battery life, and the display brightness.
One of the biggest selling points of smartphones is that they take pictures that you can send to your friends and family with just a few finger taps. Because the quality of the camera is becoming an increasingly important part of a consumer’s smartphone buying experience, we want to make sure that our readers get the low-down on the camera’s performance.
All of the smartphone camera testing takes place in our camera lab, where we are able to maintain very specific temperature, lighting, and imaging conditions. In order to ensure that each camera is assessed at its best, the smartphone rests on a tripod for stability purposes, and image exposure is adjusted every time the lighting conditions change (i.e. between each camera test). The images we use for our testing are JPEGs, unless specifically mentioned otherwise.
Color Accuracy, Noise, & Noise Reduction
To see how well a smartphone camera can reproduce known color values, we use the smartphone to take a picture of the X-Rite ColorChecker Classic chart, which comprises a grid of 24 colored squares: 18 colors, and 6 monochrome shades. This chart is evenly lit at 1500 lux, using a standard D50 4700 K illuminant.
The resulting image is then piped into a program called Imatest, which is an image processing software that is also used extensively in our camera testing. Imatest quantifies the difference between the 18 known color values on the chart and the colors of those same colored grid squares captured by the smartphone camera (known as the color error, ΔC00). To make sure that the image is correctly exposed, we adjust the exposure on the smartphone camera until the images we see have consistent shading changes between each adjacent pair of the 6 monochrome grid squares.
The higher the color error in the observed color values, the lower the color accuracy score. Smaller ΔC00 values means that the colors seen in real life are being more faithfully reproduced by the smartphone camera, and will result in a higher color accuracy score.
When an object is lit, photons bounce off of the surface of the object and are translated into varying degrees of light and color in the pixels of a digital camera image via the photoelectric effect. The "noise" in an image refers to the slight variations in amount of red, green, and blue (RGB) color from pixel to pixel due to differing numbers of photons being captured by neighboring pixels. The noise is most noticeable when neighboring pixels should be identical in color content (such as two pixels within the first brown square in the ColorChecker chart), but aren’t.
To assess the noise in an image, we shoot the same ColorChecker chart, but in low light (~60 lux) conditions. To counter the low light conditions, the image processing software tends to boost the amplitude of the light in each pixel, trying to make the objects in the image more visible. This also tends to exacerbate any observed noise. We run this image through Imatest, and it quantifies the noise by calculating the average value, over the entire image, of the light and color difference between adjacent pixels.
Hand-in-hand with noise is noise reduction. Noise reduction is accomplished with an algorithm in the image processing software. When the algorithm identifies neighboring pixels that are very different, it averages those two pixels to make the image more seamless in that area. We test the noise reduction algorithm by taking a picture of our “spilled coins” chart in the same low light conditions. In this image, there are many fine edges and details with abrupt changes in color and shading; by shooting this chart in low light conditions, we are increasing the likelihood that the image will be moderately noisy.
For the most part, noise and noise reduction are in balance with one another. Smartphones with aggressive, heavy-handed noise reduction earn high marks for keeping noise to a minimum, but lose a great deal of points for failing to preserve edges. These phones tend to be among the worst performers. Higher quality phones tend to take a lighter touch with noise reduction, leaving in some image noise while preserving the majority of the fine detail in the image. Of course, there are a select few phones that have high quality noise reduction algorithms that are both aggressive about removing noise and still manage to preserve fine details.
Low quality noise reduction algorithms result in a lower noise reduction score (loss of detail and edges due to pixel averaging) and a higher noise score (low amounts of total image noise). Higher quality noise reduction algorithms result in a higher noise reduction score (preserving edging and fine details) and a lower noise score (edging and fine details can register as “noise”, in addition to actual noise that was not averaged). If a camera can produce a noise-free image in low light without the need for noise reduction then it will earn top marks in both tests.
Because people rarely take pictures with their smartphone in a dark room with very specific illumination, we also test how well the smartphone camera can reproduce the colors of the ColorChecker chart in three specific light conditions: incandescent (2800 K), daylight (5500 K), and white fluorescent (3700 K). Image exposure is adjusted between each of the three illuminants.
Smaller color error (ΔC00) values in each of the three lighting conditions means that the camera can accurately reproduce the object in a variety of different ambient light settings, and result in a high white balance score. Larger color error values means that the smartphone camera is typically dialed into one or two specific light settings, and has trouble adjusting to other ambient light settings, resulting in a low white balance score.
To test how well the combination of the camera sensor and lens can resolve fine detail across an entire image, we take a picture of the SFRplus 11”x17”, 5x9 grid chart. This chart is printed on LVT film and backlit with the D50 4700K standard illuminant. To get the best possible result, we pay close attention to the orientation of the smartphone, and adjust it as necessary until the entire chart is in focus. The resulting image is sent to Imatest, where the change in luminance (between the white of the background and the black of each square) is plotted as a line over a number of pixels.
Most cameras enhance their perceived sharpness by overshooting these luminance changes, so a dark gray box on a light gray background will have a white “halo” effect around it; our eyes will perceive the edges of the box to be slightly sharper as a result. While this can improve image quality, we mostly correct for this oversharpening so as to not reward phones that over-process their images.
One way around this processing would be to only test RAW images. While we do take RAW images where available, all of our results are based on JPEG images, since that’s what most users will use and RAW is not yet available on all smartphones.
A camera and lens combination that shows a rapid change in the luminance over a small number of pixels (steep slope) means that the camera can successfully resolve very abrupt changes in luminance, and results in a higher sharpness score. A camera and lens combination that shows a more gradual change in luminance over a larger number of pixels (shallow slope) means that the camera struggles to reproduce that fine transition from white to black, and will result blurrier images and a lower sharpness score.
These days, smartphones cameras have to not only shoot high quality images, they also have to record high quality (most likely 4K) video. We test for both video motion and video sharpness so that our readers will know that they’re getting awesome video, whether the subject is moving rapidly or standing completely still.
For video sharpness, we record a few seconds of the CamAlign Mega Trumpet chart, as provided by DSC Labs, at bright light conditions. This chart shows lines with varying thicknesses and distances from one another. As the video is recording, we rotate the smartphone position slightly in both the horizontal and vertical directions. The idea is that, while the camera is moving horizontally or vertically, it will be more difficult for the camera to resolve very thin lines that are packed closely together. By watching the video, we can determine the maximum video sharpness, where the video transitions from actually seeing distinct black lines to recording lines undulating in a blurred, dark grey area.
A higher video sharpness in both the horizontal and vertical directions means that the image sensor and video recorder can pick out very fine detail, even on a small scale, resulting in a higher video sharpness score. A lower video sharpness in both the horizontal and vertical directions means that the image sensor and video recorder cannot resolve very fine detail, and will result in lower video quality and a lower video sharpness score.
To test the quality of the video motion, we record a few seconds of our custom-made motion quality rig, which is visually busy and has some moving parts. Because there’s so much going on in this setup, we assess how well the video does at interpolation (filling in perceived gaps in visual information), smoothness (making motion look continuous instead of juddering), artifacting (where the camera cannot process more complicated visual patterns, and instead produces noisy garbage), and more.
Ideally, high-quality cameras should produce videos that closely mimic a real life experience. Any problems with the aforementioned qualities (interpolation, smoothness, and artifacting) will make it immediately obvious to the viewer that they are watching a video, and will result in a lower video motion score. The more seamless the transition from real life to video, the higher the video motion score.
In addition to being our cameras and our camcorders, we also demand that our smartphones act as pocket-sized televisions. With the uptick in streaming and on demand content services, more people are spending more time staring at their smartphone screens. Our display tests will tell you everything you need to know about your smartphone viewing experience.
If a phone has multiple display modes, we will test them all, and report the scores from the display mode that scored the highest overall.
After measuring the size of the screen, we calculate the horizontal and vertical pixels per inch (PPI) using the known screen resolution. Higher horizontal and vertical PPI values mean that both the image in general and the details in particular will be crisp and clear, and will result in a higher resolution score. Lower PPI values means that it will be difficult to pick out fine details, resulting in a lower resolution score.
An important aspect of any type of display is the ability to reproduce discrete shades of gray. "Gamma" refers to the mathematical relationship between the input signal intensity needed to render a shade of gray and the observed output light levels. We aim for a gamma value of between 2.2-2.4 because these values generate distinct shades of gray that are visible in multiple lighting conditions (in a dark room, on the bus in daylight, etc.).
To measure the grayscale gamma, we use our Konica Minolta CS-200 to measure the luminance (i.e. the brightness) of the screen for 13 shades of gray with known input signal levels.
If these measurements bear out a gamma value between 2.2-2.4, that means that the smartphone is making an effort to render the shades of gray distinguishable to the human eye. This gives the image lots of depth, and will result in both a great viewing experience and a higher grayscale gamma score. The further away the gamma value is from our desired range, the more washed out or overly dark the picture will look, and the lower the grayscale gamma score will be.
Contrast is the ratio between the lowest possible luminance (the dimmest rendering of black) and the highest possible luminance (the brightest rendering of white). Using the maximum and minimum luminance values from the grayscale gamma test, we calculate the contrast ratio for the smartphone’s display. Large contrast values mean that the display can render very bright whites and very deep black shades, which makes for a viewing experience that more closely replicates the real world, and results in a higher contrast score. Smaller contrast values involve some combination of dimmer white shades and brighter black shades, which can make images looked washed out and poorly defined. This results in lower contrast scores.
Battery & Benchmark Testing
A smartphone can have an amazing display and a fantastic camera, but that doesn’t matter if the phone’s battery and operating system can’t support extended use. Our tests reveal which phones can keep up with a constant usage demand, and which phones will need to stay close to an outlet.
Prior to battery and benchmark testing, each phone is charged up to 100%.
We use the battery test from Primate Lab’s Geekbench 3.4 program to assess the battery life of each smartphone. The Geekbench battery test runs a number of operations and calculations that require a lot of power. Geekbench runs until the phone’s battery is completely run down, and then reports a battery score. That score is directly related to the number of hours it took to draw down the smartphone’s battery to 0%. For reference, battery test results from users with different phones around the world are reported on Primate Lab’s website.
Higher Geekbench battery test scores mean that the smartphone will be able to last for a long time without being recharged. This results in a higher battery score. Lower Geekbench battery test scores mean that a phone may need to be recharged frequently, much to the user’s chagrin. This results in lower battery scores.
Benchmark tests comprise a series of scripts that ask the smartphone to perform very specific actions aimed at drawing processing power from the CPU and/or the GPU. Benchmark tests differ from the battery test in the types of tasks the phone is asked to complete. Our three benchmark tests, which include Geekbench, AnTutu, and 3DMark, test everything from the user experience (i.e. how well it runs multiple programs simultaneously), to calculating the decimal places of pi, to rendering high quality video with a lot of quick cuts and rapid motion, and much more.
High scores on benchmark tests means that phone can take whatever punishment (not counting physical) the user can dish out, such as playing games or watching movies for hours at a time, while also checking email and saving pictures from Facebook to your phone. Obviously, people prefer a versatile, robust phone, so high marks on the benchmark tests will result in a high performance score. Phones that cannot switch rapidly between multiple tasks and do a poor job rendering complex video content will have lower benchmark test scores, and will result in a lower performance score.
Once all of the performance testing is done, we also assess the phone’s hardware and features. Some hardware aspects we look for include the amount of RAM built into the phone, if the battery is removable, and if it is capable of charging wirelessly. Some of the features we look for include an accelerometer, a stylus, and whether the phone is dust/water/shockproof.
People generally prefer to have their phones chock full of neat hardware and features, as it makes for a more flexible and fun user experience; consequently, this will result in higher spec-based scores. Smartphones with minimal hardware and feature add-ons may be desirable for customers looking for a simpler, more streamlined user experience, but at the end of the day, the absence of those features means that the smartphone just cannot do as much as its tech- and feature-heavy counterparts. This will result in a lower spec-based score.
While we test the smartphones extensively, there is still important information that cannot be quantified by objective tests. After all of the testing is said and done, we actually use the phone for a few days to see how it feels on a personal level. We answer questions like: How rapid is the feedback from the buttons/touchscreen? How customizable is the smartphone? How well do the pre-loaded apps actually work? Does the phone feel cheap, or is it made of nicer, stronger materials? Strong positive answers to these questions will result in high subjective scores, while phones that actively annoy us will have lower subjective scores.
By combining our extensive objective testing and our more subjective, experiential assessments, we can solid recommendations (or warnings) to our readers concerning the latest smartphones available on the market today. To see our thoughts on the newest smartphones, check out our expanding library of smartphone reviews. If you want to see our top ranking smartphones, check out our Best Right Now articles, like the Best Smartphones under $500 and the Best Android Smartphones.