Comparing the reliability of ARM and Qualcomm Android graphics drivers, S8 to S8 The GraphicsFuzz ShaderTest GLES test suite reveals a variety of graphics driver issues on the Samsung Galaxy S8. The Qualcomm GPU model suffers from considerably more rendering issues and crashes than the ARM GPU model. . Skip to our analysis of the results follow-on posts evaluating: See also: NVIDIA shield Google Pixel devices Hugues Evrard, Paul Thomson and I have recently founded a company, , specializing in automated testing of graphics drivers. The company is the fruit of research at Imperial College London that we described last year through . GraphicsFuzz a series of Medium stories Click above to try out our GraphicsFuzz Android benchmark web app. Our initial product is , a large suite of fragment shaders that GPU designers can license in order to thoroughly test the reliability of OpenGL ES drivers. ShaderTest GLES This is the first in a series of stories where we’re going to look at the reliability of Android ES drivers, as judged by running them on a representative sample of the shaders. OpenGL ShaderTest GLES You can also try out a selection of these shaders on your Android devices via the — try out the web app in your browser and let us know what you find! GraphicsFuzz Android benchmark ShaderTest GLES: testing shader compiler quality We’ll start with some background on ShaderTest GLES, but ! if you are impatient you can skip to the results Traditional GPU test suites focus on rendering speed or API feature-completeness. ShaderTest GLES adds another dimension to the story by testing . After all, a fast rendering result may not be particularly useful if it is wrong! shader compiler quality Talking with software engineers in industry, we have heard time and again that GPU driver developers prioritize testing their drivers against high-profile games and apps— e.g. the top 100 games in the Google Play Store. An unfortunate side-effect of this is that developers of less popular apps must work around driver bugs to make their apps work on consumer devices. Coupled with the slow speed at which Android updates are pushed to end users, this creates a self-perpetuating situation where GPU driver bugs are not seen as a serious issue because popular games run fine (possibly using workarounds), and GPU driver bugs are not worth reporting because it is necessary to work around the bugs anyway if you want consumers to be able to use your app. It also discourages developers from writing interesting shaders: until the driver quality situation improves, they are unlikely to work on a range of devices. The ShaderTest GLES approach ( .) See this post for some background on shaders and shader compilers ShaderTest GLES consists of many . Each family contains one that is used to render a single frame, and several hundred designed to render a visually identical frame to the reference, but to exercise the driver in radically different ways in the process. shader families reference shader variant shaders The variant shaders are obtained from the reference shaders by applying transformations to the source code of the reference shader that should not affect rendering results. For instance, wrapping a block of code in the reference shader in a loop of the form: semantics-preserving transformations — for (int i = 0; i < 1; i++) {// original code from reference shader} is a simple example of a semantics-preserving transformation. ShaderTest GLES finds bugs in graphics drivers by identifying rendering mismatches, errors and crashes, across families of equivalent shaders. ShaderTest ES highlights driver bugs and issues when: a variant shader is by the driver’s shader compiler — a ; rejected compile failure a variant shader causes some component of the driver to — a ; crash crash error a variant shader leads to a noticeably different image being rendered compared with the reference shader — a ; rendering issue a variant shader causes the driver to bail out due to time or memory constraints — a . resource issue and always indicate bugs. usually arise due to errors, whereby the shader compiler generates wrong code; they sometimes also manifest due to floating-point sensitivity in the graphics systems — ShaderTest GLES can shed light on both issues. A big win here is that miscompilations that trigger rendering issues are very difficult to detect and thus often evade traditional testing techniques. provide insights into the nonfunctional constraints of the graphics driver. Compile failures crash errors Rendering issues miscompilation Resource issues To the Samsungs! Let’s kick off by comparing the results for some reference and variant shaders with respect to one of the most popular Android devices on the market: the Samsung Galaxy S8 phone. Depending on territory, the S8 ships with either an ARM or Qualcomm GPU, so let’s see how the drivers from these GPU designers fare. . Our GraphicsFuzz site shows a comparison between the ARM and Qualcomm S8 models across 10 shader families A snapshot of , which compares the ARM and Qualcomm S8s head-to-head. our online results table Comparing results between the ARM and Qualcomm GPU drivers — we’ll simply call them “ARM” and “Qualcomm” from now on — the ARM model comes off looking more reliable overall. We find the following issues: Rendering issues Qualcomm suffers from significantly more : show five rendering issues from Qualcomm compared with one from ARM. rendering issues our tables For example, . Crashes aside, ARM universally renders the reference image: check out the results for shader family 001 The reference image for shader family 001 . All variants should render a similar image. while with Qualcomm we get the reference image for most variants, but a couple of erroneous images in the mix too: On Qualcomm, we get the expected image (left) for most variants in , but two erroneous images (middle and right), likely due to shader compiler bugs. shader family 001 The reference shader for shader is not floating-point-sensitive, so these discrepancies very likely correspond to shader compiler bugs. shader family 001 We also observe one ARM rendering issue, for variant 77 of : shader family 007 Reference image for (left) compared with bad image rendered by variant 77 (right) on the ARM S8. shader family 007 Nondeterminism Our testing revealed two issues on Qualcomm where the driver behaves nondeterministically: sometimes the shader compiler crashes during compilation; other times compilation succeeds and an incorrect image is rendered. This occurs for variant 123 of , and variant 51 of . shader family 007 shader family 008 Each pair consists of an expected reference image (left image in pair), and a bad variant image rendered by Qualcomm (right image in pair). However, the Qualcomm driver fluctuates nondeterministically between producing this wrong image vs. crashing during compilation. Crash errors Both drivers exhibit a number of crashes on our sample: 28 for Qualcomm vs. 15 for ARM. All are compile-time segmentation faults — Fatal signal 11 (SIGSEGV). The crash data available on Android doesn’t give anything away related to the cause of these crashes. Interestingly, neither driver can handle variant 34 of without crashing. However, as we’ll see in posts, other graphics drivers can handle this variant. shader family 006 future Compile failures The ARM driver exhibits one compiler failure, which occurs for variant 216 of . The issue appears to relate to mishandling of the ternary operator, (?:). Qualcomm shows more compiler failures —nine in total (combining compiler and linker errors), though several of these errors appear to be due to a single underlying issue related to handling of switch statements. shader family 003 Resource issues Our results don’t highlight any Qualcomm resource issues, but do show a resource issue for ARM on variant 122 of , which gives a GL_OUT_OF_MEMORY during rendering. shader family 003 , and try out the ! Check out the full S8 vs. S8 results table GraphicsFuzz Android benchmark web app Next time… … . we’ll see how the Android drivers for NVIDIA’s Shield TV hold up
