Why Using Static Analysis Is Hard

Using static analysis is hard, counter-intuitive and not-automatic. We want to answer some questions related to the use of static analyzers: why this is hard for some users and what are the solutions to help.

Static analysis is the operation of finding issues in the code before it is compiled or executed. Programs called static analyzers perform static analysis by inspecting the source code and finding defects. Defects can belong to multiple categories (security, safety, documentation) and have various severities.

One famous example of static analysis tool usage is detection of buffer overflow in C programs. Vulnerabilities can be discovered by looking at the source code and  memory operations. In Javascript, static analyzers such as jshint or flow provide useful information on formatting and type checking.

You can find a long list of available tools on this GitHub repository.

Static analysis does not detect if the code correctly implements the customers requirements. It only checks for defects in your  code. On the other hand, the code you wrote might not reflect what it is supposed to implement.

Absolutely not. Testing complements and static analysis both must be used:

Static analysis must be used to ensure that the code is syntactically and semantically correct

Testing must be used to ensure that the code correctly implements the requirements. Moreover, code coverage must be measured to guarantee that all the code is thoroughly tested.

Static analysis is important because it detects potential defects developers did not even think about. Even if the implementation is correct (and tests pass), the code might still have some issues, such as slow performance (algorithms are not optimal), poor practices (not using the right functions or syntax) or just lack of documentation.

Static analyzers catch such issues and therefore, help developers shipp better code.

Static analysis tools are very useful and integral to produce better code. However, they also can be very time consuming for developers because you need to:

The following sections explain each aspect.

The first issue is to configure and calibrate the tool you are using. Static analyzers come with hundreds or thousands of rules that check code. These rules may or may not be useful to you and require time to understand each of them in order to get exactly what you want.

Most of the time, you calibrate the tool, after a few runs you find out the rules that are relevant to your programs and you remove the rules that do not seem to make sense. However, this process is conducted manually and very time consuming.

Most static analysis tools do not have context behind your code and detect issues based on inference rules. However, many defects depend on the context of your code and not all reported defects are relevant. These are called false positives: A false positive is when the static analysis tool reports a defect when none are present. In addition to the false positives, there might also be some rules you want to avoid (for example, you might have a very personal style about code formatting and want to just avoid all formatting-related defects).

Finding false positives can be confusing: developers question the  report’s legitimacy at first glance. Identifying and removing these false positives takes significant time for developers.

The last issue related to static analysis is to triage the results and “find the needle in the haystack”. Some static analysis tools are very verbose, reporting a lot of potential defects. This makes it hard to know what tools are relevant and which should be fixed. Developers must then take the list of all reported issues and filter them manually to finally find ones that might potentially impact their code.

The best way to simplify the use of static analysis tools is to use a tool already calibrated and tailored to your project. This service is provided by code analysis platforms, such as Codiga. Such platforms train their algorithms to remove false positives and report issues only relevant to your project.

By using such a platform, you take the benefits from static analysis tools, wasting no time configuring them or filtering results.

Static analysis tools are very useful for developers and should be used as much as possible. However, they require significant time to be tailored in order to produce a list of meaningful results for your project. By using analysis platforms such as Codiga, you can use a ready-to-use, pre-calibrated tool to help you find relevant issues for your projects.