Introduction

A literature review on software quality showed that software quality is a broad topic covering several factors, requiring a diverse set of metrics to adequately evaluate and control. My goal with the survey to software professionals was to compare their opinions of the definition of software quality with the definitions found in the literature, and to understand how well-known and utilized software methods, metrics, and visualizations are within software teams. Guiding questions for the survey included:

• How do software professionals define quality?
• What methods do teams most frequently use for improving quality?
• How often do teams/companies manage quality through metrics and visualizations?
• How well do developers know available software quality metrics?
• How likely are software professionals to make use of metric visualization tools for monitoring software quality during software evolution?

Background

The survey consisted of 23 questions, and was administered online using kwiksurveys.

In total, 75 responses across 21 countries and more than 17 companies—not all subjects specified their place of work—were accumulated. The table below shows the full breakdown of the frequency of responses by country.

Responders averaged 10.2 years of software industry experience and identified themselves into the following breakdown of job roles (approx.): 79% developers, 16% managers, 4% test/QA engineers, and 1% business analysts.

Over half (50.7%) of the responders identified as working for a company with 6–50 employees, while another significant portion (30.7%) identified as working for companies with greater than 1,000 employees. The vast majority of responders (89.3%) identified as working within teams of 10 or less, almost evenly split between a size of 1–5 (49.3%) and a size of 6–10 (40.0%). See the tables below for the full distribution of responders' company and team sizes.

Quality

When asked which definition best defines software quality, responders overwhelmingly choose "a broad mix of factors" (77%) over "fit of need" (17%) and "lack of defects" (5%); however, the latter two definitions weighted heavily in the responders' perceptions of quality. When asked to rank a set of eight quality factors in how important the factors represent code quality, Correctness and Reliability were predominantly ranked first and second, respectively. These results confirm the quality definitions found in the literature, with both the Correctness (i.e. "fit of need") and Reliability (i.e. "defects") attributes standing out from the others. With quality in manufacturing very much tied to defects and reliability, it is interesting that, here, "lack of defects" was thought to be less important than "fit of need."

The figure below shows a graph of the rank selection distribution for how important the subjects felt each attribute represents quality.

Ranks given to attributes for importance in representing code quality.

The table below shows the attributes sorted by their weighted-average ranking.

By far, the most important attribute was Correctness, which was placed in the first rank by nearly 70% of the subjects. Second, third, and fourth ranks were also clearly shown to be Reliability, Usability, and Maintainability, respectively. Reusability was clearly the lowest ranked factor, on average, while the remaining three factors (Testability, Efficiency, and Flexibility) were tightly bunched between the fourth and eight ranked factors. While most factors display a single peak within the rank positioning, Efficiency was the only factor that demonstrated two distinct local-maxima (centered around ranks three and seven), suggesting that this factor may have importance in only certain environments or applications.

Subjects were then asked to rank three popular trade-off factors (cost, quality, and schedule) in how influential the factors were for a typical software project at their company. The responses predominantly showed schedule ranked first, quality ranked second, and cost ranked third, with resulting weighted-rank averages of 1.69, 1.93, and 2.32 (out of 3), respectively. The figure below shows the full rank distribution for all three factors.

Rank distributions of schedule, quality, and cost influence for a typical software project.

In a follow-up question to responders who ranked either schedule or cost as more influential than quality, (summarized) reasons given for their selections included:

• Typical projects involve fast-paced, exploratory prototyping.
• Schedule is typically driven by outside factors, such as dependent projects, partner relationships, or regulatory changes.
• Cost and/or schedule are typically the primary focus within contracts.
• Deadlines and schedule receive a high level of focus from management.
• Focus on low cost leads to situations such as the use of less experienced developers who lack the skills to improve quality, or a lack of proper management resources.

Although quality was not ranked as a top consideration for software projects, it was reassuring to learn that the majority of subjects did report on their teams' regular use of multiple methods during the development and/or release process for improving quality. The complete tally of responses can be seen in the table below.

The most popular method used was "informal discussion with colleagues" (84.0%), followed closely by "automated testing" (78.7%) and "human quality assurance (QA) testing" (77.3%). The least used method was "software metrics", which received marks from only 17.3% of responders. The two responses marked "Other" both made mention of static code-analysis tools. The minimal use of metrics as a method for improving quality is disconcerting since it shows that most development teams do not quantitatively know if, or to what degree, their current quality improvement methods are helping them.

When asked to elaborate on methods used for improving quality, including how and why they are used, the result was a wide range of responses, with most describing choices that were provided in the previous question. Notable answers not included in the answer choices included: refactoring, bug monitoring, coding guidelines, stress-test tools, branching strategies (e.g. feature branching), and continuing education. Additional insights gathered on choices that were presented include:

• Related to automated testing were mentions of test-driven development (TDD) and continuous integration.
• Several responses mentioned the fact that automated testing was either under-utilized within their software products, new to their environment, and/or undergoing active improvement.
• Those who mentioned pair programming indicated use of the method for complex or critical bugs and code changes.
• Informal discussions were utilized for completing complex bugs or features, identifying potential issues, learning from problems teammates were facing, and completing changes requiring quick turn-around.
• Code reviews were utilized during new projects, release/sprint iterations, and mentorship situations.
• Human QA testing was used for verifying software correctness and usability, validating releases, and building test plans.
• Responses mentioning formal meetings involved project reviews or retrospectives.

Out of the 75 total subjects, 10 (13.3%) responded that their team does not use any methods for improving quality. Reasons cited included: tight deadlines, lack of time, lack of experience, bad tool support, non-production software, too many projects, lack of management, and team/company culture. The multiple mentions of time constraints here align with the earlier results that show schedule as a more influential trade-off factor on software projects over quality.

Metrics

In the previous section, it was shown that software metrics was the least utilized method for improving quality as part of development processes. However, when asked if their team currently utilizes any metrics for the intentional purpose of improving code quality, just over half of the responders (38 of 75) answered "yes."

The subjects were then asked how regularly their team used metrics to evaluate seven different areas, which represented a one-to-one mapping to the measurement goals derived in the Goal, Question, Metric (GQM) exercise (not presented here). Answer choices were a frequency-type Likert scale with the following labels: Never, Rarely, Sometimes, Regularly, Constantly. The figure below depicts the choice distributions for all seven measurement areas, and the table below shows the measurement areas sorted by most frequently used (a weighted-average ordering assuming approximately equal intervals of the frequency-type Likert scale, with values 1 to 5 representing "Never" to "Constantly", respectively).

Distributions of the frequency that teams make use of metrics to address measurement goals presented in GQM exercise.

While over half of the responders answered "yes" to currently utilizing metrics for improving code quality, they did so at a very low frequency in each of the measurement areas questioned. Only two areas (issues in issue database and test suites or benchmarks) had a median frequency above "Rarely," and were also not much more utilized themselves with a median frequency of "Sometimes." Likewise, only two areas had a most-selected frequency above "Never"—complexity of software had a mode of "Rarely" and issues in issue database had a mode of "Regularly." Issues in issue database was the most-frequently utilized metric area, clearly seen as on outlier within the "Regularly" frequency choice in Figure measurement goal frequency. In fact, aside from that single outlying point, there is no metric area for the "Regularly" and "Constantly" frequencies that had more selections than the least-selected metric area for the "Rarely" and "Never" frequencies.

Next, the subjects were asked to rate their knowledge and use of 28 different metrics on a Likert scale consisting of the four choices: Never heard of or do not know, Know of but have not used, Have used before, and Use regularly. The list of metrics used for the survey was primarily taken from a GQM derivation (omitted from this post), and was found throughout a range of topics in the software engineering literature, including:

• software sizing and effort [1] [2] [19] [24]
• complexity [4] [17] [26]
• defects [7] [27] [13] [18]
• code duplication [23]
• testing [21] [32] [10] [3] [31] [30] [33]
• evolution [15]
• coupling [16]
• cohesion and object-oriented design [9] [6] [8] [29] [11]
• information flow [20]
• people and organizational structure [28]
• various combinations of the proceeding topics [5] [14] [25] [34] [22] [12]

The table below shows the full count (and percentage) breakdown of responses for each metric. The data is sorted by highest weighted average (assuming approximate equal intervals of a Likert scale, weighted from 1 to 4 representing the selections "Never heard of or do not know" to "Use regularly"), which allows identification of the central tendency of selections for each metric.

From the data collected, several insights were captured:

• No metric was used regularly by a majority of the responders. The only metric that came close (42.7%) was Unit test failures or pass/fail ratio.
• Only four metrics have been used before or were used regularly by a majority of responders: Unit test failures or pass/fail ratio, Line coverage, Lines of code, and Number of classes/functions/files/modules. These four metrics were also the only ones that were both unknown to less than 10% of responders and used regularly by more than 10% of responders.
• Aside from the four metrics mentioned in the previous bullet, Branch/path coverage was the only other metric that was used regularly by more than 10% of the responders; however, a higher percentage of responders (19%) had never heard of this metric (about three times more than the four metrics mentioned above).
• Several metrics were highly known, yet not used before or used regularly, including: Mean time between defect/error, Time-to-fix defect, Defect density, Code churn, Depth of inheritance tree, Depth of nesting, and Defect count or distribution. It us unknown, however, if the high levels of knowledge of these metrics are due to familiarity of their use within the software industry or rather due to the ease of which their definitions can be gleaned from their names.
• Nearly one-third of the metrics presented were completely unknown by a majority of responders, with most metrics in this group used regularly by zero responders and used before by about 5% or less of responders. Metrics in this group consisted of Fan-in and fan-out, Defect slippage ratio, Halstead metrics, Mutant killing percentage, and several of the people and organizational metrics described by Zimmermann et. al [34].

The final two questions in the survey related to metrics were on the topic of the use of metrics to evaluate employee performance. About 15% of responders indicated that their company uses software metrics for evaluating employee performance. Surprisingly, nearly twice that amount (28.0%) either agreed or strongly agreed with the use of metrics for evaluating employee performance. The full results can been seen in the table below, broken down by manager and non-manager roles (as identified earlier in the survey, see Background).

From the manager vs. non-manager results, we expectedly see that those in manager roles were much more likely to agree (50.0%) or strongly agree (8.3%) than were those in non-manager roles. Those in non-manager roles were most likely to be neutral (32.0%) or to disagree (23.8%). Both manager and non-manager roles differed by less than 1% in the strongly disagree segment and, in aggregate, very few (4.0%) responders identified as strongly agreeing with the use of metrics for employee performance evaluation.

Visualization

The survey to software professionals also included four questions on the topic of visualizing software metrics. Overall, responders overwhelmingly felt that it would be useful to visualize metrics about their software projects over time, with over 81% who either agreed or strongly agreed with this statement and less than 6% who either disagreed or strongly disagreed. However, when asked if their team currently uses any tools to visualize software metrics over time, less than 30% answered "yes." In a similar question, subjects were asked how likely they would be to use a software metrics visualization tool to help improve the quality of software they write. Here, responders were also positive, albeit slightly less enthusiastic. The majority of responders fell into the likely (42.7%) or neutral (25.3%) segments, with an equal percentage (14.6%) in both the unlikely and very likely segments. The full results for these two questions can be seen in the tables below.

From these results, combined with the earlier result that just over half of the responders answering "yes" to their team currently utilizing metrics for improving code quality, there clearly exists a gap between those who use metrics and those who use metrics visualizations. Additionally, the positive responses to the perceived usefulness of metrics visualizations, and willingness to use metrics visualizations, suggests there is opportunity and demand for such tools.

On the topic of visualization tools, subjects were asked how they would like to use such a tool (see figure below for a chart of all responses). The most popular choices, selected by about 61--71% of the responders, were: integrated into build/test server, integrated into issue tracker, and a self-hosted, stand-alone product with web interface. Least popular were: integrated into editor/IDE and a hosted service with web interface. Four responders filled in a selection for other, mentioning a dedicated build server, as well as a script or command-line interface.

Number of responses on how subjects would like to use a tool for calculating and visualizing metrics describing their software.

References