commit 5b15ce12a6a735de00f28b818aa0f07488f2dc6e Author: Andy Ko Date: Sun Apr 16 11:59:49 2017 -0700 First check-in of book. diff --git a/architecture.html b/architecture.html new file mode 100644 index 0000000..4dcf91c --- /dev/null +++ b/architecture.html @@ -0,0 +1,91 @@ + + + + + + + + + + + + + + + + + + Architecture + + + +

+ + +

+ Credit: Creative Commons 0 + + +

Architecture

Andrew J. Ko

+ +

Once you have a sense of what your design must do (in the form of requirements or other less formal specifications), the next big problem is one of organization. How will you order all of the different data, algorithms, and control implied by your requirements? With a small program of a few hundred lines, you can get away without much organization, but as programs scale, they quickly become impossible to manage alone, let alone with multiple developers. Much of this challenge occurs because requirements change, and every time they do, code has to change to accommodate. The more code their is and the more entangled it is, the harder it is to change and more likely you are to break things.

+ +

This is where architecture comes in. Architecture is a way of organizing code, just like building architecture is a way of organizing space. The idea of software architecture has at its foundation a principle of information hiding: the less a part of a program knows about other parts of a program, the easier it is to change. The most popular information hiding strategy is encapsulation: this is the idea of designing self-contained abstractions with well-defined interfaces that separate different concerns in a program. Programming languages offer encapsulation support through things like functions and classes, which encapsulate data and functionality together. Another programming language encapsulation method is scoping, which hides variables and other names from other parts of program outside a scope. All of these strategies attempt to encourage developers to maximize information hiding and separation of concerns. If you get your encapsulation right, you should be able to easily make changes to a program's behavior without having to change everything about it's implementation.

+ +

When encapsulation strategies fail, one can end up with what some affectionately call a "ball of mud" architecture or "spaghetti code". A more precise concept is cross-cutting concerns, which are things like features and functionality that span multiple different components of a system, or even an entire system. There is some evidence that cross-cutting concerns can lead to difficulties in program comprehension and long-term design degradation (Walker et al. 2012), all of which reduce productivity and increase the risk of defects. As long-lived systems get harder to change, they can take on technical debt, which is the degree to which an implementation is out of sync with a team's understanding of what a product is intended to be. Many developers view such debt as emerging from primarily from poor architectural decisions (Ernst et al. 2015). Over time, this debt can further result in organizational challenges (Khadka et al. 2014), making change even more difficult.

+ +

The preventative solution to this problems is to try to design architecture up front, mitigating the various risks that come from cross-cutting concerns (defects, low modifiability, etc.) (Fairbanks 2010). A popular method in the 1990's was the Unified Modeling Language (UML), which was a series of notations for expressing the architectural design of a system before implementing it. Recent studies show that UML generally not used and generally not universal (Petre 2013). More recent developers have investigated ideas of architectural styles, which are patterns of interactions and information exchange between encapsulated components. Some common architectural styles include:

+ +

Client/server, in which data is transacted in response to requests. This is the basis of the Internet and cloud computing (Cito et la. 2015).
Pipe and filter, in which data is passed from component to component, and transformed and filtered along the way. Command lines, compilers, and machine learned programs are examples of pipe and filter architectures.
Model-view-controller (MVC), in which data is separated from views of the data and from manipulations of data. Nearly all user interface toolkits use MVC, including popular modern frameworks such as React.
Peer to peer (P2P), in which components transact data through a distributed standard interface. Examples include Bitcoin, Spotify, and Gnutella. +

+ +

Architectural styles come in all shapes and sizes. Some are smaller design patterns of information sharing (Beck et al. 2006), whereas others are ubiquitous but specialized patterns such as the architectures required to support undo and cancel in user interfaces (Bass et al. 2004).

+ +

One fundamental unit of which an architecture is composed is a component. This is basically a word that refers to any abstraction—any code, really—that attempts to encapsulate some well defined functionality or behavior separate from other functionality and behavior. Components have interfaces that decide how it can communicate with other components. It might be a class, a data structure, a set of functions, a library, or even something like a web service. All of these are abstractions that encapsulate interrelated computation and state. The second fundamental unit of architecture is connectors. Connectors are abstractions (code) that transmit information between components. They're brokers that connect components, but do not necessarily have meaningful computation or state of their own. Connectors can be things like function calls, web service API calls, events, requests, and so on.

+ +

Even with carefully selected architectures, systems can still be difficult to put together, leading to architectural mismatch (Garlan et al. 1995). When mismatch occurs, connecting two styles can require dramatic amounts of code to connect, imposing significant risk of defects and cost of maintenance. One common example of mismatches occurs with the ubiquitous use of database schemas with client/server web-applications. A single change in a database schema can often result in dramatic changes in an application, as every line of code that uses that part of the scheme either directly or indirectly must be updated (Qiu et al. 2013). This kind of mismatch occurs because the component that manages data (the database) and the component that renders data (the user interface) is highly "coupled" with the database schema: the user interface needs to know a lot about the data, its meaning, and its structure in order to render it meaningfully.

+ +

The most common approach to dealing with both architectural mismatch and the changing of requirements over time is refactoring, which means changing the architecture of an implementation without changing its behavior. Refactoring is something most developers do as part of changing a system (Murphy-Hill et al 2009, Silva et al. 2016). Refactoring code to eliminate mismatch and technical debt can simplify change in the future, saving time (Ng et al. 2006) and prevent future defects (Kim et al. 2012). + +

Next chapter: Specifications + +

Podcasts

+ + + +

Software Engineering Daily, React JS with Sebastian Marbage and Christopher Chedeua

+ + + + + + + + + diff --git a/communication.html b/communication.html new file mode 100644 index 0000000..82d2826 --- /dev/null +++ b/communication.html @@ -0,0 +1,91 @@ + + + + + + + + + + + + + + + + + + + Communication + + + +

Back to table of contents

+ +

+ Credit: public domain + +

Communication

Andrew J. Ko

+ +

Because software engineering often times distributes work across multiple people, a fundamental challenge in software engineering is ensuring that everyone on a team has the same understanding of what is being built and why. In the seminal book "The Mythical Man Month", Fred Brooks argued that good software needs to have conceptual integrity, both in how it is designed, but also how it is implemented (Brooks 1995). When multiple people are responsible for implementing a single coherent idea, how can they ensure they are building the same idea?

+ +

The solution is effective communication. When communication is poor and teams are disconnected, software defects are the result (Bettenburg & Hassan 2013). The social relationships in a team also play a large role in structuring how projects evolve (Zhou & Mockus 2011). Perhaps the most notable theory underlying these ideas is Conway's Law (Conway 1968), which argues that any designed system—software included—will reflect the social relationships behind it's design. For example, look at any online banking website: the way the application is designed, how information is organized, the terminology that is used, and even the visual design is a reflection of how the teams inside that bank are organized and socially connected.

+ +

Because communication is so central, software engineers are constantly seeking information to further their work, going to their coworkers' desks, emailing them, chatting via messaging platforms, and even using social media (Ko et al. 2007). Some of the information that developers are seeking is easier to find than others. For example, in the study I just cited, it was pretty trivial to find information about how wrote a line of code or whether a build was done, but when the information they needed resided in someone else's head (e.g., why a particular line of code was written), it was slow or often impossible to retrieve it. Sometimes it's not even possible to find out who has the information. Researchers have investigated tools for trying to quantify expertise by automatically analyzing the code that developers have written, building platforms to help developers search for other developers who might know what they need to know (Mockus & Herbsleb 2002, Begel et al. 2010).

+ +

Communication is not always effective. In fact, there are many kinds of communication that are highly problematic in software engineering teams. For example, Perlow (1999) conducted an ethnography of one team and found a highly dysfunctional use of interruptions in which the most expert members of a team were constantly interrupted to "fight fires" (immediately address critical problems) in other parts of the organization, and then the organization rewarded them for their heroics. This not only made the most expert engineers less productive, but it also disincentivized the rest of the organization to find effective ways of preventing the disasters from occurring in the first place. Not all interruptions are bad, and they can increase productivity, but they do increase stress (Mark et al. 2008).

+ +

Communication isn't just about transmitting information; it's also about relationships and identity. For example, the dominant culture of many software engineering work environments—and even the perceived culture—is one that can deter many people from even pursuing careers in computer science. Modern work environments are still dominated by men, who speak loudly, out of turn, and disrespectfully, with some even bordering on sexual harassment. These are not the conditions for trusting, effective communication.

+ +

When communication is effective, it still takes time. One of the key strategies for reducing the amount of communication necessary is knowledge sharing tools, which broadly refers to any information system that stores facts that developers would normally have to retrieve from a person. By storing them in a database and making them easy to search, teams can avoid interruptions. The most common knowledge sharing tools in software teams are issue trackers, which are often at the center of communication not only between developers, but also with every other part of a software organization (Bertram et al. 2010). Community portals, such as GitHub pages or Slack teams, can also be effective ways of sharing documents and archiving decisions (Treude & Storey 2011). Perhaps the most popular knowledge sharing tool in software engineering today is Stack Overflow, which archives facts about programming language and API usage.

+ +

Because all of this knowledge is so critical to progress, when developers leave an organization and haven't archived their knowledge somewhere, it can be quite disruptive to progress. Organizations often have single points of failure, in which a single developer may be critical to a team's ability to maintain and enhance a software product (Rigby et al. 2016). When newcomers join a team and lack the right knowledge, they introduce defects (Foucault et al. 2015). Some companies try to mitigate this by rotating developers between projects, "cross-training" them to ensure that the necessary knowledge to maintain a project is distributed across multiple engineers.

+ +

What does all of this mean for you as an individual developer? To put it simply, don't underestimate the importance of talking. Know who you need to talk to, talk to them frequently, and to the extent that you can, write down what you know both to lessen the demand for talking and mitigate the risk of you not being available, but also to make your knowledge more precise and accessible in the future. It often takes decades for engineers to excel at communication. The very fact that you know why communication is important gives you an critical head start.

+ + Next chapter: Productivity + +

Podcasts

+ + + +

Software Engineering Daily. Female Pursuit of Computer Science with Jennifer Wang.

Software Engineering Daily. The State of Programming with Stack Overflow Co-Founder Jeff Atwood.

+ + + + + + + + + diff --git a/comprehension.html b/comprehension.html new file mode 100644 index 0000000..44d395e --- /dev/null +++ b/comprehension.html @@ -0,0 +1,173 @@ + + + + + + + + + + + + + + + + + + Comprehension + + + +

Back to table of contents

+ + +

+ Credit: public domain + +

Program Comprehension

Andrew J. Ko

+ +

Despite all of the different activities that we've talked about so far—communicating, coordinating, planning, designing, architecting—really, most of a software engineers time is spent reading code (Maalej et al. 2014). Sometimes this is their own code, which makes this reading easier. Most of the time, it is someone else's code, whether it's a teammate's, or part of a library or API you're using. We call this reading program comprehension.

+ +

Being good at program comprehension is a critical skill. You need to be able to read a function and know what it will do with its inputs; you need to be able to read a class and understand its state and functionality; you also need to be able to comprehend a whole implementation, understanding its architecture. Without these skills, you can't test well, you can't debug well, and you can fix or enhance the systems you're building or maintaining. In fact, studies of software engineers' first year at their first job show that a significant majority of their time is spent trying to simply comprehend the architecture of the system they are building or maintaining and understanding the processes that are being followed to modify and enhance them (Dagenais et al. 2010).

+ + + +

What's going on when developers comprehend code? Usually, developers are trying to answer questions about code that help them build larger models of how a program works. Because program comprehension is hard, they avoid it when they can, relying on explanations from other developers rather than trying to build precise models of how a program works on their own (Roehm et al. 2012). When they do try to comprehend code, developers are usually trying to answer questions. Several studies have many general questions that developers must be able to answer in order to understand programs (Sillito et al. 2006, LaToza & Myers 2010). Here are over forty common questions that developers ask:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Which type represents this domain concept or this UI element or action?	Where in the code is the text in this error message or UI element?
Where is there any code involved in the implementation of this behavior?	Is there an entity named something like this in that unit (for example in a project, package or class)?
What are the parts of this type?	Which types is this type a part of?
Where does this type fit in the type hierarchy?	Does this type have any siblings in the type hierarchy?
Where is this field declared in the type hierarchy?	Who implements this interface or these abstract methods?
Where is this method called or type referenced?	When during the execution is this method called?
Where are instances of this class created?	Where is this variable or data structure being accessed?
What data can we access from this object?	What does the declaration or definition of this look like?
What are the arguments to this function?	What are the values of these arguments at runtime?
What data is being modified in this code?	How are instances of these types created and assembled?
How are these types or objects related?	How is this feature or concern (object ownership, UI control, etc) implemented?
What in this structure distinguishes these cases?	What is the "correct" way to use or access this data structure?
How does this data structure look at runtime?	How can data be passed to (or accessed at) this point in the code?
How is control getting (from here to) here?	Why isn't control reaching this point in the code?
Which execution path is being taken in this case?	Under what circumstances is this method called or exception thrown?
What parts of this data structure are accessed in this code?	How does the system behavior vary over these types or cases?
What are the differences between these files or types?	What is the difference between these similar parts of the code (e.g., between sets of methods)?
What is the mapping between these UI types and these model types?	How can we know this object has been created and initialized correctly?

+ +

If you think about the massive diversity in this list, you can see why program comprehension requires expertise. You not only need to understand programming languages quite well, but you also need to have strategies for answering all of the questions above (and more) quickly, effectively, and accurately.

+ +

So how do developers go about answering these questions? Studies comparing experts and novices show that experts use prior knowledge that they have about architecture, design patterns, and the problem domain a program is built for to know what questions to ask and how to answer them, whereas novices use surface features of code, which leads them to spend considerable time reading code that is irrelevant to a question ((von Mayrhauser & Vans 1994), LaToza et al. 2007). Fundamentally, reading and comprehending source code is fundamentally different from those of reading and comprehending natural language (Binkley et al. 2013); what experts are doing is ultimately reasoning about dependencies between code (Weiser 1981). Dependencies include things like data dependencies (where a variable is used to compute something, what modifies a data structure, how data flows through a program, etc.) and control dependencies (which components call which functions, which events can trigger a function to be called, how a function is reached, etc.). All of the questions above fundamentally get at different types of data and control dependencies. In fact, theories of how developers navigate code by following these dependencies are highly predictive of what information a developer will seek next (Fleming et al. 2013), suggesting that expert behavior is highly routine.

+ +

While much of program comprehension is skill, some of it is determined by design. For example, some programming languages result in programs that are more comprehensible. One framework called the Cognitive Dimensions of Notations (Green 1989) lays out some of the tradeoffs in programming language design that result in these differences in comprehensibility. For example, one of the dimensions in the framework is consistency, which refers to how much of a notation can be guessed based on an initial understanding of a language. JavaScript is a low-consistency language because of operators like ==, which behave differently depending on what the type of the left and right operands are. Knowing the behavior for Booleans doesn't tell you the behavior for a Boolean being compared to an integer. In contrast, Java is a high consistency language: == is only ever valid when both operands are of the same type.

+ +

These differences in notation have real impact. Encapsulation through data structures leads to better comprehension that monolithic or purely functional languages (Woodfield et al. 1981, Bhattacharya & Neamtiu). Declarative programming paradigms (like the JavaScript React) have greater comprehensibility than imperative languages (Salvaneschi et al. 2014). In general, languages that are statically typed result in fewer defects (Ray et la. 2014), better comprehensibility because of the ability to construct better documentation (Endrikat et al. 2014), and result in easier debugging (Hanenberg et al. 2013). In fact, studies of more dynamic languages like JavaScript and Smalltalk (Callaú et al. 2013) show that the dynamic features of these languages aren't really used all that much anywhere. It appears then that the more you tell a compiler about what your code means, the more it helps the other developers know what it means too.

+ +

Code editors, development environments, and program comprehension tools can also be helpful. Early evidence showed that simple features like syntax highlighting and careful typographic choices can improve the speed of program comprehension (Baecker 1988). I have also worked on several tools to support program comprehension, including the Whyline, which automates many of the more challenging aspects of navigating dependencies in code, and visualizes them (Ko & Myers 2009):

+ +

The path from novice to expert in program comprehension is one that involves understanding programming language semantics exceedingly well and reading a lot of code, design patterns, and architectures. Anticipate that as you develop these skills, it will take you time to build robust understandings of what a program is doing, slowing down your writing, testing, and debugging.

+ + Next chapter: Verification + +

Podcasts

+ + + +

Software Engineering Daily, Language Design with Brian Kernighan.

+ + + + + + + + + diff --git a/debugging.html b/debugging.html new file mode 100644 index 0000000..0139823 --- /dev/null +++ b/debugging.html @@ -0,0 +1,112 @@ + + + + + + + + + + + + + + + + + + Debugging + + + +

Back to table of contents

+ +

+ Credit: public domain + +

Debugging

Andrew J. Ko

+ +

Despite all of your hard work at design, implementation, and verification, your software has failed. Somewhere in its implementation there's a line of code, or multiple lines of code, that, given a particular set of inputs, causes the program to fail. How do you find those defective lines of code?

+ +

To start, you have to reproduce the failure. Failure reproduction is a matter of identifying inputs to the program (whether data it receives upon being executed, user inputs, network traffic, or any other form of input) that causes the failure to occur. If you found this failure while you were executing the program, then you're lucky: you should be able to repeat whatever you just did and identify the inputs or series of inputs that caused the problem, giving you a way of testing that the program no longer fails once you've fixed the defect. If someone else was the one executing the program (for example, a user, or someone on your team), you better hope that they reported clear steps for reproducing the problem. When bug reports lack clear reproduction steps, bugs often can't be fixed (Bettenburg et al. 2008).

+ +

If you can reproduce the problem, the next challenge is to localize the defect, trying to identify the cause of the failure in code. There are many different strategies for localizing defects. At the highest level, one can think of this process as a hypothesis testing activity (Gilmore 1991):

+ +

Observe failure
Form hypothesis of cause of failure
Devise a way to test hypothesis, such as analyzing the code you believe caused it or executing the program with the reproduction steps and stopping at the line you believe is wrong.
If the hypothesis was supported (meaning the program failed for the reason you thought it did), stop. Otherwise, return to 1.

+ +

The problems with the strategy above are numerous. First, what if you can't think of a possible cause? Second, what if your hypothesis is way off? You could spend hours generating hypotheses that are completely off base, effectively analyzing all of your code before finding the defect.

+ +

Another strategy is working backwards (Ko & Myers):

+ +

Observe failure
Identify the line of code that caused the failing output
Identify the lines of code that caused the line of code in step 2 and any data used on the line in step 2
Repeat three recursively, analyzing all lines of code for defects along the chain of causality

+ +

The nice thing about this strategy is that you're guaranteed to find the defect if you can accurately identify the causes of each line of code contributing to the failure. It still requires you to analyze each line of code and potentially execute to it in order to inspect what might be wrong, but it requires potentially less work than guessing. My doctoral dissertation work investigated how to automate this strategy, allowing you to simply click on the fault output and then immediately see all upstream causes of it (Ko & Myers).

+ +

Yet another strategy called delta debugging is to compare successful and failing executions of the program (Zeller 2002):

+ +

Identify a successful set of inputs
Identify a failing set of inputs
Compare the differences in state from the successful and failing executions
Identify a change to input that minimizes the differences in states between the two executions
Variables and values that are different in these two executions contain the defect

+ +

This is a powerful strategy, but only when you have successful inputs and when you can automate comparing runs and identifying changes to inputs.

+ +

One of the simplest strategies is to work forward:

+ +

Execute the program with the reproduction steps
Step forward one instruction at a time until the program deviates from intended behavior
This step that deviates or one of the previous steps caused the failure

+ +

This strategy is easy to follow, but can take a long time because there are so many instructions that can execute.

+ +

For particularly complex software, it can sometimes be necessary to debug with the help of teammates, helping to generate hypotheses, identify more effective search strategies, or rule out the influence of particular components in a bug (Aranda and Venolia 2009).

+ +

Ultimately, all of these strategies are essentially search algorithms, seeking the events that occurred while a program executed with a particular set of inputs that caused its output to be incorrect. Because programs execution millions and potentially billions of instructions, these strategies are necessary to reduce the scope of your search.

+ +

Once you've found the defect, what do you do? It turns out that there are usually many ways to repair a defect. How professional developers fix defects depends a lot on the circumstances: if they're near a release, they may not even fix it if it's too risky; if there's no pressure, and the fix requires major changes, they may refactor or even redesign the program to prevent the failure (Murphy-Hill et al. 2013). This can be a delicate, risky process: in one study of open source operating systems bug fixes, 27% of the incorrect fixes were made by developers who had never read the source code files they changed, suggesting that key to correct fixes is a deep comprehension of exactly how the defective code is intended to behave (Yin et al. 2011).

+ +

Podcasts

+ + + +

Software Engineering Daily, Debugging Stories with Haseeb Qureshi

+ + + + + + + + + + diff --git a/evolution.html b/evolution.html new file mode 100644 index 0000000..e94933b --- /dev/null +++ b/evolution.html @@ -0,0 +1,75 @@ + + + + + + + + + + + + + + + + + + Evolution + + + +

Back to table of contents

+ +

+ Credit: public domain + +

Evolution

Andrew J. Ko

+ +

Programs change. You find bugs, you fix them. You discover a new requirement, you add a feature. A requirement changes because the world changes, you revise a feature. The simple fact about programs are that they're rarely stable, but rather a constantly changing, living documents that shift as much as the world around them shift.

+ +

Nowhere is this constant evolution more apparent then in our daily encounters with software updates. The apps on our phones are constantly being updated to improve our experiences, while the web sites we visit potentially change every time we visit them, without us noticing. These different models have different notions of who controls changes to user experience: should software companies control when your experience changes or should you? And with systems with significant backend dependencies, is it even possible to give users control over when things change?

+ +

To manage change, developers use all kinds of tools and practices.

+ +

One of the most common ways of managing change is to refactor code. Refactoring helps developers modify the architecture of a program while keeping its behavior the same, enabling them to implement or modify functionality more easily. For example, one of the most common and simple refactorings is to rename a variable (renaming its definition and all of its uses). This doesn't change the architecture of a program at all, but does improve its readability. Other refactors can be more complex. For example, consider adding a new parameter to a function: all calls to that function need to pass that new parameter, which means you need to go through each call and decide on a value to send from that call site. Studies of refactoring in practice have found that refactorings can be big and small, that they don't always preserve the behavior of a program, and that developers perceive them as involving substantial costs and risks (Kim et al. 2012).

+ +

Another fundamental way that developers manage change is version control systems. As you know, they help developers track changes to code, allowing them to revert, merge, fork, and clone projects in a way that is traceable and reliable. While today the most popular version control system is Git, there are actually many types. Some are centralized, representing one single ground truth of a project's code, usually stored on a server. Commits to centralized repositories become immediately available to everyone else on a project. Other version control systems are distributed, such as Git, allowing one copy of a repository on every local machine. Commits to these local copies don't automatically go to everyone else; rather, they are pushed to some central copy, from which others can pull updates. Research comparing centralized and distributed revision control systems mostly reveal tradeoffs rather than a clear winner. Distributed version control, for example, appears to lead to commits that are smaller and more scoped to single changes, since developers can manage their own history of commits to their local repository (Brindescu et al. 2014). Google uses one big centralized version control repository for all of its projects, however, because it offers one source of truth, simplified dependency management, large-scale refactoring, and flexible team boundaries (Potvin & Levenberg 2016).

+ +

When code changes, you need to test it, which often means you need to build it, compiling source, data, and other resources into an executable format suitable for testing (and possibly release). Build systems can be as simple as nothing (e.g., loading an HTML file in a web browser interprets the HTML and displays it, requiring no special preparation) and as complex is hundreds and thousands of lines of build script code, compiling, linking, and managing files in a manner that prepares a system for testing, such as those used to build operating systems like Windows or Linux. To write these complex build procedures, developers use build automation tools like make, ant, gulp and dozens of others, each helping to automate builds. In large companies, there are whole teams that maintain build automation scripts to ensure that developers can always quickly build and test. In these teams, most of the challenges are social and not technical: teams need to clarify role ambiguity, knowledge sharing, communication, trust, and conflict in order to be productive, just like other software engineering teams (Phillips et al. 2014).

+ +

Perhaps the most modern form of build practice is continuous integration. This is the idea of completely automating not only builds, but also the running of a collection of tests, every time a bundle of changes is pushed to a central version control repository. The claimed benefit of continuous integration is that every major change is quickly built, tested, and ready for deployment, shortening the time between a change and the discovery of failures. This only works if builds are fast. For example, some large projects like Windows can take a whole day to build, making continuous integration of the whole operating system infeasible. When builds and tests are fast, continuous integration can accelerate development, especially in projects with large numbers of contributors (Vasilescu et al. 2015) + +

One last problem with changes in software is managing the releases of software. Good release management should archive new versions of software, automatically post the version online, make the version accessible to users, keep a history of who accesses the new version, and provide clear release notes describing changes from the previous version (van der Hoek et al. 1997). By default, all of this is quite manual, but many of these steps can be automated, streamlining how teams release changes to the world. You've probably encountered these most in the form of software updates to applications and operating systems.

+ + Next chapter: Debugging + +

Podcasts

+ + + +

Software Engineering Daily, Continuous Delivery with David Rice.

+ + + + + + + + + + diff --git a/history.html b/history.html new file mode 100644 index 0000000..bd1d60b --- /dev/null +++ b/history.html @@ -0,0 +1,82 @@ + + + + + + + + + + + + + + + + + + + The history of software engineering + + + +

Back to table of contents

+ + +

+ Credit: unknown + + +

A brief history of software engineering

Andrew J. Ko

+ +

Computers haven't been around for long. If you read one of the many histories of computing and information, such as James Gleick's The Information, or Jonathan Grudin's History of HCI, you'll learn that before digital computers, computers were people, calculating things manually. And that after digital computers, programming wasn't something that many people did. It was reserved for whoever had access to the mainframe and they wrote their programs on punchcards like the one above. Computing was in no way a ubiquitous, democratized activity—it was reserved for the few that could afford and maintain a room-sized machine.

+ +

Because programming required such painstaking planning in machine code and computers were slow, most programs were not that complex. Their value was in calculating things faster than a person could do by hand, which meant thousands of calculations in a minute rather than one calculation in a minute. Computer programmers were not solving problems that had no solutions; they were translating existing solutions (for example, a quadratic formula) into the notation a computer understood. Their power wasn't in creating new realities or facilitating new tasks, it was accelerating old tasks.

+ +

The birth of software engineering, therefore, did not come until programmers started solving problems that didn't have existing solutions, or were new ideas entirely. Most of these were done in academic contexts to develop things like basic operating systems and methods of input and output. These were complex projects, but as research, they didn't need to scale; they just needed to work. It wasn't until the late 1960's, when the first truly large software projects were attempted commercially, and software had to actually work.

+ +

The IBM 360 operating system was one of the first big projects of this kind. Suddenly, there were multiple people working on multiple components, all which interacted. Different parts of the program needed to coordinate, which meant different people needed to coordinate, and the term software engineering was born. Programmers and academics from around the world, especially those that were working on big projects, began to start conferences so they could meet and discuss their challenges. In the first software engineering conference in 1968, attendees speculated about why projects were shipping late, why they were over budget, and what to do about it.

+ +

In these early days of software engineering, programmers, managers, and researchers discovered many problems that had no clear solutions:

+ +

When you're solving a problem that doesn't yet have a solution, what is a good process for building a solution?
When software does so many different things, how can you know software "works"?
How can you make progress when no one on the team understands every part of the program?
When people leave a project, how do you ensure their replacement has all of the knowledge they had?
When no one understands every part of the program, how do you diagnose defects?
When people are working in parallel, how do you prevent them from clobbering each other's work?
If software engineering is about more than coding, what skills does a good coder need to have?
What kinds of tools and languages can accelerate a programmers work and help them prevent mistakes?

+ +

These questions are at the foundation of the field of software engineering and are the core content of this course. Some of them have pretty good answers. For example, the research community rapidly converged toward the concept of a version control systems, software testing, and a wide array of high-level programming languages such as Fortran (Metcalf 2002), LISP (McCarthy 1978), C++ (Stroustrup 1996), and Smalltalk (Kay 1996), all of which were precursors to today's modern languages such as Java, Python, and JavaScript.

+ +

Other questions, particularly those concerning the human aspects of software engineering, were hopelessly difficult to understand and improve. One of the seminal books on these issues was Fred P. Brooks, Jr.'s The Mythical Man Month. In it, he presented hundreds of claims about software engineering. For example, he hypothesized that adding more programmers to a project would actually make productivity worse at some level, not better, because of the added burden of knowledge sharing. He also claimed that the first implementation of a solution is usually terrible and should be treated like a prototype: used to learn and then discarded. These and other claims have been the foundation of decades of years of research, all in search of some deeper answer to the questions above.

+ +

If we step beyond software engineering and think more broadly about the role that software is playing in society today, there are also other, newer questions that we've only begun to answer. If every part of society now runs on code, what responsibility do software engineers have to ensure that code is right? What responsibility do software engineers have to avoid algorithmic bias? If our cars are to soon drive us around, who's responsible for the first death: the car, the driver, or the software engineers who built it? These ethical questions are in some ways the future of software engineering, likely to shape it's regulatory context, its processes, and it's responsibilities.

+ +

There are also economic roles that software plays in society that it didn't before. Around the world, it's a major source of job growth, but also a major source of automation, eliminating jobs that people used to do. These larger forces that software is playing on the world demand that software engineers have a stronger understanding of the roles that software plays in society, as the decisions that engineers make can have profoundly impactful unintended consequences.

+ +

We're nowhere close to having deep answers about these questions, neither the old ones or the new ones. We know a lot about programming languages and a lot about testing. These are areas amenable to automation and so computer science has rapidly improved and accelerated these parts of software engineering. The rest of it, as we shall see in this, has not made much progress. In this class, we'll discuss what we know and the much larger space of what we don't.

+ + Next chapter: Organizations + +

Cooperative Software Design

Andrew J. Ko

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

+ After teaching software engineering for many years, I've been frustrated by the lack of a simple, concise, and practical introduction to the human aspects of software engineering for students interested in becoming software engineers. + + In response, I've distilled my lectures from the past decade into these brief writings. They don't represent everything we know about software engineering (in particular, I don't discuss the deep technical contributions from the field), but the chapters do synthesize the broad evidence we have about how teams have to work together to succeed. + + I hope you enjoy! If you see something missing or wrong, please send me feedback. +	Chapter 1. History of software engineering
	Chapter 2. Software engineering organizations
	Chapter 3. Communication
	Chapter 4. Productivity
	Chapter 5. Software quality
	Chapter 6. Requirements engineering
	Chapter 7. Architecture
	Chapter 8. Functional specifications
	Chapter 9. Process
	Chapter 10. Comprehension
	Chapter 11. Verification
	Chapter 12. Monitoring
	Chapter 13. Evolution
	Chapter 14. Debugging

+ + + + + + + diff --git a/monitoring.html b/monitoring.html new file mode 100644 index 0000000..eb57110 --- /dev/null +++ b/monitoring.html @@ -0,0 +1,111 @@ + + + + + + + + + + + + + + + + + + + Monitoring + + + +

Back to table of contents

+ +

+ Credit: public domain + +

Monitoring

Andrew J. Ko

+ +

The first application I ever wrote was a complete and utter failure.

+ +

I was an eager eighth grader, full of wonder and excitement about the infinite possibilities in code, with an insatiable desire to build, build, build. I'd made plenty of little games and widgets for myself, but now was my chance to create something for someone else: my friend and I were making a game and he needed a tool to create pixel art for it. We had no money for fancy Adobe licenses, and so I decided to make a tool.

+ +

In designing the app, I made every imaginable software engineering mistake. I didn't talk to him about requirements. I didn't test on his computer before sending the finished app. I certainly didn't conduct any usability tests, performance tests, or acceptance tests. The app I ended up shipping was a pure expression of what I wanted to build, not what he needed to be creative or productive. As a result, it was buggy, slow, confusing, and useless, and blinded by my joy of coding, I had no clue.

+ +

Now, ideally my "customer" would have reported any of these problems to me right away, and I would have learned some tough lessons about software engineering. But this customer was my best friend, and also a very nice guy. He wasn't about to trash all of my hard work. Instead, he suffered in silence. He struggled to install, struggled to use, and worst of all struggled to create. He produced some amazing art a few weeks after I gave him the app, but it was only after a few months of progress on our game that I learned he hadn't used my app for a single asset, preferring instead to suffer through Microsoft Paint. My app was too buggy, too slow, and too confusing to be useful. I was devastated.

+ +

Why didn't I know it was such a complete failure? Because I wasn't looking. I'd ignored the ultimate test suite: my customer. I'd learned that the only way to really know whether software requirements are right is by watching how it executes in the world through monitoring.

+ +

Discovering Failures

+ +

Of course, this is easier said than done. That's because the (ideally) massive numbers of people executing your software is not easily observable. Moreover, each software quality you might want to monitor (performance, functional correctness, usability) requires entirely different methods of observation and analysis. Let's talk about some of the most important qualities to monitor and how to monitor them.

+ +

These are some of the easiest failures to detect because they are overt and unambiguous. Microsoft was one of the first organizations to do this comprehensively, building what eventually became known as Windows Error Reporting (Gelrum et al 2009). It turns out that actually capturing these errors at scale and mining them for repeating, reproducible failures is quite complex, requiring classification, progressive data collection, and many statistical techniques to extract signal from noise. In fact, Microsoft has a dedicated team of data scientists and engineers whose sole job is to manage the error reporting infrastructure, monitor and triage incoming errors, and use trends in errors to make decisions about improvements to future releases and release processes. This is now standard practice in most companies and organizations, including other big software companies (Google, Apple, IBM, etc.), as well as open source projects (eg, Mozilla). In fact, many application development platforms now include this as a standard operating system feature.

+ +

Performance, like crashes, kernel panics, and hangs, is easily observable in software, but a bit trickier to characterize as good or bad. How slow is too slow? How bad is it if something is slow occasionally? You'll have to define acceptable thresholds for different use cases to be able to identify problems automatically. Some experts in industry still view this as an art.

+ +

It's also hard to monitor performance without actually harming performance. Many tools and services (e.g., New Relic) are getting better at reducing this overhead and offering real time data about performance problems through sampling.

+ +

Monitoring for data breaches, identity theft, and other security and privacy concerns are incredibly important parts of running a service, but also very challenging. This is partly because the tools for doing this monitoring are not yet well integrated, requiring each team to develop its own practices and monitoring infrastructure. But it's also because protecting data and identity is more than just detecting and blocking malicious payloads, but also about recovering from ones that get through, developing reliable data streams about application network activity, monitoring for anomalies and trends in those streams, and developing practices for tracking and responding to warnings that your monitoring system might generate. Researchers are still actively inventing more scalable, usable, and deployable techniques for all of these activities.

+ +

Discovering Missing Requirements

+ +

Usability problems and missing features, unlike some of the preceding problems, are even harder to detect or observe, because the only true indicator that something is hard to use is in a user's mind. That said, there are a couple of approaches to detecting the possibility of usability problems.

+ +

One is by monitoring application usage. Assuming your users will tolerate being watched, there are many techniques for automatically instrumenting applications for user interaction events, for mining these events for problematic patterns, and for browsing and analyzing these patterns for more subjective issues (Ivory & Hearst 2001). Modern tools and services like Intercom make it easier to capture, store, and analyze this usage data, although they still require you to have some upfront intuition about what to monitor. More advanced, experimental techniques in research automatically analyze undo events as indicators of usability problems (Akers et al. 2009).

+ +

All of the usage data above can tell you what your users are doing, but not why. For this, you'll need to get explicit feedback from support tickets, support forums, product reviews, and other critiques of user experience. Some of these types of reports go directly to engineering teams, becoming part of bug reporting systems, while others end up in customer service or marketing departments. While all of this data is valuable for monitoring user experience, most companies still do a bad job of using anything but bug reports to improve user experience, overlooking the rich insights in customer service interactions (Chilana et al. 2011).

+ +

Although bug reports are widely used, they have significant problems as a way to monitor: for developers to fix a problem, they need detailed steps to reproduce the problem, or stack traces or other state to help them track down the cause of a problem (Bettenburg et al. 2008); these are precisely the kinds of information that are hard for users to find and submit, given that most people aren't trained to produce reliable, precise information for failure reproduction. Additionally, once the information is recorded in a bug report, even interpreting the information requires social, organizational, and technical knowledge, meaning that if a problem is not addressed soon, an organization's ability to even interpret what the failure was and what caused it can decay over time (Aranda & Venolia 2009). All of these issues can lead to intractable debugging challenges.

+ +

Larger software organizations now employ data scientists to help mitigate these challenges of analyzing and maintaining monitoring data and bug reports. Most of them try to answer questions such as (Begel & Zimmermann 2014):

+ +

"How do users typically use my application?"
"What parts of a software product are most used and/or loved by customers?"
"What are best key performance indicators (KPIs) for monitoring services?"
"What are the common patterns of execution in my application?"
"How well does test coverage correspond to actual code usage by our customers?"

+ +

The most mature data science roles in software engineering teams even have multiple distinct roles, including Insight Providers, who gather and analyze data to inform decisions, Modeling Specialists, who use their machine learning expertise to build predictive models, Platform Builders, who create the infrastructure necessary for gathering data (Kim et al. 2016). Of course, smaller organizations may have individuals who take on all of these roles.

+ +

All of this effort to capture and maintain user feedback can be messy to analyze because it usually comes in the form of natural language text. Services like AnswerDash (a company I co-founded) structure this data by organizing requests around frequently asked questions. AnswerDash imposes a little widget on every page in a web application, making it easy for users to submit questions and find answers to previously asked questions. This generates data about the features and use cases that are leading to the most confusion, which types of users are having this confusion, and where in an application the confusion is happening most frequently. This product was based on several years of research in my lab (Chilana et al. 2013).

+ + Next chapter: Evolution + +

Podcasts

+ + + +

Software Engineering Daily, Performance Monitoring with Andi Grabner

Software Engineering Daily, The Art of Monitoring with James Turnbull

Software Engineering Daily, Debugging Stories with Haseeb Qureshi + + + + + + + + + + diff --git a/organizations.html b/organizations.html new file mode 100644 index 0000000..0b37652 --- /dev/null +++ b/organizations.html @@ -0,0 +1,104 @@ + + + + + + + + + + + + + + + + + + + Software engineering organizations + + + +

Back to table of contents

+ +

+ Credit: Andrew J. Ko + +

Software organizations

Andrew J. Ko

+ +

The photo above is candid shot of some of the software engineers of AnswerDash, a company I co-founded in 2012. There are a few things to notice. First, you see one of the employees explaining something, while others are diligently working off to the side. It's not a huge team; just a few engineers, plus several employees in other parts of the organization in another room. This, as simple as it looks, is pretty much what all software engineering work looks like. Some organizations have one of these teams; others have thousands.

+ +

What you can't see is just how much complexity underlies this work. You can't see the organizational structures that exist to manage this complexity. Inside this room and the rooms around it were processes, standards, reviews, workflows, managers, values, culture, decision making, analytics, marketing, sales. And at the center of it were people executing all of these things as well as they could to achieve the organization's goal.

+ +

Organizations are a much bigger topic than I could possibly address here. To deeply understand them, you'd need to learn about organizational studies, organizational behavior, information systems, and business in general.

+ +

The subset of this knowledge that's critical to understand about software engineering is limited to a few important concepts. The first and most important concept is that even in software organizations, the point of the company is rarely to make software; it's to provide value (Osterwalder et al. 2015). Software is sometimes the central means to providing that value, but more often than not, it's the information flowing through that software that's the truly valuable piece. Requirements, which will discuss in more detail soon, help engineers organize how software will provide value.

+ +

The individuals in a software organization take on different roles to achieve that value. These roles are sometimes spread across different people and sometimes bundled up into one person, depending on how the organization is structured, but the roles are always there. Let's go through each one in detail so you understand how software engineers relate to each role. + +

Marketers look for opportunities to provide value. In for-profit businesses, this might mean conducting market research, estimating the size of opportunities, identifying audiences, and getting those audiences attention. Non-profits need to do this work as well in order to get their solutions to people, but may be driven more by solving problems than making money.
Product managers decide what value the product will provide, monitoring the marketplace and prioritizing work.
Designers decide how software will provide value. This isn't about code or really even about software; it's about envisioning solutions to problems that people have.
Software engineers write code with other engineers to implement requirements envisioned by designers. If they fail to meet requirements, the design won't be implemented correctly, which will prevent the software from providing value.
Sales takes the product that's been built and try to sell it to the audiences that marketers have identified. They also try to refine an organization's understanding of what the customer wants and needs, providing feedback to marketing, product, and design, which engineers then address.
Support helps the people using the product to use it successfully and, like sales, provides feedback to product, design, and engineering about the product's value (or lack thereof) and it's defects.

+ +

As I noted above, sometimes the roles above get merged into individuals. When I was CTO at AnswerDash, I had software engineering roles, design roles, product roles, sales roles, and support roles. This was partly because it was a small company when I was there. As organizations grow, these roles tend to be divided into smaller pieces. This division often means that different parts of the organization don't share knowledge, even when it would be advantageous (Chilana 2011).

+ +

Note that in the division of responsibilities above, software engineers really aren't the designers by default. They don't decide what product is made or what problems that product solves. They may have opinions—and a great deal of power to enforce their opinions, as the people building the product—but it's not ultimately their decision.

+ +

There are other roles you might be thinking of that I haven't mentioned:

+ +

Managers exist in all roles when teams get to a certain size, helping to move information from between higher and lower parts of an organization. Even engineering managers are primarily focused on organizing and prioritizing work, and not doing engineering.
Data scientists, although a new role, typically facilitate decision making on the part of any of the roles above (Begel & Zimmermann 2014). They might help engineers find bugs, marketers analyze data, track sales targets, mine support data, or inform design decisions. They're experts at using data to accelerate and improve the decisions made by the roles above.
Researchers, also called user researchers, also help people in a software organization make decisions, but usually product decisions, helping marketers, sales, and product managers decide what products to make and who would want them. In many cases, they can complement the work of data scientists, providing qualitative work to triangulate quantitative data.

+ +

Every decision made in a software team is under uncertainty, and so another important concept in organizations is risk (Boehm 1991). It's rarely possible to predict the future, and so organizations must take risks. Much of an organization's function is to mitigate the consequences of risks. Data scientists and researchers mitigate risk by increasing confidence in an organization's understanding of the market and it's consumers. Engineers manage risk by trying to avoid defects and moving fast.

+ +

Open source communities are organizations too. The core activities of design, engineering, and support still exist in these, but how much a community is engaged in marketing and sales depends entirely on the purpose of the community. Big, established open source projects like Mozilla have revenue, buildings, and a CEO, and while they don't sell anything, they do market. Others like Linux (Lee & Cole 2013) rely heavily on contributions both from volunteers (Ye & Kishida 2003), but also paid employees from companies that depend on Linux, like IBM, Google, and others. In these settings, there are still all of the challenges that come with software engineering, but fewer of the constraints that come from a for-profit or non-profit motive.

+ +

All of the above has some important implications for what it means to be a software engineer:

+ +

Engineers are not the only important role in a software organization. In fact, they may be less important to an organization's success than other roles because the decisions they make (how to implement requirements) have smaller impact on the organization's goals than other decisions (what to make, who to sell it to, etc.).
Engineers have to work with a lot of people working with different roles. Learning what those roles are and what shapes their success is important to being a good collaborator (Li et al. 2017).
While engineers might have many great ideas for product, if they really want to shape what they're building, they should be in a product role, not an engineering role.

+ +

All that said, without engineers, products wouldn't exist. They ensure that every detail about a product reflects the best knowledge of the people in their organization, and so attention to detail is paramount. In future chapters, we'll discuss all of the ways that software engineers manage this detail, mitigating the burden on their memories with tools and processes.

+ + Next chapter: Communication + +

Process

Andrew J. Ko

+ +

So you know what you're going to build and how you're going to build it. What process should you go about building it? Who's going to build what? What order should you build it in? How do you make sure everyone is in sync while you're building it? And most importantly, how to do you make sure you build well and on time? These are fundamental questions in software engineering with many potential answers. Unfortunately, we still don't know which of those answers are right.

+ +

At the foundation of all of these questions are basic matters of project management: plan, execute, and monitor. But developers in the 1970's and on found that traditional project management ideas didn't seem to work. The earliest process ideas followed a "waterfall" model, in which a project begins by identifying requirements, writing specifications, implementing, testing, and releasing, all under the assumption that every stage could be fully tested and verified. (Recognize this? It's the order of topics we're discussing in this class!). Many managers seemed to like the waterfall model because it seemed structured and predictable; however, because most managers were originally software developers, they preferred a structured approach to project management (Weinberg 1982). The reality, however, was that no matter how much verification one did of each of these steps, there always seemed to be more information in later steps that caused a team to reconsider it's earlier decision (e.g., imagine a customer liked a requirement when it was described in the abstract, but when it was actually built, they rejected it, because they finally saw what the requirement really meant).

+ +

In 1988, Barry Boehm proposed an alternative to waterfall called the Spiral model (Boehm 1988): rather than trying to verify every step before proceeding to the next level of detail, prototype every step along the way, getting partial validation, iteratively converging through a series of prototypes toward both an acceptable set of requirements and an acceptable product. Throughout, risk assessment is key, encouraging a team to reflect and revise process based on what they are learning. What was important about these ideas were not the particulars of Boehm's proposed process, but the disruptive idea that iteration and process improvement are critical to engineering great software.

+ +

Around the same time, two influential books were published. Fred Brooks wrote The Mythical Man Month (Brooks 1995), a book about software project management, full of provocative ideas that would be tested over the next three decades, including the idea that adding more people to a project would not necessarily increase productivity. Tom DeMarco and Timothy Lister wrote another famous book, Peopleware: Productive Projects and Teams (DeMarco 1987), arguing that the major challenges in software engineering are human, not technical. Both of these works still represent some of the most widely-read statements of the problem of managing software development processes.

+ +

These early ideas in software project management led to a wide variety of other discoveries about process. For example, organizations of all sizes can improve their process if they are very aware of what the people in the organization know, what it's capable of learning, and if it builds robust processes to actually continually improve process (Dybȧ 2002, Dybȧ 2003). This might mean monitoring the pace of work, incentivizing engineers to reflect on inefficiencies in process, and teaching engineers how to be comfortable with process change.

+ +

Beyond process improvement, other factors emerged. For example, researchers discovered that critical to team productivity was awareness of teammates' work (Ko et al. 2007). Teams need tools like dashboards to help make them aware of changing priorities and tools like feeds to coordinate short term work (Treude & Storey 2010). Moreover, researchers found that engineers tended to favor non-social sources such as documentation for factual information, but social sources for information to support problem solving (Milewski 2007). Decades ago, developers used tools like email and IRC for awareness; now they use tools like Slack, Trello, GitHub, and JIRA, which have the same basic functionality, but are much more polished, streamlined, and customizable.

+ +

In addition to awareness, ownership is a critical idea in process. This is the idea that for every line of code, someone is responsible for it's quality. The owner might be the person who originally wrote the code, but it could also shift to new team members. Studies of code ownership on Windows Vista and Windows 7 found that less a component had a clear owner, the more pre-release defects it had and the more post-release failures were reported by users (Bird et al. 2011). This means that in addition to getting code written, having clear ownership and clear processes for transfer of ownership are key to functional correctness.

+ +

Pace is another factor that affects quality. Clearly, there's a tradeoff between how fast a team works and the quality of the product it can build. In fact, interview studies of engineers at Google, Facebook, Microsoft, Intel, and other large companies found that the pressure to reduce "time to market" harmed nearly every aspect of teamwork: the availability and discoverability of information, clear communication, planning, integration with others' work, and code ownership (Rubin & Rinard 2016). Not only did a fast pace reduce quality, but it also reduced engineers' personal satisfaction with their job and their work. I encountered similar issues as CTO of my startup: while racing to market, I was often asked to meet impossible deadlines with zero defects and had to constantly communicate to the other executives in the company why this was not possible (Ko 2017).

+ +

Because of the importance of awareness and communication, the distance between teammates is also a critical factor. This is most visible in companies that hire remote developers, building distributed teams. The primary motivation for doing this is to reduce costs or gain access to engineering talent that is distant from a team's geographical center, but over time, companies have found that doing so necessitates significant investments in travel and socialization to ensure quality, minimizing geographical, temporal and cultural separation (Smite 2010). Researchers have found that there appear to be fundamental tradeoffs between productivity, quality, and/or profits in these settings (Ramasubbu et al. 2011). For example, more distance appears to lead to slower communication (Wagstrom & Datta 2014). Despite these tradeoffs, most rigorous studies of the cost of distributed development have found that when companies work hard to minimize temporal and cultural separation, the actual impact on defects was small (Kocaguneli et al. 2013). Some researchers have begun to explore even more extreme models of distributed development, hiring contract developers to complete microtasks over a few days without hiring them as employees; early studies suggest that these models have the worst of outcomes, with greater costs, poor scalability, and more significant quality issues (Stol & Fitzgerald 2014).

+ +

While all of these research was being conducted, industry explored its own ideas about process, devising frameworks that addressed issues of distance, pace, ownership, awareness, and process improvement. Extreme Programming (Beck 1999) was one of these frameworks and it was full of ideas: be iterative, do small releases, keep design simple, write unit tests, refactor to iterate, use pair programming, integrate continuously, everyone owns everything, use an open workspace, work sane hours. Beck described in his original proposal that these ideas were best for "outsourced or in-house development of small- to medium-sized systems where requirements are vague and likely to change", but as industry often does, it began hyping it as a universal solution to software project management woes and adopted all kinds of combinations of these ideas, adapting them to their existing processes. In reality, the value of XP appears to depend on highly project-specific factors (Müller & Padberk 2013), while the core ideas that industry has adopted are valuing feedback, communication, simplicity, and respect for individuals and the team (Sharp & Robinson 2004).

+ +

At the same time, Beck began also espousing the idea of "Agile" methods, which celebrated many of the values underlying Extreme Programming, such as focusing on individuals, keeping things simple, collaborating with customers, and being iterative. This idea of begin agile was even more popular and spread widely in industry and research, even though many of the same ideas appeared much earlier in Boehm's work on the Spiral model. Researchers found that Agile methods can increase developer enthusiasm (Syed-Abdulla et al. 2006), that agile teams need different roles such as Mentor, Co-ordinator, Translator, Champion, Promoter, and Terminator (Hoda et al. 2010), and that teams are combing agile methods with all kinds of process ideas from other project management frameworks such as Scrum (meet daily to plan work, plan two-week sprints, maintain a backlog of work) and Kanban (visualize the workflow, limit work-in-progress, manage flow, make policies explicit, and implement feedback loops) (Al-Baik & Miller 2015). I don't define any of these ideas here because there aren't standard definitions to share.

+ +

Ultimately, all of this energy around process ideas in industry is exciting, but disorganized. None of these efforts really get to the core of what makes software projects difficult to manage. One effort in research to get to this core by contributing new theories that explain these difficulties. The first is Herbsleb's Socio-Technical Theory of Coordination (STTC). The idea of the theory is quite simple: dependencies in engineering decisions (e.g., this function calls this other function, this data type stores this other data type) define the social constraints that the organization must solve in a variety of ways to build and maintain software (Herbsleb & Mockus 2003, Herbsleb 2016). The better the organization builds processes and awareness tools to ensure that the people who own those engineering dependencies are communicating and aware of each others' work, the fewer defects that will occur. Herbsleb referred this alignment as sociotechnical congruence, and conducted a number of studies demonstrating its predictive and explanatory power.

+ +

In my recent work (Ko 2017), I extend this idea to congruence with beliefs about product value, claiming that successful software products require the constant, collective communication and agreement of a coherent proposition of a product's value across UX, design, engineering, product, marketing, sales, support, and even customers. A team needs to achieve Herbsleb's sociotechnical congruence to have a successful product, but that alone is not enough: the rest of the organization has to have a consistent understanding of what is being built and why, even as that understanding evolves over time.

+ + Next chapter: Comprehension + +

Podcasts

+ + + +

Software Engineering Daily (2016). Git Workflows with Tim Pettersen.

Software Engineering Daily (2017). Engineering Management with Mike Borozdin.

Software Engineering Daily (2017). Tech Leadership with Jeff Norris.

+ + + + + + + + + diff --git a/productivity.html b/productivity.html new file mode 100644 index 0000000..8dbb290 --- /dev/null +++ b/productivity.html @@ -0,0 +1,84 @@ + + + + + + + + + + + + + + + + + + + Productivity + + + +

Back to table of contents

+ +

+ Credit: unknown + +

Productivity

Andrew J. Ko

+ +

When we think of productivity, we usually have a vague concept of a rate of work per unit time. Where it gets tricky is in defining "work". On an individual level, work can be easier to define, because developers often have specific concrete tasks that they're assigned. But until they're not, it's not really easy to define progress (well, it's not that easy to define "done" sometimes either, but that's a topic for a later chapter). When you start considering work at the scale of a team or an organization, productivity gets even harder to define, since an individual's productivity might be increased by ignoring every critical request from a teammate, harming the team's overall productivity.

+ +

Despite the challenge in defining productivity, there are numerous factors that affect productivity. For example, at the individual level, having the right tools can result in an order of magnitude difference in speed at accomplishing a task. One study I ran found that developers using the Eclipse IDE spent a third of their time just physically navigating between source files (Ko et al. 2005). With the right navigation aids, developers could be writing code and fixing bugs 30% faster. In fact, some tools like Mylyn automatically bring relevant code to the developer rather than making them navigate to it, greatly increasing the speed which with developers can accomplish a task (Kersten & Murphy 2006). Long gone are the days when developers should be using bare command lines and text editors to write code: IDEs can and do greatly increase productivity when used and configured with speed in mind.

+ +

That said, productivity is not just about individual developers. Because communication is a key part of team productivity, an individual's productivity is as much determined by their ability to collaborate and communicate with other developers. In a study spanning dozens of interviews with senior software engineerings, Li et al. found that the majority of critical attributes for software engineering skill (productivity included) concerned their interpersonal skills, their communication skills, and their ability to be resourceful within their organization (Li et al. 2015). Similarly, LaToza et al. found that the primary bottleneck in productivity was communication with teammates, primarily because waiting for replies was slower than just looking something up (LaToza et al. 2006). Of course, looking something up has its own problems. While StackOverflow is an incredible resource for missing documentation (Mamykina et al. 2001), it also is full of all kinds of misleading and incorrect information contributed by developers without sufficient expertise to answer questions (Barua et la. 2014). Finally, because communication is such a critical part of retrieving information, adding more developers to a team has surprising effects. One study found that adding people to a team slowly enough to allow them to onboard effectively could increase reduce defects, but adding them too fast led to increases in defects (Meneely et al. 2011).

+ +

Another dimension of productivity is learning. Great engineers are resourceful, quick learners (Li et al. 2015); new engineers must be even more resourceful, even though their instincts are often to hide their lack of expertise from exactly the people they need help from (Begel & Simon 2008). Experienced developers know that learning is important and now rely heavily on social media such as Twitter to follow industry changes, build learning relationships, and discover new concepts and platforms to learn (Singer et al. 2012).

+ +

Unfortunately, learning is no easy task. One of my earliest studies as a researcher investigated the barriers to learning new programming languages and systems, finding six distinct types of content that are challenging (Ko & Myers 2004). To use a programming platform successfully, they need to overcome design barriers, which are the abstract computational problems that must be solved, independent of the languages and APIs. They need to overcome selection barriers, which involve finding the right abstractions or APIs to achieve the design they have identified. They need to overcome use and coordination barriers, which involve operating and coordinating different parts of a language or API together to achieve novel functionality. They need to overcome comprehension barriers, which involve knowing what can go wrong when using part of a language or API. And finally, they need to overcome information barriers, which are posed by the limited ability of tools to inspect a program's behavior at runtime during debugging. Every single one of these barriers has its own challenges, and developers encounter them every time they are learning a new platform, regardless of how much expertise they have.

+ +

Aside from individual and team factors, productivity is also influenced by the particular features of a project's code and how the project is managed (Vosburgh et al. 1984, DeMarco & Lister 1985). In fact, these might actually be the biggest factors in determining developer productivity. This means that even a developer that is highly productive individually cannot rescue a team that is poorly structured working on poorly architected code. This might be why highly productive developers are so difficult to recruit to poorly managed teams.

+ + Next chapter: Quality + +

Podcasts

+ + + +

Software Engineering Daily, Reflections of an Old Programmer

Software Engineering Daily, Hiring Engineers with Ammon Bartram

+ + + + + + + + + diff --git a/quality.html b/quality.html new file mode 100644 index 0000000..d82ef71 --- /dev/null +++ b/quality.html @@ -0,0 +1,124 @@ + + + + + + + + + + + + + + + + + + + Software Quality + + + + +

Back to table of contents

+ +

+ Credit: Anton Croos + +

Software Quality

Andrew J. Ko

+ +

There are numerous ways a software project can fail: projects can be over budget, they can ship late, they can fail to be useful, or they can simply not be useful enough. Evidence clearly shows that success is highly contextual and stakeholder-dependent: success might be financial, social, physical and even emotional, suggesting that software engineering success is a multifaceted variable that cannot explained simply by user satisfaction, profitability or meeting requirements, budgets and schedules (Ralph & Kelly 2014).

+ +

One of the central reasons for this is that there are many distinct software qualities that software can have and depending on the stakeholders, each of these qualities might have more or less importance. For example, a safety critical system such as flight automation software should be reliable and defect-free, but it's okay if it's not particularly learnable—that's what training is for. A video game, however, should probably be fun and learnable, but it's fine if it ships with a few defects, as long as they don't interfere with fun (Murphy-Hill et al. 2014).

+ +

There are a surprisingly large number of software qualities (Boehm 1976):

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Correctness	The extent to which a program behaves according to its specification. If your specifications are ambiguous, correctness is ambiguous.
Reliability	The extent to which a program behaves the same way over time. If your online banking app crashes sometimes, it's not reliable. (It's probably also not correct, unless it's specification said that it should crash randomly).
Robustness	The extent to which a program behaves similarly in different operating environments. A touch screen is less robust because it stops working consistently in the rain. Web forms that reject my last name as invalid are not robust.
Performance	The extent to which a program uses computing resources economically. Synonymous with "fast" and "zippy". Performance is directly determined by how many instructions a program has to execute to accomplish it's operations, but it is difficult to measure because operations, inputs, and the operating environment can vary widely.
Learnability	The ease with which a person can learn to operate a program. Learnability is multi-dimensional and can be difficult to measure (Grossman et al. 2009)
User efficiency	The speed with which a person can perform tasks with a program. For example, think about how many taps and keystrokes it takes you to log in to an app on your phone compared to using a fingerprint sensor like Apple's TouchID.
Accessibility	The diversity of physical or cognitive abilities that can successfully operate software. For example, something that can only be used with a mouse is less accessible than something that can be used with a mouse, keyboard, or speech. Software can be designed for all abilities, and even automatically adapted for individual abilities (Wobbrock et al. 2011).
Usefulness	The extent to which software solves a problem. Utility is often the most important quality because it subsumes all of the other lower-level qualities software can have (e.g., part of what makes a messaging app useful is that it's performant, user efficient, and reliable). That also makes it less useful as a concept, because it can be so difficult to measure for most problems. That said, usefulness is not always the most important quality. For example, if you can sell a product to a customer and get a one time payment of their money, it might not matter that the product has low usefulness.
Verifiability	The effort required to verify that software does what it is intended to do. For example, it is hard to verify a safety critical system without either proving it correct or testing it in a safety-critical context (which isn't safe). Take driverless cars, for example: for Google to test their software, they've had to set up thousands of paid drivers to monitor and report problems on the road. In contrast, verifying that a simple static HTML web page works correctly is as simple as opening it in a browser.
Maintainability	The extent to which software can be corrected, adapted, or perfected. This depends mostly on how comprehensible the implementation of a program is.
Reusability	The extent to which a program's components can be used for unintended purposes. APIs are quite reusable, whereas black box embedded software (like the software built into your car's traction systems) is not.
Portability	The extent to which an implementation can run on different platforms and environments
Interoperability	The extent to which a system uses standard interfaces.
Security	The extent to which a system prevents access to information that is restricted to a certain population

+ +

Although the list above is not complete, you might have already noticed some tradeoffs between different qualities. A secure system is necessarily going to be less learnable, because there will be more to learn to operate it. A robust system will likely be less maintainable because it it will likely have more code to account for its diverse operating environments. Because one cannot achieve all software qualities, and achieving each quality takes significant time, it is necessary to prioritize qualities for each project.

+ +

These external notions of quality are not the only qualities that matter. For example, developers often view projects as successful if they offer intrinsically rewarding work (Procaccino et al. 2005). That may sound selfish, but if developers aren't enjoying their work, they're probably not going to achieve any of the qualities very well. Moreover, there are many organizational factors that can inhibit developers' ability to obtain these rewards. Project complexity, internal and external dependencies that are out of a developers control, process barriers, budget limitations, deadlines, poor HR planning, and pressure to ship can all interfere with project success (Lavallee & Robillard 2015).

+ +

As I've noted before, the person most responsible for isolating developers from these organizational problems, and most responsible for prioritizing software qualities is a product manager. Check out the podcast below for one product manager's perspectives on the challenges of balancing these different priorities.

+ + Next chapter: Requirements + +

Podcasts

+ +

Software Engineering Daily, Product Management with Suzie Prince

+ + + + + + + diff --git a/requirements.html b/requirements.html new file mode 100644 index 0000000..8600641 --- /dev/null +++ b/requirements.html @@ -0,0 +1,79 @@ + + + + + + + + + + + + + + + + + + Requirements + + + +

Back to table of contents

+ +

+ Credit: public domain + +

Requirements

Andrew J. Ko

+ +

Once you have a problem, a solution, and a design specification, it's entirely reasonable to start thinking about code. What libraries should we use? What platform is best? Who will build what? After all, there's no better way to test the feasibility of an idea than to build it, deploy it, and find out if it works. Right?

+ +

It depends. This mentality towards product design works fine with building and deploying something is cheap and getting feedback has no consequences. Simple consumer applications often benefit from this simplicity, especially early stage ones, because there's little to lose. But what if a beta isn't cheap to build? What if your product only has one shot at adoption? What if you're building something for a client and they want to define success? Worse yet, what if your product could kill people if it's not built properly? In these settings, software teams take an approach of translating a design into a specific explicit set of goals that must be satisfied in order for the implementation to be complete. We call these goals requirements and we call this process of requirements engineering (Sommerville & Sawyer 1997).

+ +

The relationship between requirements and design is somewhat murky. In design disciplines, designers tend to express requirements in the form of prototypes and mockups. These implicitly state requirements, because they suggest what the software is supposed to do without saying it directly. For some types of requirements, they actually imply nothing. For example, how responsive should a web page be to be? A prototype doesn't really say; an explicit requirement of an average page load time of less than 1 second is quite explicit. Requirements can therefore be thought of more like an architect's blueprint: they provide explicit definitions and scaffolding of project success.

+ +

And yet, like design, requirements come from the world and the people in it and not from software (Jackson 2001). Therefore, the methods that people use to do requirements engineering are quite similar to design methods. Requirements engineers do interviews, conduct user research, create prototypes, and iteratively converge toward requirements (Lamsweerd 2008). The big difference between design and requirements engineering is that requirements engineers take the process one step further than designers, enumerating in detail every property that the software must satisfy. They sometimes even use formal methods to specify requirements, allowing them to automatically identify conflicting requirements, so they don't end up proposing a design that can't possibly exist. Some even use systems to make requirements "traceable", meaning the high level requirement can be linked directly to the code that meets that requirement (Mader & Egyed 2015). All of this formality has tradeoffs: not only does it take more time to be so precise, but it can negatively effect creativity in concept generation as well (Mohanani et al. 2014).

+ +

Expressing requirements in natural language can mitigate these effects, at the expense of precision. They just have to be complete, precise, non-conflicting, and verifiable. For example, consider a design for a simple to do list application. It's requirements might be something like the following:

+ +

Users must be able to add to do list items with a single action.
To do list items must consist of text and a binary completed state.
Users must be able to edit to do list item text.
Users must be able to toggle the completed state.
Users must be able to delete to do list items.
All edits to do list item state must save without user intervention.

+ +

Let's review these requirements against the criteria for good requirements that I listed above:

+ +

Is it complete? I can think of a few more requirements: is the list ordered? How long does state persist? Are there user accounts? Where is data stored? What does it look like? What kinds of user actions must be supported? Is delete undoable? Even just on these completeness dimension, you can see how even a very simple application can become quite complex. When you're generating requirements, your job is to make sure you haven't forgotten important requirements.
Is the list precise? Not really. When you add a to do list item, is it added at the beginning? The end? Wherever a user request it be added? How long can the to do list item text be? Clearly the requirement above is ambiguous.
Are the requirements non-conflicting? I think they are since they all seem to be satisfiable together. But some of the missing requirements might conflict: we can't know until we're sure our list is relatively complete.
Finally, are they verifiable? Some more than others. Is there a way to guarantee that the state saves successfully all the time? That may be difficult to prove given the vast number of ways the operating environment might prevent saving.

+ +

Now, the flaws above don't make the requirements "wrong". They just make them "less good." The more complete, precise, non-conflicting, and testable your requirements are, the easier it is to anticipate risk, estimate work, and evaluate progress, since requirements essentially give you a to do list for building and testing your code.

+ + Next chapter: Architecture + +

Functional Specifications

Andrew J. Ko

+ +

When you make something with code, you're probably used to figuring out a design as you go. You write a function, you choose some arguments, and if you don't like what you see, perhaps you add a new argument to that function and test again. This cowboy coding as some people like to call it can be great fun! It allows systems to emerge more organically, as you iteratively see your front-end design emerge, the design of your implementation emerges too, co-evolving with how you're feeling about the final product.

+ +

As you've probably noticed by now, this type of process doesn't really scale, even when you're working with just a few other people. That argument you added? You just broke a bunch of functions one of your teammates was planning and when she commits her code, now she gets merge conflicts, which cost her an hour to fix because she has to catch up to whatever design change you made. This lack of planning quickly turns into an uncoordinated mess of individual decision making. Suddenly you're spending all of your time cleaning up coordination messes instead of writing code.

+ +

The techniques we've discussed so far for avoiding this boil down to specifying what code should do, so everyone can write code according to a plan. We've talked about requirements specifications, which are declarations of what software must do from a users' perspective. We've also talked about architectural specifications, which are high-level declarations of how code will be organized, encapsulated, and coordinated. At the lowest level are functional specifications, which are declarations about the properties of input and output of functions in a program. + +

In their simplest form, a functional specification can be just some natural language that says what a function is supposed to do:

+ +

+// Return the smaller of the two numbers, or if they're equal, the second number.
+function min(a, b) {
+	return a < b ? a : b;
+}		
+

+ +

This comment achieves the core purpose of a specification: to help other developers understand what the requirements and intended behavior of a function are. As long as everyone sticks to this "plan" (everyone calls the function with only numbers and the function always returns the smaller of them), then there shouldn't be any problems.

+ +

The comment above is okay, but it's not very precise. It says what is returned and what properties it has, but it only implies that numbers are allowed without saying anything about what kind of numbers. Are decimals allowed or just integers? What about not-a-number (the result of dividing 1 by 0). Or infinity?

+ +

To make these clearer, many languages use static typing to allow developers to specify types explicitly:

+ +

+// Return the smaller of the two integers, or if they're equal, the second number.
+function min(int a, int b) {
+	return a < b ? a : b;
+}		
+

+ +

Because an int is well-defined in most languages, the two inputs to the function are well-defined.

+ +

Of course, if the above was JavaScript code (which doesn't support static typing), JavaScript does nothing to actually verify that the data given to min() are actually integers. It's entirely fine with someone sending a string and an object. This probably won't do what you intended, leading to defects.

+ +

This brings us to a second purpose of writing functional specifications: to help verify that functions, their input, and their output are correct. There are many ways to use specifications to verify correctness. By far, one of the most widely used is assertions (Clarke & Rosenblum 2006). Assertions consist of two things: 1) a check on some property of a function's input or output and 2) some action to notify about violations of these properties. For example, if we wanted to verify that the JavaScript function above had integer values as inputs, we would do this:

+ +

+// Return the smaller of the two numbers, or if they're equal, the second number.
+function min(a, b) {
+	if(!Number.isInteger(a)) alert("First input to min() isn't an integer!");
+	if(!Number.isInteger(b)) alert("Second input to min() isn't an integer!");
+	return a < b ? a : b;
+}		
+

+ +

These two new lines of code are essentially functional specifications that declare "If either of those inputs is not an integer, the caller of this function is doing something wrong". This is useful to declare, but assertions have a bunch of problems: if your program can send a non-integer value to min, but you never test it in a way that does, you'll never see those alerts. This form of dynamic verification is therefore very limited, since it provides weaker guarantees about correctness. That said, a study of the use of assertions in a large database of GitHub projects shows that use of assertions is related to fewer defects (Casalnuovo et al. 2015) (though note that I said "related": we have no evidence that assertions actually prevent defects. It may be possible that developers who use assertions are just better at avoiding defects.)

+ +

Assertions are related to the broader category of error handling language features. Error handling includes assertions, but also programming language features like exceptions and exception handlers. Error handling is a form of specification in that checking for errors usually entails explicitly specifying the conditions that determine an error. For example, in the code above, the condition Number.isInteger(a) specifies that the parameter a must be an integer. Other exception handling code such as the Java throws statement indicates the cases in which errors can occur and the corresponding catch statement indicates what is to done about errors. It is difficult to implement good exception handling that provides granular, clear ways of recovering from errors (Chen et al. 2009). Evidence shows that modern developers are still exceptionally bad at designing for errors; one study found that errors are not designed for, few errors are tested for, and exception handling is often overly general, providing little ability for users to understand errors or for developers to debug them (Ebert et al. 2015). These difficulties appear to be because it difficult to imagine the vast range of errors that can occur (Maxion & Olszewski 2000).

+ +

Researchers have invented many forms of specification that require more work and more thought to write, but can be used to make stronger guarantees about correctness (Woodcock et al. 2009). For example, many languages support the expression of formal pre-conditions and post-conditions that represent contracts that must be kept. Because these contracts are essentially mathematical promises, we can build tools that automatically read a function's code and verify that what it computes exhibits those mathematical properties using automated theorem proving systems. For example, suppose we wrote some formal specifications for our example above to replace our assertions (using a fictional notation for illustration purposes):

+ +

+// pre-conditions: a in Integers, b in Integers
+// post-conditions: result <= a and result <= b
+function min(a, b) {
+	return a < b ? a : b;
+}		
+

+ +

The annotations above require that, no matter what, the inputs have to be integers and the output has to be less than or equal to both values. The automatic theorem prover can then start with the claim that result is always less than or equal to both and begin searching for a counterexample. Can you find a counterexample?

+ +

There are definite tradeoffs with writing detailed, formal specifications. The benefits are clear: many companies have written formal functional specifications in order to make completely unambiguous the required behavior of their code, particularly systems that are capable of killing people or losing money, such as flight automation software, banking systems, and even compilers that create executables from code (Woodcock et al. 2009). In these settings, it's worth the effort of being 100% certain that the program is correct because if it's not, people can die.

+ +

When the consequences aren't so high, other factors dominate: writing functional specifications is very hard and very time consuming, you need tools to verify the annotations themselves, and you have to maintain annotations. These barriers deter many developers from writing them (Schiller et al. 2014). Some forms of specifications, like the UML diagrams we described when discussing architecture, lack the benefits of formal specifications and require a lot of work to create, leading many practitioners to find them not worth the effort (Petre 2013).

+ +

Specifications can have other benefits. The very act of writing down what you expect a function to do in the form of test cases can slow developers down, causing to reflect more carefully and systematically about exactly what they expect a function to do (Fuci et al. 2016). Perhaps if this is true in general, there's value in simply stepping back before you write a function, mapping out pre-conditions and post-conditions in the form of simple natural language comments, and then writing the function to match your intentions.

+ + Next chapter: Process + +

Verification

Andrew J. Ko

+ +

How do you know a program does what you intended? Part of this is being clear about what you intended (by writing specifications, for example), but your intents, however clear, are not enough: you need evidence that your intents were correctly expressed computationally. To get this evidence, we do verification.

+ +

There are many ways to verify code. A reasonable first instinct is to simply run your program. After all, what better way to check whether you expressed your intents then to see with your own eyes what your program does? This is an empirical approach is called testing. Some testing is manual, in that a human executes a program and verifies that it does what was intended. Some testing is automated, in that the test is run automatically by a computer. Another way to verify code is to analyze it, using logic to verify its correct operation. As with testing, some analysis is manual, since humans do it. We call this manual analysis inspection, whereas other analysis is automated, since computers do it. We call this program analysis. This leads to a nice complementary set of verification technique along two axes:

+ + + + + + + + + + + + + + + + + +

	manual	automatic
empirical	manual testing	automated testing
analytical	inspection	program analysis

+ +

To discuss each of these and their tradeoffs, first we have to cover some theory about verification. The first and simplest ideas are some terminology:

+ +

A defect is some subset of a program's code that exhibits behavior that violates a program's specifications. For example, if a program was supposed to sort a list of numbers in increasing order and print it to a console, but a flipped inequality in the sorting algorithm made it sort them in decreasing order, the flipped inequality is the defect.
A failure is the program behavior that results from a defect executing. In our sorting example, the failure is the incorrectly sorted list printed on the console.
A bug vaguely refers to either the defect, the failure, or both. When we say "bug", we're not being very precise, but it is a popular shorthand for a defect and everything it causes.

+ +

Note that because defects are defined relative to intent, whether a behavior is a failure depends entirely the definition of intent. If that intent is vague, whether something is a defect is vague. Moreover, you can define intents that result in behaviors that seem like failures: for example, I can write a program that intentionally crashes. A crash isn't a failure if it was intended! This might be pedantic, but you'd be surprised how many times I've seen professional developers in bug triage meetings say:

+ +

"Well, it's worked this way for a long time, and people have built up a lot of workarounds for this bug. It's also really hard to fix. Let's just call this by design. Closing this bug as won't fix."

+ +

Testing

+ +

So how do you find defects in a program? Let's start with testing. Testing is generally the easiest kind of verification to do, but as a practice, it has questionable efficacy. Empirical studies of testing find that it is related to fewer defects in the future, but not strongly related, and it's entirely possible that it's not the testing itself that results in fewer defects, but that other activities (such as more careful implementation) result in fewer defects and testing efforts (Ahmed et al. 2016). At the same time, modern developers don't test as much as they think they do (Beller et al. 2015). Moreover, students are often not convinced of the return on investment of automated tests and often opt for laborious manual tests (even though they regret it later) (Pham et al. 2014). Testing is therefore in a strange place: it's a widespread activity in industry, but it's not very systematically, and it doesn't seem to help very much when we try to measure its benefits.

+ +

Why is this? One possibility is that no amount of testing can prove a program correct with respect to its specifications. Why? It boils down to the same limitations that exist in science: with empiricism, we can provide evidence that a program does have defects, but we can't provide complete evidence that a program doesn't have defects. This is because even simple programs can execute in a infinite number of different ways.

+ +

Consider this JavaScript program:

+ +

+function count(input) {
+	while(input > 0)
+		input--;
+	return input;
+}

+ +

The function should always return 0, right? How many possible values of input do we have to try manually to verify that it always does? Well, if input is an integer, then there are 2³² possible integer values. That's not infinite, but that's a lot. But what if input is a string? There are an infinite number of possible strings because they can have any sequence of characters of any length. Now we have to manually test an infinite number of possible inputs. So if we were restricting ourselves to testing, we will never know that the program is correct for all possible inputs. In this case, automatic testing doesn't even help, since there are an infinite number of tests to run.

+ +

There are some ideas in testing that can improve how well we can find defects. For example, rather than just testing the inputs you can think of, focus on all of the lines of code in your program. If you find a set of tests that can cause all of the lines of code to execute, you have one notion of test coverage. Of course, lines of code aren't enough, because an individual line can contain multiple different paths in it (e.g., value ? getResult1() : getResult2()). So another notion of coverage is executing all of the possible control flow paths through the various conditionals in your program. Executing all of the possible paths is hard, of course, because every conditional in your program doubles the number of possible paths (you have 200 if statements in your program? That's 2²⁰⁰ possible paths, which is more paths than there are atoms in the universe).

+ +

There are many types of testing that are common in software engineering:

+ +

Unit tests verify that low-level functions return the correct output. For example, a program that implemented a function for finding the day of the week for a given date might also include unit tests that verify for a large number of dates that the correct day of the week is returned. They're good for ensuring widely used low-level functionality is correct.
Integration tests verify that when all of the functionality of a program is put together into the final product, it behaves according to specifications. Integration tests often operate at the level of user interfaces, clicking buttons, entering text, submitting forms, and verifying that the expected feedback always occurs. Integration tests are good for ensuring that important tasks that users will perform are correct.
Regression tests verify that previously fixed defects do not break again. For example, imagine you find a defect that causes logins to fail; you might write a test that verifies that this cause of login failure does not occur, in case someone breaks the same functionality again, even for a different reason. Regression tests are good for ensuring that you don't break things when you make changes to your application.

+ +

Which tests you should write depends on what risks you want to take. Don't care about failures? Don't write any tests. If failures of a particular kind are highly consequential to your team, you should probably write tests that check for those failures. As we noted above, you can't write enough tests to catch all bugs, so deciding which tests to write and maintain is a key challenge.

+ +

Analysis

+ +

Now, you might be thinking that it's obvious that the program above is defective for some integers and strings. How did you know? You analyzed the program rather than executing it with specific inputs. For example, when I read (analyzed) the program, I thought:

+ +

"if we assume input is an integer, then there are only three types of values to meaningfully consider with respect to the > in the loop condition: positive, zero, and negative. Positive numbers will always decrement to 0 and return 0. Zero will return zero. And negative numbers just get returned as is, since they're less then zero, which is wrong with respect to the specification. And in JavaScript, strings are never greater than 0 (as if that even makes sense), so the string is returned, which is wrong."

+ +

The above is basically an informal proof, using logic to divide the possible states of input and their effect on the program's behavior. It used symbolic execution to verify all possible paths through the function, finding the paths that result in correct and incorrect values. The strategy was an inspection because we did it manually. If we had written a program that read the program to perform this deduction automatically, we would have called it program analysis.

+ +

The benefits of analysis is that it can demonstrate that a program is correct in all cases. This is because they can handle infinite spaces of possible inputs by mapping those infinite inputs onto a finite space of possible executions. It's not always possible to do this in practice, since many kinds of programs can execute in infinitely possible ways, but it gets us closer.

+ +

One popular type of automatic program analysis tools is a static analysis tool. These tools read programs and identify potential defects using the types of formal proofs like the ones above. They typically result in a set of warnings, each one requiring inspection by a developer to verify, since some of the warnings may be false positives (something the tool thought was a defect, but wasn't). Although static analysis tools can find all kinds of defects, they aren't yet viewed by developers to be that useful because the false positives are often large in number and the way they are presented make them difficult to understand (Johnson et al. 2013).

+ +

Not all analytical techniques rely entirely on logic. In fact, one of the most popular methods of verification in industry are code reviews, also known as inspections. The basic idea of an inspection is to read the program analytically, following the control and data flow inside the code to look for defects. This can be done alone, in groups, and even included as part of process of integrating changes, to verify them before they are committed to a branch. Modern code reviews, while informal, help find defects, stimulate knowledge transfer between developers, increase team awareness, and help identify alternative implementations that can improve quality (Bacchelli & Bird 2013). One study found that measures of how much a developer knows about an architecture can increase 66% to 150% depending on the project (Rigby & Bird 2013). That said, not all reviews are created equal: the best ones are thorough and conducted by a reviewer with strong familiarity with the code (Kononenko et al. 2016); including reviewers that do not know each other or do not know the code can result in longer reviews, especially when run as meetings (Seaman & Basili 1997). Soliciting reviews asynchronously by allowing developers to request reviewers of their peers is generally much more scalable (Rigby & Storey 2011), but this requires developers to be careful about which reviews they invest in.

+ +

Beyond these more technical considerations around verifying a program's correctness are organizational issues around different software qualities. For example, different organizations have different sensitivities to defects. If a $0.99 game on the app store has a defect, that might not hurt its sales much, unless that defect prevents a player from completing the game. If Boeing's flight automation software has a defect, hundreds of people might die. The game developer might do a little manual play testing, release, and see if anyone reports a defect. Boeing will spend years proving mathematically with automatic program analysis that every line of code does what is intended, and repeating this verification every time a line of code changes. What type of verification is right for your team depends entirely on what you're building, who's using it, and how they're depending on it.

+ + Next chapter: Monitoring + +

Architecture

Further reading

Podcasts

Communication

Further reading

Podcasts

Program Comprehension

Further reading

Podcasts

Debugging

Further reading

Podcasts

Evolution

Further reading

Podcasts

A brief history of software engineering

Further reading

Cooperative Software Design

Monitoring

Discovering Failures

Discovering Missing Requirements

Further reading

Podcasts

Software organizations

Further reading

Process

Further reading

Podcasts

Productivity

Further reading

Podcasts

Software Quality

Further reading

Podcasts

Requirements

Further reading

Functional Specifications

Further reading

Verification

Testing

Analysis

Further reading