Can inversion of control go awry?

Is there any downside to using inversion of control? Absolutely

Can the testing code become a disaster? Totally

Can you create a big mess of tangled code with inversion of control? Daily

Is it harder to read code that has been “distilled” so that compile time and runtime dependencies are different? Yep

So what is the conclusion? We need to train our brains to think in new ways and build/find tools to help us write code that supports automated testing (and therefore works and can be maintained).

We need to get better at distilling out the un-testable parts. We need to learn how to test more un-testable code. We need to learn better patterns of clearly doing inversion of control. We need to train our brains to think in different ways. We need to keep groping towards a better way of building software.

Distill code to make it testable

I am adding a small module to a big application. If I add the module in the default way it won’t be easy to test. The default way is to simply slap the new code onto the existing app and stir it all together nicely. If I want to test the new code I must run the entire application. And that is hard to do in an automated way.

To make this example concrete, let’s say I need to add table support to a word processor. The default approach is to just start adding code to the word processor code until I have tables working. But when I am done, the “table module” is very tightly integrated with the word processor app.

Here is an alternative approach: There are two kinds of code that are mixed into the new module that must be distilled out. The first, is application specific code. Find all the parts that are specific to this use of the new module and pull them together at the top. This is the common approach of making a library. This involves a bit of abstraction, pulling out constants and other details that apply to the application. So I make a table library and then use that table library in the application.

This part is fairly well understood, if not followed. The second aspect is much less well understood. The second kind of code that must be distilled out is system access code. System access code is any code that goes outside of memory and touches real resources. For example, reading a file, talking on the network, accessing a database, reading the system clock. This is all system access code that is harder to test than normal code.

Just as I distilled the application code out to the top of the new module, I need to distill the system access code out to the bottom of the new module. So imagine different pieces of code working together: the application code is on top (the word processor), making calls down to the new module (the table module), which in turn is calling bits of system access code that are plugged in underneath it.

The final step is to use inversion of control, to allow the application code to pass the system access code into the module. From an object construction perspective this pulls the system access code up on top of the module and puts it under the control of the application. (This point is complicated unless you understand inversion of control. The module still makes calls down to the system access code, but the system access code is constructed by the application code. So there is a runtime dependency from the module to the system access code. But, the system access code implements interfaces defined in the module, so the compile time dependencies are such that the system access code depends on the module, not vice-versa).

With inversion of control in place, I can create fake system access code that is just normal code (i.e. only uses memory, does not access other system resources). This makes it easy to test. For example, suppose the table module in the word processor needs to read a config file to know how many columns to create by default. With the file access code distilled out of the module, I can write simple automated tests that give the module different strings as “config files”.

To do this really well I want to only distill out pure application code and pure system access code. I want the distilled parts to be as small as possible. Why? Because they are going to be harder to test. I am going to put the module under extensive automated testing. So the more code that is in the module, the more code that will be tested. Which means: the more code that will work.

This approach also simplifies the task of automated testing because everything that must be faked for the automated test is gathered together at the top of the library (remember the system access code is “on top” with the application code from a compile time perspective). This means I can have a nice neat bit of code that “fakes” the system access, and then everything below that is the “real” module running.

So, to make my modules testable I need to keep them free of application code and free of system access code. I use inversion of control to allow the application code to control what system access code to use.

Requirements, people, and monsters (part 4)

The final complex system is the application we build: the system. Over time the system becomes a monster. A monster that threatens the team by causing damage, demanding attention, creating more urgent work, growing out of control, refusing to cooperate, and generally causing pain for the development team. The source code grows into a monster. The running system grows into another kind of monster.

The task is to tame the beast. The monster is supposed to serve the team and the users, not the other way around. We need to get the beast in a cage, tame him, get a bit in his mouth, and steer him where we want. We must make the monster serve people.

Once again, the agile movement shows us many of the key techniques we need. Create automated unit tests around each piece of the system. Create automated end-to-end functional tests that confirm the whole thing works as expected. These tests create a cage that constrains the monster.

Build with an eye to creating visible workings. The system cannot be a black box, it has to show its users what it is doing. The system has to provide useful logging and monitoring. Now people can reason about its behavior rather than making up superstitions to explain the rampages of the beast.

With extensive tests in place the development team has a safety net that emboldens them to keep the design from deteriorating into a big ball of mud. When a coder is working on a piece of code and when they see how terrible the code is they can make it better and count on the tests to help them keep things working. The code base can be steered in the direction of good design. The monster has a bit in his mouth.

Software development projects are dominated by these three complex systems: the requirements, the team, and the application itself. Each one of these offers endless opportunities for learning. Any one of them can run out of control and cause misery. Welcome to the joyful world of software development.

Requirements, people, and monsters (part 3)

The second major system in a development effort is the team. The key distinguishing characteristic of the team is: it is composed of people. People in all of their irrational, emotional, egocentric, and goofy glory.

People are not parts. People are not resources. People do not equate simply to developer hours.

You can’t even get along with the people in your family. How are you ever going to get along with the people on your development team?

For some the answer is a stern “professionalism” that pursues an illusion of the lack of any kind of emotion and requires team members to do the same. Thus sweeping their people-ness under the rug.

For others the solution is a harsh command-and-control environment. “I am the boss, you will do what I say. I don’t care about your personal issues.”

The stereotype of computer guys is that we like “hard science”: numbers, code, engineering. Not soft things like: psychology and people and emotions. Then one day you realize the people around you form the most complex system you will ever have to work with. People are the ultimate computers. People are the most complex software system we will ever deal with. The interactions between people form the most complex systems we will ever see.

But more than that, people are the most significant things we will encounter in our computer careers.

When you realize these things you will be driven out of your computer science hole and into the field of learning about and trying to understand people. Here are some key lessons about people.

People are all the same

Everyone else is just like you. We look at another person. We have a natural tendency to think we are categorically different than they. The stupid manager who has no idea what the code is doing. Our first thought is that they are a different type of person than us. The reality is they are the same type of person as us. The fact that they look stupid to us is more about where the two of us happened to meet. We meet after the manager had gone down his management path and I down my coder path. The difference between us is based on relation to each other, not fundamental differences in the kind of person he is and the kind of person I am. In your future you will go down a similar path and be viewed in a similar light.

But, even if you never become a manager, someone will accuse you of being a stupid manager some day. In the same way that politics works. Someone is a staunch liberal all of their life. Then they say something vaguely pro-business. Suddenly they are labeled a right-wing conservative.

With that in mind, consider:

The idiot programmer who can’t figure out the API. That is you.

The slacker who quits working and leaves you hanging when the deadline arrives. That is you.

The stupid manager who is blind to what is really happening. That is you.

The hotshot kid who thinks he knows everything. That is you.

The inept programmer who created an unmaintainable mess. That is you.

The clueless project manager who is completely. That is you.

Those are all you in the sense that they are people like you who are dealing with their circumstances and trying to get through the day as best they can. Someday you will look like them to someone else.

People are all different

You learn a bit about people and you start to categorize them. People hate to be categorized (at least I do!).

People hate to be labeled.

People hate to be psycho-analyzed.

People hate to feel like someone else has them “figured out”.

It is dehumanizing. Like reducing a person down to a “database administrator”. When you pigeon-hole someone you don’t pay due respect to their people-ness.

We are all unique and must be dealt with as individuals, not just as members of some categories.

The agile movement also brought these issues to the fore.

…we have come to value: Individuals and interactions over processes and tools…

Agile brought us daily meetings where you talk to each other. Agile brought us the idea of developers estimating their own work and signing up for their own tasks. Humanizing ideas.

To summarize then:

• People are the core of the development process
• People are the most complicated and unpredictable system in the process
• People are the most significant things we deal with
• (Oh.. and we all are one)

So go learn about people. Maybe read a classic like Peopleware. Or learn from a master like Jerry Weinberg and books like:

• The Secrets of Consulting
• Becoming a Technical Leader
• More Secrets of Consulting

WARNING: These are not your typical computer books. They may take you far outside of your comfort zone.

Requirements, people, and monsters (part 2)

Requirements. There is an ever growing, massive pile of requirements. This pile threatens to crush the development team under its weight. The agile movement taught us how to deal with this. Well… before that life taught us how to deal with this… one step at a time. The key to dealing with the pile of requirements is focus.

Planning Game

The planning game is one of the main ways to tame the requirements. First it does this by breaking the pile down into a few piles. Talk with the project sponsor about the requirements. Divide them into three buckets:

• items essential for launch

• items that are very important

• items that can be postponed

The names of these buckets are adjusted to accommodate your current project sponsor. The only important thing is that roughly two-thirds of the requirements are moved into a category of things “not to work on”.

Now apply this process iteratively. Take the remaining pile of “essential” items and continue breaking it down into sets of three buckets until the remaining items is small enough that you feel your sanity restored. This should be a couple of months worth of work.

That is the first key to the planning game, it is a form of triage that allows the team to focus on the essentials. The second aspect is related. Divide the essential elements into a series of very small releases, or iterations. Choose an iteration length of one to two weeks and layout the essential requirements into a series of iterations.

As much as possible have the project sponsor make the decisions about what features go in which iterations. The most important aspect of this part is to have the project sponsor make the changes to the iteration schedule. Give them a quota for how much can be in each iteration and then place the burden on them to arrange the requirements such that there is not too much in any one iteration. This is key because it creates a mechanism for the painful truth to be communicated directly to the sponsor. The truth is that 6 days of work cannot be crammed into 5 days.

If the game is setup correctly the sponsor feels this truth as a direct consequence of their actions, not as some arbitrary bad news that the development team has to tell them. So if the sponsor puts five more requirements into the next iteration then they should see that they need to remove five requirements in order to “make it fit”.

Iterations

Regardless of whether you can make the planning game work, iterations are essential for dealing with the pile. Take a tiny set of features from the pile and focus exclusively on them for a week or two and get them done. That is the essence of surviving and conquering the pile. (If you have a steady stream of support issues then leave room for this in your iteration plans.)

Successfully completing each iteration becomes the defining rhythm of the team. It is the basic stroke of the development engine.

Checklists

Completing iterations provides vital motivation to the team. But we frail humans need to be rewarded more often than once every week or two. We need daily reinforcement. So break the features for an iteration down into smaller steps that can be completed in a few hours. Get these listed for all of the team to see. Now developers can feel the satisfaction of crossing something off the list every day. Crossing items off of the list assures the team that the pile is being conquered.

So the team can survive the crush of the pile by dodging it. Don’t try and carry the weight of all of the requirements. Let the weight rest on the ground. Instead take on small parts of the pile at a time. Always take an amount that you can do and you build a pattern of success day after day, week after week.

Requirements, people, and monsters

Software development projects are dominated by three complex systems. First there are the requirements. Not just a few requirements but a massive mountain of requirements:

And it is a dangerous mountain that threatens to come tumbling down as an avalanche:

Crushing the development team. The second system: the team. A group of people. And that is their distinguishing characteristic. They are people. Weird people, creative people, confused people, confusing people, thinking people. That is the source of both their great annoyance and their charm… they are people:

But the people are not alone, they have built an application. They have created a monster:

A monster that requires constant care and feeding. A monster that occasionally goes on rampages threatening life and limb and demanding immediate attention.

That is the dangerous world we live in. A bunch of people under threat of being crushed in an avalanche of requirements and living with a monster. Sounds like fun, eh?

Build a prototype instead of a proposal

When you first start working on an application you can create an end-to-end working system in a remarkably short time. I don’t just mean a false demo that provides screens that don’t work. I mean a real working application. Now it will be very short on features. It will be an anemic application. But, it will be a working end-to-end system.

Some developers will not deliver a working end-to-end system quickly. Instead they will write specification documents, build data models, create architecture diagrams, build frameworks, construct development environments, do UI design, create detailed estimates, etc.

Some developers are scared to show sponsors a working system early, because the sponsor will never understand that the application is not finished, it is merely started.

Use the fact that you can create a working application very quickly to your advantage. Use this to win business. Use this to win real engagement with your sponsor.

If you are bidding on a project, instead of producing a proposal, produce the first version of the application. Imagine the buyer comparing a written proposal with a working app.

If you want to really engage with a sponsor and get them excited about the project, then show them a working prototype instead of a proposal.

And don’t just do this once, keep doing it. If you deliver new features that the sponsor wants every week then you will become an ally of the sponsor rather than an antagonist for them to fight against.

Sieze slack time

There is rhythm to life. Times change. One day there is intense pressure to get things done. The code needs to be finished immediately, there are production problems, or some such thing. Then a few days later there is apparent calm. Nothing is obviously broken and there is no immediate pressure to deliver. What to do in these times?

One answer is to work ahead. Attack the list of things to do as if there were immediate pressure. Why? Because there is a good chance that tomorrow things will change and the app will need to be done. If you work ahead now then you can alleviate that pressure in advance.

This is in essence what the planning game is all about. It takes the tremendous pressure that is felt at the end of the project and brings that back in time to the beginning of the project. By forcing hard decisions about priorities to be made early the planning process some of the end pressure is moved to the beginning of the process. In this way, the pressure is distributed and the ultimate pressure is relieved.

What else can you during slack times? This is a great time to invest in some strategic efforts that can shape the entire project and be leveraged for the duration of the effort. For example, invest time at the beginning of the project to establish automatic build servers and automatic testing. These efforts set the project on a trajectory that will become the standard for how the project operates.

Whatever you do, realize that the slack time will not last and seize it to drive the project on to greater success.

TANSTAAFL

I think I am like most developers. I tend to like working with code. In most cases I would rather not mess with pesky users. And I certainly don’t want to spend time fighting production problems. Recently I found myself in a position where I was isolated from users and production problems. I certainly wasn’t going to complain. Although I would often think to myself, “boy, it seems like I should be talking to our users”. I ignored those thoughts and just enjoyed the isolation.

I was wrong.

That isolation has been removed and I see that all of my work suffered from working in a vacuum. Previously I was cutoff from feedback from real users. That feedback is what should have been keeping me growing and learning how to make the system better.

I should have paid attention to what I knew to be true. I should have pushed myself into the user world. I should have taken the pain in order to learn and grow.

I should have remembered Heinlein’s advice, “There Ain’t No Such Thing As A Free Lunch“.

Be ready to ship at a moment’s notice

The software game is unpredictable. The customer might decide they need the software today instead of next week. Or the priorities might change and you will be called off of your current task. To ensure that your work is not wasted it has to be ready to ship at all times.

Do your work in such a way that is is always done. Not that it is completely done, but it is always partially done. The key is that it must be ready to ship at a moments notice.

How do you do that? You first implement features as thin, end-to-end strands. Then you bulk it up one additional strand at a time. To put it another way: first get a rough version of the feature working, then add the bells and whistles, one-by-one.