Requirements, people, and monsters (part 3)

The second major system in a development effort is the team. The key distinguishing characteristic of the team is: it is composed of people. People in all of their irrational, emotional, egocentric, and goofy glory.

People are not parts. People are not resources. People do not equate simply to developer hours.

You can’t even get along with the people in your family. How are you ever going to get along with the people on your development team?

For some the answer is a stern “professionalism” that pursues an illusion of the lack of any kind of emotion and requires team members to do the same. Thus sweeping their people-ness under the rug.

For others the solution is a harsh command-and-control environment. “I am the boss, you will do what I say. I don’t care about your personal issues.”

The stereotype of computer guys is that we like “hard science”: numbers, code, engineering. Not soft things like: psychology and people and emotions. Then one day you realize the people around you form the most complex system you will ever have to work with. People are the ultimate computers. People are the most complex software system we will ever deal with. The interactions between people form the most complex systems we will ever see.

But more than that, people are the most significant things we will encounter in our computer careers.

When you realize these things you will be driven out of your computer science hole and into the field of learning about and trying to understand people. Here are some key lessons about people.

People are all the same

Everyone else is just like you. We look at another person. We have a natural tendency to think we are categorically different than they. The stupid manager who has no idea what the code is doing. Our first thought is that they are a different type of person than us. The reality is they are the same type of person as us. The fact that they look stupid to us is more about where the two of us happened to meet. We meet after the manager had gone down his management path and I down my coder path. The difference between us is based on relation to each other, not fundamental differences in the kind of person he is and the kind of person I am. In your future you will go down a similar path and be viewed in a similar light.

But, even if you never become a manager, someone will accuse you of being a stupid manager some day. In the same way that politics works. Someone is a staunch liberal all of their life. Then they say something vaguely pro-business. Suddenly they are labeled a right-wing conservative.

With that in mind, consider:

The idiot programmer who can’t figure out the API. That is you.

The slacker who quits working and leaves you hanging when the deadline arrives. That is you.

The stupid manager who is blind to what is really happening. That is you.

The hotshot kid who thinks he knows everything. That is you.

The inept programmer who created an unmaintainable mess. That is you.

The clueless project manager who is completely. That is you.

Those are all you in the sense that they are people like you who are dealing with their circumstances and trying to get through the day as best they can. Someday you will look like them to someone else.

People are all different

You learn a bit about people and you start to categorize them. People hate to be categorized (at least I do!).

People hate to be labeled.

People hate to be psycho-analyzed.

People hate to feel like someone else has them “figured out”.

It is dehumanizing. Like reducing a person down to a “database administrator”. When you pigeon-hole someone you don’t pay due respect to their people-ness.

We are all unique and must be dealt with as individuals, not just as members of some categories.

The agile movement also brought these issues to the fore.

…we have come to value: Individuals and interactions over processes and tools…

Agile brought us daily meetings where you talk to each other. Agile brought us the idea of developers estimating their own work and signing up for their own tasks. Humanizing ideas.

To summarize then:

• People are the core of the development process
• People are the most complicated and unpredictable system in the process
• People are the most significant things we deal with
• (Oh.. and we all are one)

So go learn about people. Maybe read a classic like Peopleware. Or learn from a master like Jerry Weinberg and books like:

• The Secrets of Consulting
• Becoming a Technical Leader
• More Secrets of Consulting

WARNING: These are not your typical computer books. They may take you far outside of your comfort zone.

Use Java thread pool to isolate poorly behaved objects

There is an evil in software systems. Some services are not well behaved. When you call them you may block forever waiting for them to respond. These are toxic services. The really bad aspect of such services is they are infectious.

Imagine you have created a component. It is processing many simultaneous requests from callers. It has many threads running. If your component calls a toxic service in the most obvious way then it will become toxic. The way this happens is that a thread running in your code calls the toxic service. If the toxic service is misbehaving, then your thread blocks forever. A little bit later another one of your threads call the toxic service and it too blocks forever. This will continue until all of your threads are blocked waiting for the toxic service.

If you observe the external behavior of your component at this time you will see that it is toxic just like the toxic service you are using. Sometimes callers get a normal response from your component; sometimes they block forever. You are toxic.

And it gets worse. Consider services offered by your component that don’t use the toxic service. Even these are choked out by all the busy threads blocked on the toxic service. So the failure spreads to include operations that are unrelated to the underlying source of the problem. Even if only 1 out of every 100 operations use the toxic service, the failure will still spread and quickly block all 100 operations.

And it gets worse. Because your component continues to consume resources without bound you will bring down other components running on the same system. The problem also propagates upstream to the calling systems. If the calling components are written in a naive fashion (like your component) then they too will become toxic and continue spreading the love.

This is the way systems die.

So what is the solution? The general solution is to move from synchronous to asynchronous constructs. Invoke the toxic service through asynchronous messaging instead of blocking synchronously on a thread waiting for a response. Calls to the toxic service show up in queues. These queues can be monitored, bounded, and managed.

The problem with the asynchronous messaging solution is that it requires a fairly dramatic change to the design of your code.

Here is a simplified implementation of the asynchronous idea that does not disrupt the design of your code. Allocate a pool of threads for dealing with the toxic resource. Give this pool a safe upper limit. You will never sacrifice more than N threads to the toxic service. This is the foundation of the approach. It provides a safety mechanism to keep the toxicity from spreading without limit.

When your code needs to call the toxic service, instead of calling it directly, it asks one of the brave volunteers from the thread pool to call the toxic service. This is a sacrificial thread that may never return… Set a timeout and wait for a little while to see how the thread fares. If it returns, great! The toxic service is working. If it doesn’t return then give up and tell the caller about the problem.

This leaves the sacrificial thread in the thread pool “hung” waiting for the toxic service. But, since we have an upper bound on the pool size, once we reach the limit we will stop calling the toxic service. If we stop calling it, then it cannot claim any more threads.

This approach leaves your component free to continue servicing requests that don’t require the toxic service. This approach does not consume unlimited system resources so other components on the same computer continue to operate. This approach doesn’t block calling code indefinitely. So the toxic service has been contained!

Now for some code. The Java libraries provide the code needed to implement the strategy. I ran this code on Java 1.6.

Here is an example of a toxic object. Notice how it randomly blocks forever.

import java.util.Random;


// A poorly behaved class.
public class ToxicService implements Service {
  private static final int FOREVER = 10000;

private final Random random = new Random();

public void go() throws ServiceException,
      InterruptedException {
    if (oneOutOf3()) {
      // Sometimes it blocks.
      blockForever();
    } else if (oneOutOf3()) {
      // Sometimes it fails.
      throw new ServiceException();
    }
    // Sometimes it works!
  }

void blockForever() throws InterruptedException {
    // This is what makes this object toxic. Sometimes it
    // blocks forever
    // when called. (Ok… not really forever in this
    // example, but long
    // enough to see the problem.)
    try {
      Thread.sleep(FOREVER);
    } catch (InterruptedException e) {
      // Bad code - ignore interruption, just keep running
      Thread.sleep(FOREVER);
    }
  }

boolean oneOutOf3() {
    return random.nextInt(2) == 0;
  }
}

public interface Service {
  public void go() throws ServiceException,
      InterruptedException;
}

// Exception thrown by Service.
public class ServiceException
    extends Exception {
  private static final long serialVersionUID = 1L;
}

The following class shows how this ToxicService can be wrapped in a way that will protect the calling code from the bad behavior. This class uses the thread pool tools built into Java to isolate the calls to the toxic object in separate threads. A timeout is used to give up on these calls if they don’t return quickly.

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;


// A class that "contains" the badness of a ToxicService.
public class ContainedService implements Service {
  // Toxic service that is being contained.
  final Service service = new ToxicService();

// Thread pool for running toxic calls.
  ThreadPoolExecutor executor =
      (ThreadPoolExecutor) Executors.newFixedThreadPool(5);

public void go() throws ServiceException,
      InterruptedException {
    List<Callable<Object>> toRun =
        new ArrayList<Callable<Object>>();
    toRun.add(new Callable<Object>() {
      public Object call() throws Exception {
        // Call the service.
        service.go();
        return null;
      }
    });
    List<Future<Object>> futures =
        executor.invokeAll(toRun, 1000,
            TimeUnit.MILLISECONDS);
    try {
      // Find out what happened when the service was
      // called.
      futures.get(0).get();
    } catch (ExecutionException e) {
      // Propagate the exception that is part of the
      // interface.
      if (ServiceException.class.isAssignableFrom(e
          .getCause().getClass())) {
        throw (ServiceException) e.getCause();
      }
      throw new RuntimeException(e);
    }
  }

public void shutdown() {
    // Shutdown the thread pool.
    executor.shutdown();
  }
}

The ContainedService class can be used like this:

import java.util.concurrent.CancellationException;


public class ContainedMain {
  public static void main(String[] args)
      throws InterruptedException {
    ContainedService service = new ContainedService();
    for (int i = 0; i < 10; i++) {
      try {
        service.go();
        System.out.println("success");
      } catch (CancellationException e) {
        System.out.println("timeout");
      } catch (ServiceException e) {
        System.out.println("failed");
      }
    }
    service.shutdown();
  }
}

This code works, but it is hand-crafted for the specific object being contained. A better solution would allow us to capture the timeout behavior in a generic form to be applied to arbitrary objects. We can create an InvocationHandler that will generically wrap each call with the timeout behavior.

Notice how the invoke(…) method handles all calls generically and the exception handling deals with all exceptions generically.

import java.lang.reflect.InvocationHandler;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;


// A generic class for timing out operations on poorly
// behaved objects.
public class TimeoutHandler implements InvocationHandler {
  // Toxic object. Calls to this object will timeout.
  private final Object target;

// Thread pool used to call toxic object.
  ThreadPoolExecutor executor =
      (ThreadPoolExecutor) Executors.newFixedThreadPool(5);

public TimeoutHandler(Object target) {
    this.target = target;
  }

public Object invoke(Object proxy, final Method method,
      final Object[] args) throws Throwable {
    List<Callable<Object>> toRun =
        new ArrayList<Callable<Object>>();
    toRun.add(new Callable<Object>() {
      public Object call() throws Exception {
        // Call the toxic method.
        return method.invoke(target, args);
      }
    });
    List<Future<Object>> futures =
        executor.invokeAll(toRun, 1000,
            TimeUnit.MILLISECONDS);
    try {
      // Discover result of toxic call.
      return futures.get(0).get();
    } catch (ExecutionException e) {
      // Unwrap interface specific exceptions.
      if (InvocationTargetException.class
          .isAssignableFrom(e.getCause().getClass())) {
        throw ((InvocationTargetException) e.getCause())
            .getCause();
      }
      throw e.getCause();
    }
  }

public void shutdown() {
    // Clean up thread pool.
    executor.shutdown();
  }
}

This InvocationHandler can be used with the Java Proxy mechanism like this:

import java.lang.reflect.Proxy;
import java.util.concurrent.CancellationException;


public class ProxyMain {
  public static void main(String[] args)
      throws InterruptedException {
    TimeoutHandler timeoutHandler =
        new TimeoutHandler(new ToxicService());
    Service service =
        (Service) Proxy.newProxyInstance(Thread
            .currentThread().getContextClassLoader(),
            new Class[] { Service.class }, timeoutHandler);
    for (int i = 0; i < 10; i++) {
      try {
        service.go();
        System.out.println("success");
      } catch (CancellationException e) {
        System.out.println("timeout");
      } catch (ServiceException e) {
        System.out.println("failed");
      }
    }
    timeoutHandler.shutdown();
  }
}

Download the sample code.

Note: This example code uses threads to solve the problem without ever using synchronized, wait(), or notify(). This is important! It means the code has a decent chance of working. The java.util.concurrent package deals with the low-level tricky threading issues for you.

Career advice from Larry Smith

Career advice from Larry Smith:

http://economics.uwaterloo.ca/smithdat/career2.html

“Conduct your career as your business”

Make a plan for yourself. Don’t allow yourself to be simply carried along by the currents of the day.

“You never let your employer have full control of your mind’s agenda. Control of the agenda is control.”

Develop yourself uniquely as a product offering:

“The goal is to be unlike others, to be premium priced, to have distinctive function. And there is only one way to do that… Your ideas make you distinctive.”

Market your offering broadly:

“When employed, you distribute your ideas as broadly as possible…”

“You create a resume of distinctive accomplishment. You do not describe routine duties, letting the job title speak for itself. Rather you describe what exceptional work you did…”