School is a terrible preparation for work

School trains people to think there is a predefined point of success. School trains people to place an upper bound on success. School prescribes a set of lectures to go to, a specific number of assignments to do. Tests have a limit. The best you can do is ace the test. There is a fixed number of classes needed to get a degree. School instills the notion that there is an upper limit on success and it is reached by getting the right items checked off on a predefined list.

School is antipreneurial. It opposes entrepreneurial thinking. Coding for money on the web means you are operating your own business. Entrepreneurial thinking is what you need to be successful coding for money on the web. School is a terrible preparation for working on the web.

Can inversion of control go awry?

Is there any downside to using inversion of control? Absolutely

Can the testing code become a disaster? Totally

Can you create a big mess of tangled code with inversion of control? Daily

Is it harder to read code that has been “distilled” so that compile time and runtime dependencies are different? Yep

So what is the conclusion? We need to train our brains to think in new ways and build/find tools to help us write code that supports automated testing (and therefore works and can be maintained).

We need to get better at distilling out the un-testable parts. We need to learn how to test more un-testable code. We need to learn better patterns of clearly doing inversion of control. We need to train our brains to think in different ways. We need to keep groping towards a better way of building software.

10 ways to spot a good coder

Some coders are really good. How can you identify them?

• You gotta know how to read a stack trace. I know this is a pathetically low bar, but I have seen countless developers just stare in wonder at a stack trace. So even this low bar will eliminate a bunch of people.

• Know your tools Know your IDE. Know your editor. Know your operating system. Select the right “power” tools and make them serve you. (An interesting corollary is that if you code on Windows then you have to use Cygwin to be a good developer.)

• Know your language If you have things to say in code, then you need to know how to talk. Learn every part of your language syntax. Get a broad understanding of the libraries your language offers.

• You have to be able to download a 3rd party package, get the source code compiling, make some changes to the source and get the hacked library working. Maybe they even have to use a decompiler to get the source.

• Know how to use a debugger.

• Know how to use a profiler. At some point the code is too slow and you need to know why. A good developer can bust out a profiler and get an answer.

• Read a spec. Sure every developer will read the spec before coding. Good developers read the spec when they are done coding… then they code up all the stuff they missed.

• Read code. Writing code is more fun, but a good developer reads someone else’s code, understand it, and make sensible changes to it.

• Work from the command line. Yeah GUIs are great, but you are doing way too much manual work if you don’t drop down to the command line and script out what you need.

• Create the build script It’s easy to find a good developer on a project. Find the guy who made the build script. He is the one who actually knows how the stuff works.

So what is the common theme here? The bad developers are faced with something new and they stare in wonder at all the meaningless symbols in front of them. Whether it be a stack trace or someone else’s code, or the confusion of buttons in our tools, or the build script. The good coders are filled with the same wonder, for a moment, but they apply their brain to gain understanding. They dig in, read, understand, learn. Programming is a knowledge business if there ever was one.

Invest $20 in a RentACoder experiment

If you are a developer you need to experience what it is like to use RentACoder to buy code.

Here is what you do:

• Think of a little piece of code you want written. Think of something you would pay $20 for.

• Write a one page spec for what you want.

• Create an account on RentACoder. This is a bit painful, particularly the part where they validate your financial credentials. But hey, no pain, no gain.

• Post the spec on RentACoder, watch the bids come in, pick a likely looking developer, and have them code it for you.

Here is why:

• Learn what developers around the world will code for $20.

• See what kind of code your competition produces.

• See what it is like working with a developer. What does the developer do to help make the effort a success or a failure?

• Experience buying software development services. Most coders spend their lives coding, but never get a chance to buy coding services. Experiencing the buying side will help you know how to relate to your buyers better.

If you are serious about your craft of programming it is well worth your time, and money to do a RentACoder experiment. Think of it as a bit of market research. It is guaranteed to open your eyes in some ways.

Give it a try and post a comment back here to let me know what you learn!

3 questions to ask about a programming language

I often come across ad-hoc mini-programming languages. For example, a scripting extension to an app or even something like CSS. When I am introduced to one of these mini-programming language I try to remember the basic elements of programming as described in the Structure and Interpretation of Computer Programs.

Ask the questions:

• what are the primitives - what are the basic things the language does?

• what are the means of combination - how do you put primitives together?

• what are the means of abstraction - how do you create a new primitive from combinations?

Asking these questions leads very quickly to a clear understanding of the power or the limitations of the language. If the language falls short, these questions provide a framework for crisply describing the problem. Moreover, if you are designing such a mini-language (which is a bad idea) then these questions will focus your attention on providing a powerful solution.

10 (or so) tips for learning a new programming language

The first time you hear about some new programming language you have a big, empty, black space in your head. That is your ignorance. The new language is foreign, unknown, a dark void. Learning the language is all about dispelling the darkness in your head and to replace it with a mental model of how the language works. You must cast the light of knowledge on the subject. Here are techniques I use:

Get your hands on it

You may have all sorts of intentions to learn something, but if you don’t have it then it is hard to learn. So first go download the thing you want to learn. You need to be able to hold it in your hand, look at it, and then you can consider actually using it.

Make it do something

This is the classical “hello world” program. The point is to make the language do something, anything. The broader lesson is to choose some initial tasks and make the language do those things. You must start to assert control. You don’t need to write perfect code or even good code at this stage. You simply need to make the system do something. This is the first glimmer of light in the darkness.

Study history

Find out where the language came from. Go back and find the original source material to understand the history of the language. Why was it first created? What kind of problems was it designed to solve? This will help you understand its strengths. What languages influenced its syntax? Does it have a C heritage, a Perl heritage? What accidents of history are responsible for you ever hearing about it (how many languages are created that nobody ever knows about)? Don’t be content to read some distilled version of the history. Look for the original, definitive source that puts the language into context. This is a beginning towards understanding the nature of the darkness.

Browse

Google, flip through books, look at the language source code, visit mailing lists, look at different products. Spend some time just exploring all the different kinds of information there are about the language. At this point you can’t remember much of it, but you will pick up some tidbits, you will start to get a feel for some of the main issues, and you will discover good resources to learn more. Many of these sources will give contradictory and just plain wrong advice, so don’t believe everything you read. This is not so much shedding light on the subject as it is shopping for places to get light.

Read the spec

Find the manual. Don’t just read some magazine articles about how an API works. Find the actual spec or API docs. Find the definitive source that says how it works. This is your map in the darkness. Find the definitive reference materials and keep them close at hand.

Identify the masters

Explore the language community enough to find out who the recognized “masters” of the language are. These could be the people who created the language, but they must be the people who are publicly putting the language to the best use in building real systems. These are the people who define the programming idioms that are characteristic of the language. These are the people who are setting the standard for how to best use the language. Find a guide for the journey into the dark.

Read the writings of the masters

Read what the masters have to say about the language. For .NET this means reading Jeff Richter For C this means Kernighan and Ritchie For Smalltalk you need to read Kent Beck

The main point is to read the good stuff. Don’t content yourself with reading some third-hand story on a miscellaneous web site. Once you learn the language better then you will be in a position to sift through such material, but it can’t be your main source. So by all means browse the web and read all kinds of stuff, but you must look to the experts for your primary education. Follow the master guide into the darkness.

Build something yourself

You can read books and tutorials all day. What the author does will make complete sense. “Of course” you will say… to make blue text appear on the screen that is obviously the right command to use. The problem is that your brain tricks you into saying “of course” to the answers you are given. What you need to do is to try and make the language do something of your own design. So what if you want green text and what if you want it to be tilted at an angle? Well… the book doesn’t say how to do that… now your brain has to engage and actually learn how to make the language do what you want rather than simply assent to what it reads in a book. Now you are traveling through the darkness and gaining first hand experiences to truly understand what the language is all about.

Read a complete description of the system

Find a good reference document that broadly covers everything the language can do. You need to have a complete survey of all of the classes of things that the language provides. For example, if you are learning Java, you need to know not just the basic syntax, but you also need to know that it is possible to integrate native code, to decompile Java code, to run on quite different Java virtual machines, to create a security sandbox, etc. You don’t need to learn all of these things in any kind of detail, but you need to become familiar with the range of possibilities of the language. This is an attempt to establish a general border of the big black void in your mind.

Learn some aspect in great detail

Choose some aspect of the language that you are interested in and become an absolute expert in that topic. Learn everything there is to know about it. Explore every last option, leave no stone left unturned. For example, maybe you take on the issue of threading. Explore every primitive the language offers for thread synchronization, learn all of the mechanics of creating threads, study the idioms that capture the communities sense of “best practice” with threads, learn how threads are implemented on different computer platforms, etc.. This is now a thin beam of light that extends all the way to the bottom of the darkness. There is no remaining mystery around the topic. Nothing hiding in the dark.

Study the work of the masters

Find code written by a master. So not only are you studying what the master has to say about the language, but rather you are learning what the master has said in the language. Read lots and lots of code written by the master. You want to just soak in well written code so it seeps into your thinking and into your style in ways that you are not even aware of. For Lisp this is reading Paul Graham’s On Lisp Here the goal is to learn to navigate the language, not just to get to the other side, but to get there in style, to do good work, to be an expert.

Hang out in the community

Find a community of experts in the language. Often this will mailing list archives or some kind of online forum. Browse through the archives. Spend a few weeks getting the latest messages. This is a way of immersing yourself in the culture of the language. So you will learn how such people think, what kinds of solutions are preferred.

Read the critics

Find the clearest thinking critics of the language and read what they have to say. This is a way to discover some of the hidden pits in the darkness.

Look to the future

Learn where the language is going. You don’t want to simply know how it works today, you want to know how it will work tomorrow. To be an expert you need to be ahead of the general adoption curve. For Java this means learning about upcoming JSR’s or just finding a good resource like Alex’s Java 7 page

So that is all there is to it. If you are trying to learn a new language and you are stuck then look back over this list, find something you haven’t done and go do it.

Of course these are just the starting points. To really know a language you need to build a real system and take it to production… and support it… a few times… over the span of several years… but that is another post.

Linux RAID and LVM configuration

Recently I have had the opportunity to learn how to setup RAID and LVM on Linux. Specifically on Ubuntu 6.06.

It is striking how easy it is to setup once you know the words to use. It is just as striking to see how all the pieces plug together… like legos. The things made: partitions, RAIDs, logical volumes, these are all compatible parts that can be plugged together in many different ways. It is a beautiful thing.

Here is a script to play with these commands (please don’t run this on your system unless you understand basically what it does, it could mess up your system):

# create some loopback devices to play with
# these are simple files 
# that can be used like devices
# this is just for testing

head -c 10000000 < /dev/zero > test_dev0
head -c 10000000 < /dev/zero > test_dev1

losetup -f

losetup /dev/loop0 test_dev0
losetup /dev/loop1 test_dev1

# partition the devices

sfdisk -f -S16065 -uS /dev/loop0 << EOF
63,16065,FD
EOF

sfdisk -f -uS /dev/loop1 << EOF
63,16065,FD
EOF

# create a RAID out of the two devices
mdadm --create --verbose /dev/md2 --level=1  \
     --raid-devices=2 /dev/loop0 /dev/loop1

# create an LVM physical volume from the RAID
pvcreate /dev/md2

# create an LVM volume group 
# and put the physical volume in it
vgcreate demo_vg /dev/md2

# create an LVM logical volume in the volume group
lvcreate --size 4M -n demo_lv demo_vg

# create a filesystem in the logical volume
# the -j option creates an ext3 file system
mkfs -j /dev/mapper/demo_vg-demo_lv

# make a directory to mount the new file system
mkdir demo

# mount the file system
mount /dev/mapper/demo_vg-demo_lv demo

A few notes on this:

Some of the loopback commands came from www.linux.com

Normally you would create a device by installing a disk, rather than the loopback device commands that are at the beginning. However, this is a good way to get experience with the commands before working on real disks.

The script partitions the devices using sfdisk. I used this command because it was something I could script. In practice I use cfdisk to partition the drives. The key steps are to setup the sizes of the partitions and to set the partition type. In this case set it to FD, to mark it as a Linux RAID partition. I recommend that all of the partitions that are placed into a RAID together be of the exact same size.

The RAID created in the script is named /dev/md2. This name depends on what is available on your system.

For a real system, you will want to get the mount to be performed via /etc/fstab so it is automatic.

Some commands to see the creation:

# see loopback device
losetup /dev/loop0

# see partition table (actually this seems to not work 
# with the loopback device)
sfdisk -l /dev/loop0

# see the RAID
cat /proc/mdstat

# list all physical volumes
pvscan

# list all volume groups
vgs

# list all logical volumes
lvs

# another way of seeing logical volumes
ls /dev/mapper

# see what is mounted
mount

A script to tear down the demo file system:

umount demo

rmdir demo

lvremove -f /dev/demo_vg/demo_lv

vgremove demo_vg

mdadm --stop /dev/md2

losetup -d /dev/loop0
losetup -d /dev/loop1

rm test_dev0
rm test_dev1

Exploring ruby class and metaclass relationships

The Ruby class structure is difficult to understand without drawing pictures. Three class relationships are important:

1. class
2. superclass
3. metaclass

In order to sort it out I consider three categories of objects:

1. plain objects
2. classes
3. metaclasses

Plain objects are just plain old instances of objects. So if I have a Dog class and this code:

d = Dog.new

Then d is a plain object.

In this example Dog is a class. In Ruby, classes can usually be considered as just special types of objects. In the following code, Dog is a class and Animal is a class.

class Animal
end

class Dog < Animal
end

Other relevant classes include Class itself and Object.

That leaves metaclasses. Here is a good explanation of metaclasses (after reading his stuff I feel like I should write in the wonderful gibberish that he uses… alas, I don’t have that talent so I have to settle for plain old boring English). The basic story of metaclasses goes something like this: Plain objects hold state, not behavior. An object’s methods are defined in its class. The behavior of all Dog instances is defined in the Dog class. A metaclass is a special class that allows behavior to be defined at the instance level. Adding a method to the metaclass of object d, creates a method that will only apply to object d.

Similarly, a class can have a metaclass. This is analogous to the situation I just described. Methods on the metaclass only apply to a single instance of an object, in this case the object happens to be a class. Methods on a class metaclass end up looking a lot like class methods from Java.

The bottom line is that every object in Ruby can have a metaclass. This includes plain old objects, classes, and metaclasses. They can all have metaclasses that define methods just for them.

I keep saying they can have a metaclass. The first time a metaclass is referenced it will be created. Until then it does not exist.

I use this bit of code (also from why) to access an object’s metaclass:

class Object
  def metaclass
    class << self
      self
    end
  end
end

The syntax, class <<, is used to access an object’s metaclass. This small bit of code adds a method called metaclass to the base class, Object. This new method uses the class << construct to get access to the current object’s metaclass and then returns that metaclass. Within the context of the class << construct, self refers to the metaclass.

Finally, as I said, a metaclass itself can also have a metaclass. So you can keep adding metaclasses to your heart’s content.

Those are the three types of objects; now the three types of relationships.

The class of an object tells you what type of object it is. The class of plain objects is simple, it is their class. The class of a class is easy, it is always Class. This includes special classes such as Object. The class of Object is also Class. Similarly the class of all metaclasses is also Class.

The superclass relationship is mostly straightforward too. Plain objects don’t have a superclass. The superclass of classes is what you would expect. It follows the normal object-oriented inheritance structure. In Ruby, Class has a superclass of Module which has a superclass of Object. All classes descend from Object.

The superclass of metaclasses is a bit complicated. Based on experiments that I conducted with Ruby 1.8.5 I found that the metaclass inheritance structure changes in surprising ways. In the simple case, a plain old object’s metaclass starts out with a superclass of the object’s class’ metaclass.

This also applies to a class metaclass which means by default a class metaclass has a superclass which is the Class metaclass.

However, when a metaclass is added to the object’s metaclass, the superclass of the object’s metaclass changes to point to the next metaclass out. With the outermost metaclass pointing to itself as superclass.

The final wierdity is that the metaclasses of Object and Module have a superclass of Class. This explains how metaclasses end up in the Class hierarchy. And this too changes once a metaclass is added to the metaclass of Object or Module.

Charting out the inheritance structure of metaclasses is complicated by the fact that metaclasses are created on demand and when they are created they change the metaclass inheritance structure. So simply retrieving a metaclass to see what its superclass is causes the inheritance structure to change. Fun.

NOTE: These results are different than what the docs say.

If superclass is complicated, the metaclass relationship is quite simple. Every object’s metaclass is its own unique metaclass. This applies to plain objects, classes, and metaclasses.

Now all of these diagrams can be combined to get the whole picture of how plain objects, classes, and metaclasses relate to each other.

These are the conventions in the diagrams:

1. plain objects are ovals
2. classes are white squares
3. metaclasses are blue squares
4. class relationships are shown as lines pointing to the north-west
5. superclass relationships are lines pointing up
6. metaclass relationships are lines pointing to the left

More complete code to explore these relationships:

class Animal
end

class Dog < Animal
end

class Object
  def metaclass
    class << self
      self
    end
  end
end

d = Dog.new

def dump (d, code) 
  puts "#{code} = #{eval code}"
end

dump d, "d.class"
dump d, "d.metaclass"

puts
dump d, "d.metaclass.class"
dump d, "d.metaclass.superclass"
dump d, "d.metaclass.metaclass"

puts
dump d, "d.class.class"
dump d, "d.class.superclass"
dump d, "d.class.metaclass"

puts
dump nil, "Class.class"
dump nil, "Class.superclass"
dump nil, "Class.metaclass"

Generate a real parser for your custom language

Many years ago I found myself in the position of writing telephony applications for the phone company. We were using an IBM system that expected you to use a graphical programming tool to create a state machine describing the program. The state machine had primitives like play a message to the caller, collect digits from the caller, etc. Writing programs as state machines in their dismal environment was brutally painful. You ended up doing ridiculous things like building while loops out of primitives that checked a condition, took an action, and then looped back to the top. All of this by adding blocks and drawing lines between them… ugh!

So I longed for a simple C style syntax to write my code. Not knowing any better I got a copy of the Dragon Book from the library, downloaded lex and yacc and created a prototype of a language that compiled to a state machine format the IBM system could read. Once I had a protoype working I was convinced I could really build such a thing. I switched to JavaCC and created a full-blown language.

The way tools like lex/yacc and JavaCC work is that you create an input file that specifies the grammar for your language. You tell it what the keywords are and what order they are to appear in. Then the tool generates a parser for your language. It produces an abstract syntax tree that you can then walk to produce the compiled output that you need.

This was undoubtedly the most fun I ever had programming. And this was before I became a serious professional programmer. In fact, this experience was one of the factors that drove me from engineering to programming as a profession. So I highly recommend the experience of creating a language in this way. Furthermore, if you are going to create your own language this is the way to do it. You need a real parser for it, not some home-grown monstrosity. So using something like yacc or JavaCC is the way to go. You can hand someone the grammar specification and they can read it to see how to speak your new language.

It is quite amazing the power you feel having such a power tool at your fingertips. On future jobs I would whip out JavaCC and create a code generator at the drop of the hat.

Back to the original telephony project… once I had the basic language working, I then proceeded to create another language: a domain specific language that compiled to my C style language. This DSL was tailored to creating the kinds of voice applications that we were building. Unlike the IBM graphical programming tool, my DSL made it trivial to create the apps that we needed to create. I became dramatically more productive than my teammates by using these tools.

So what was the next step? Getting my teammates to use it of course. And why wouldn’t they want to? It would save them tons of time. Well, as it turns out, I was never able to get anyone else to even look at it seriously, much less use it. What good was the formal specification of the grammar for the language? None. Nobody read it. And if they did look at it, it’s not the most self-explanatory thing. What you really need is lots of examples that show the language in use. Nobody learns a language by reading the grammar specification.

So you can create your own language for yourself, but there is not much chance of getting others to use it.

Respect the web

As an enterprise architect I spent years holding web technologies in disdain. Sure, you could make some neat little pages but it was certainly no way to build real apps. Over the last few years I have seen the light and come to appreciate the wonderful, messy, hacked-together, really useful real world of the web.

So here is my guide:

1. HTML - It is not just an inconsequential view that may be used to render your data. It is a really useful way of marking up data, not just for browsers to view but for code to parse. Forget custom XML schemas. Use HTML. If your data is in HTML then your code can parse it easily, your browser can render it easily, and you can use CSS to format it nicely.
2. HTTP - Don’t dismiss it as some low-level communication layer that you will use to tunnel your application protocol. Embrace HTTP as your application protocol. Read the spec. See what it offers and use it. Even if you don’t have a web client use HTTP to communicate with your server. Don’t bury HTTP at the bottom of the protocol stack, but embrace it and build an app to use its features.
3. JavaScript - It is not just a toy scripting language for creating pop-ups or doing validation. It is a powerful dynamic language in its own right.
4. REST - Learn about the principles that have made the web the success that it is. Embrace these principles in your own apps.

Even if you are building “enterprise” apps use web technologies. Even enterprise applications need to inter-operate and that is what the web does. As Mark Baker says, the web is the distributed object system you have been looking for.