The Regenerate Web

RollerSkateImplementation agile 9 September 2007 Reactions

A key property of agile development is figuring out how to make a system go live with a small subset of features. We build software for the business value it offers, the quicker we go live, the faster we get at least some of that business value.

My colleague Dave Leigh-Fellows told me one of my favorite examples of this kind of thinking. It came when we has working for a brokerage firm. They had a new kind of product that they wanted to get into the market. The full software support for this was a web page that the customer filled in that generated the necessary transactions against the back-end system. But Dave came up with a way to get the product into the market faster than that.

• Version 1 was a static web page that described the product and provided a telephone number to call. Some temporary staff then spoke to the customer and entered the information into the back end system.
• Version 2 was a web form that captured the data the customer filled in. However this version didn't load that data into to the back end system. Instead the web form generated a fax. They hired some more temps to get the orders from the fax machine to the people that keyed the information into the back end system. Since the fax machines were a bit of a distance away, this is where the roller-skates came in.
• Version 3 hooked the web form into the back-end system directly.

The first two versions may not have been the most elegant solutions ever conceived, but they did get the product into the market much more quickly. I've not come across any other examples of iterative development that use roller-skates, but that may be more due to lack of imagination rather than lack of need.

DoctorWho leisure 7 September 2007 Reactions

Like most Brits my age I grew up with a sci-fi Children's program on BBC called Doctor Who. (For those who know, my doctor was Jon Pertwee, although I also saw a good bit of early Tom Baker.) It was actually the longest-running sci-fi TV series in the world, running from 1963-89.

A few years ago it was revived in the UK and has become a big hit - and not just for children. Doctor Who always had themes and scripts that went beyond the children's' audience and the series developed a huge fan base that lived off books and audio series even when the TV series died. The revival follows this with shows that are written to appeal as much to adults as kids. It was really great to sit with a couple my age and their 8 and 10 year old daughters and enjoy the new series. The scripts and acting are good, the only change is that the special effects are also good now (the old special effects made classic star trek look high-tech).

At home we don't watch much telly, the last shows we watched regularly were Buffy and Angel. Cindy, being American, had never seen Doctor Who growing up, but she loves the new series. When we get a new set of DVDs there's usually several nights of "it's late, we're tired, but maybe we can do one more".

If you've never seen Doctor Who the place to start is the opening episode of the revived series: Rose. (Wikipedia has mind-bogglingly comprehensive coverage, but I won't link from here as it's naturally full of spoilers.) Rose not only introduces the set up you'll need for other episodes (who the Doctor is, what the TARDIS is) but also does a really good job of distilling the tone of Doctor Who, capturing the mix of adventure and comedy.

If you like Rose then you can either carry on with the full first season or cherry pick highlights. If you prefer the latter I'll suggest my favorites. My big favorite from the first series was the two part The Empty Child / The Doctor Dances. I rate this as better than most films I've seen, certainly better than most TV. (It won a Hugo award so it's not just me.) It was written by Steven Moffat who is also known for writing the comedy series Coupling. Almost as good is Dalek. It lacks the humor but scores due to a wonderfully intense performance from Christopher Eccleston. I also really like the final two part (Bad Wolf / The Parting of the Ways) but you really need to see the whole series to appreciate it properly. (A tip if you do watch the whole series: don't watch the trailer for Bad Wolf (it appears at the end of Boom Town) as it gives away an important part of the plot of Bad Wolf.)

The second series has a different actor playing the Doctor (they have a nice technobabble rationalization to allow them to change actors easily). It doesn't quite hit the high spot of The Empty Child but is still really good. My suggestions for cherry pickers here would by The Girl in the Fireplace (another Moffat Hugo win) and The Impossible Planet / The Satan Pit.

When I said the second series didn't quite hit the heights of The Empty Child, I'm not being very reasonable because those two episodes are far too good for any TV series to live up to. However the third series (not yet all broadcast in the US - it's good to have friends in the UK) hits those high notes twice. Human Nature / The Family of Blood is a super two-parter that threatens to take away Steven Moffat's crown of writing the best episode. Moffat's response is Blink, which is as good a 45 minutes of TV as you could ever hope for. Not just has it got a great story and some cracking humor, it also achieves Doctor Who's higher purpose. You see Doctor Who is only secondarily about entertainment, it's primary purpose was always to scare the living daylights out of small children. I may be too old now to get behind the sofa, but I do remember how much I enjoyed it.

TimeZoneUncertainty design 6 September 2007 Reactions

I was in Boston, about to fly out to our office in Calgary. I look at my calendar to see if I have a meeting. First one is at 10.30am - cool no need to rush out of bed in the morning.

I turn up and am two hours late. What happened was that I was invited to the meeting that begun at 8.30am Calgary time. Lotus Notes saw my computer was set to Boston time, and helpfully converted the time zones for the two hour shift.

You could argue that this was my fault for not paying attention. After all I know this is how Notes works and was being careless when I read my diary. I don't buy this, Donald Norman noted a while ago that we tend to blame ourselves for errors that are due to bad usability - like a door with a handle that you should push.

Time zones are particularly vulnerable to this kind of problem It's most notably a problem in calendaring applications, but you see this issue in enterprise software as well. There is a temptation to try to be clever with handling time zones, but this temptation leads to trouble if the software isn't quite clever enough, which is what happened in this case.

I'd prefer Notes ignore time zones completely. You set the time for the place the meeting occurs in and that's what it should show you. Who cares about the time zones? It's only the time on the ground that usually matters. When I look at my calendar for a day in Calgary, I want to see the Calgary times for that day - wherever I happen to be when I'm looking at it.

The exception, of course, is a phone meeting that spans time zones. But here you should do something exceptional for a phone meeting rather than complicate handling a physical one. It could be something that allows you to set a flag for a phone meeting, and then you get a different time display. Or it could be as simple as just allowing you to put the timezone on the time display, and leaving the conversion to the reader. With phone meetings you think more about time-zones than you do for physical meetings (or at least I do).

The important lesson here is to make the most common case (physical meetings) simple and only do complicated things for less common cases as exceptions (quite possibly manual exceptions). Calendaring time zones get into trouble because they make the common and simple case be more complex. The problem occurs because the designers wanted to use the same data for both the simple and phone cases - but that just gets the simple case in trouble.

Usually when I hand out a prize for worst-user experience Lotus Notes is at the top of the queue. (Indeed I find it embarrassing to admit that ThoughtWorks uses the damn thing.) But the worst time-zone experience award goes to Microsoft Office, although to be fair this was many years ago. I had recently bought a PDA (running Windows CE version 2). I had put some all day meetings into my calendar, flown to Chicago, and to get the PDA to alert me to meetings I changed the timezone of the PDA to one hour earlier.

Suddenly every one of my all day meetings shifted to a day earlier. This was due to a catalog of errors. First off they had stored an all day meeting as a meeting from midnight to midnight - that's the kind of representation error on a TimePoint that often gets people into trouble. Then it was compounded by shifting the times of meeting when I changed time zones - so all day meetings were now 11pm to 11pm. This, of course, is due to putting the time zone into the meeting so it looks like it shifted when I changed time zone on the PDA. Then to cap it off, the software was clever enough to know that an all day meeting should only show the day of the meeting, but the day it chose was the day of the start of the meeting - that was now one day earlier. That's the poor timepoint representation biting back.

CustomerLoyaltySoftware design 4 September 2007 Reactions

I was in the Calgary office last week and had a good chat with John Kordyback, one of our most trusted technical leads. He's worked on, and dug into, a number of travel loyalty software systems (frequent flyer/sleeper etc) and we talked about the nature of these kinds of things and how to think about them in a more fruitful manner.

The core of of a loyalty system is a system to keep track of points (or miles). This should allow customers to see their points and also for the company to manage the unredeemed points. Although it seems that most people don't see it this way, this is essentially an accounting system, just switching points for dollars. John's observation was that repeatedly he runs into what people see as difficult problems that are much easier to deal with once you put accounting spectacles on.

An example of this is dealing with ad-hoc changes. However good your automated rules processing is, there always cases when something odd happens and you have to intervene manually. The result for many systems is and ad hoc change to totals that is error prone and unaudited. With an accounting frame of mind, however, you look at these changes as accounting adjustments and the patterns for this are well understood.

A notable difference between a loyalty program and most accounting systems is that a loyalty program is more about managing liabilities rather than managing assets. Hence there's more focus on things like risk management, as well as common themes like taxes and revenue reporting.

Many loyalty systems have multiple kinds of points, such as such as regular miles and elite qualifying miles. This is a common point of complexity. If you use an accounting viewpoint, however, you can track these easily as multiple currencies.

An interesting twist on this is potential points. If I book a flight for next month, the airline needs to know that there are miles I will earn when I fly next month (potential miles). These potential miles affect their liabilities. However it's only when I fly that they turn into real miles. Again accounting thinking can help here, we can use multiple currencies again, or use an accounts payable notion. The mechanisms are there and well understood, we just have to apply the model to the situation.

We fleshed this out in practice where we also found it really helpful to use TestDrivenDevelopment. A group of people spent a couple of weeks trying to sort out potential miles with planned design, but the core issue was cracked in a couple of days with TDD. The crucial part of this was focusing on examples to make the problem concrete.

The accounting analogy also applies, although partly less directly, to deciding how to award miles for activity. Any program has activity rules that need to be very flexible and need to cope with constant changes to the loyalty program. We can look at this as following the model of domain events triggering accounting entries through using Agreement Dispatchers. This is a pattern John and I have used lots of times and works well to these kinds of changing rules. Essentially we have agreements that represent the overall program rules for a class of participant. Each agreement consists of a set of posting rules keyed by the type of event and a date range. When an domain event occurs (a hotel stay) we look up the agreement dispatcher for the customer, and use the event to look up the right posting rule. We then run the posting rule to create the appropriate accounting entries to represent the miles for the event. The time dating of the events allow us to change posting rules over time but still be able to handle old events and correctly do automated processing of adjustments. (Some day I'll finish writing up these patterns, but what I have on the web is hopefully enough to give you some ideas.)

The second aspect of a loyalty system is tracking the customer experience. Since the accounting requires the system to record the customer's activity, the loyalty system acts as a natural base to learn from the customer's interactions with the company. Much of this is data mining - looking for patterns in customer behavior which can lead to new products and promotions. You can also use this activity history to assess the success of promotions - if you offer a mileage bonus for flying a route what is the response like?

Like me, John is a strong proponent of using ReportingDatabases, and this is a good fit for this kind of problem. The accounting side needs a very different set of data structures and uses regular updates as activities occur. The customer experience analysis is all read only, so you can use less normalized structures with regular, but not necessarily real time, feeds from the accounting side.

Taking it further, it seems reasonable to completely decouple the accounting and customer experience systems. They are both usually lodged together as a single customer loyalty system because they track the same events. Yet since they differ so much on the inside it may make more sense to treat them as two separate systems that feed off the same event stream (the accounting side would probably generate some events for the customer experience side too).

One of the habits of customer experience tracking is frequent changes to the system to support new kinds of analysis. We speculated that we could try an approach that had a single stored event log of customer activity, and plug in relatively independent 'miners' that would transform selected information from the log into more particular data structures to do different kinds of analysis. The miners could be relatively independent of each other and thus easier to build.

As you can see, our discussion did shift from looking at John's experiences to some of our joint speculations about how a system like this could be built in the future. What's clear to us is that there is a lot of room for exploring new ideas in this space that could introduce a new set of abstractions that would lead to systems that can provide better support to this business activity. More and more attention is being paid to this these days, so this seems like a fruitful territory for us to work in.

IsChangingInterfacesRefactoring refactoring 2 September 2007 Reactions

A RefactoringBoundary.

Is changing the interface of part of the code a refactoring?

The answer to this question is pretty simple - changing an interface is a refactoring providing you change all the callers too. A great example of this is Rename Method, which is an interface changing refactoring present on pretty much all refactoring tools.

The changing of all the callers is an essential part of this refactoring. Just changing an interface declaration will break the system - and thus isn't a behavior preserving change.

Interface changing refactorings do assume that you can get hold of all the callers, which is why you have to be much more careful with PublishedInterfaces. With a published interface, the interface itself is part of the observable behavior of the system.

Dynamic languages can make these changes much more awkward. Static typing really does help here in pinning down exactly which interface is being called at various points. Reflective calls that can also make it harder to find, either by embedding method names in strings or even composing them at run-time. This is another area where tests are essential even in environments that have refactoring tools.

I’ve been getting a little bit behind on my world tour trip reports, but things have been going so smoothly thanks to Liz’s heroic organization efforts that there’s not much to report!

Princeton was our smallest group so far—just 19 people, but the hotel was really nice. That afternoon we drove to Philadelphia, where we had about 55 people, including a couple of spies in the audience from our Philly-based web design firm who were there to get a sense of the kind of audience they were designing for.

I’ll interrupt this train of thought for a moment to talk a little bit about the gear I’ve been using. As I may have mentioned, my laptop is a Lenovo ThinkPad X61s, extremely small and light but with a comfortable full size keyboard. If you’ve ever thought about getting a ThinkPad but worried about the eraser-head trackpoint in between the G and the H keys that these things use as a pointer, don’t be. It takes a little time to get used to, but it works much much better than the more common touchpad because (a) you don’t have to take your hands away from the home row to use the mouse and (b) you never touch it accidentally while typing, causing the cursor to jump somewhere else.

My phone these days is the Samsung Blackjack, running Windows Mobile 5. Like Windows, it’s extremely frustrating and messy and disorganized. Also like Windows, if you’re willing to hammer away at it, you can make it do some pretty amazing things. In my world, amazing includes the fact that Liz can put things on my schedule and they’ll show up on my phone via over-the-air synchronization. Another thing I’ve come to rely upon on this trip is the high speed internet access… in most of the places I’ve been, AT&T has HSDPA access, which is pretty fast. But the phone stays in my pocket… the laptop connects over bluetooth to the phone which is running Internet Connection Sharing, a little applet that AT&T has tried to hide but which is still in the Windows folder on the phone. I’ve had HSDPA access in Seattle, Chicago, Philadelphia, Boston, New York, Princeton, even in the Hamptons. I just take out the phone, run the internet connection sharing applet and hit “connect,” then click on the appropriate bluetooth icon on the laptop, and, plink!, I’m on the internet at high speed.

That’s the main reason I don’t carry an iPhone — I use this HSDPA access every day and it rocks, and it’s inevitable that the next gen iPhone will have it, so I’ll wait, kthxbye.

OK, back to the trip report. Yesterday afternoon, I stopped by the ITA Software office in Cambridge (for all intents and purposes, the only large software shop that uses Lisp) to say hi and thank them for the useful search technology behind Orbitz, which made it possible to plan this trip even with all the multi-legged trips that brought Expedia to its knees. They were nice enough to take me out to dinner, too. Thanks!

This morning in Boston we had a huge turnout… 200 people who didn’t stop asking questions. For some reason which I can’t figure out, the demo part of my speech is taking a little bit longer every time I do it. I don’t think I’m adding things; I think I’m just explaining more. Who knows. Anyway.

There’s a new branch of Wagamama in Quincy Market. Looked exactly like the last one I ate at, in Sydney. A very nice addition to the otherwise dreadful dining alternatives of Boston’s Festival Marketplace.

The hotel internet in Boston was rather congested and I was having a lot of trouble doing anything online there, which is why yesterday’s Strategy Letter had so many typos. Sorry.

Brent Ashley: “I’ll provide some links here which will help the reader to understand how many of the points Joel makes in his essay are supported by existing technologies in various states of readiness. It’s a big pantry of ingredients that is waiting for the right chef to come along and combine them in a way that inspires the world to follow.”

Indeed countless people have already emailed me to say that “NewSDK is here, it’s (choose one) Flex Builder, Google Web Toolkit, Java Web Start, Silverlight, JavaFX, Flash, ActionScript, MORFIK, OpenLaszlo, … (many omitted)” Ahem. These are not HERE until your TAXI DRIVER has heard of them, because I assure you he’s heard of Microsoft Windows. Many of these technologies are developed by smart people who understand the world the way I talked about in the strategy letter, and are hoping to win the next platform war. But GWT is no more the NewSDK than Digital Research GEM, or IBM TopView, or Quarterdeck DESQView, or Concurrent DOS, or Microsoft Windows 1.0 was the OldSDK. They’re just horses at the starting gate.

I’m in Kitchener, Ontario right now, discovering that an even better predictor of a hotel I don’t really want to stay at is that it advertises that kings, queens, and presidents have stayed there. Sorry, darling, your hotel is charming, but I don’t care what your marketing materials say, if you really gave the Queen Mum these same shabby old towels as you gave me, Canada would be a republic by now.

Thursday morning is the Kitchener demo with 75 attendees; in the afternoon we’ll have an astonishing 240 people in Toronto, and then fly home. Tallyho!

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

IBM just released an open-source office suite called IBM Lotus Symphony. Sounds like Yet Another StarOffice distribution. But I suspect they’re probably trying to wipe out the memory of the original Lotus Symphony, which had been hyped as the Second Coming and which fell totally flat. It was the software equivalent of Gigli.

In the late 80s, Lotus was trying very hard to figure out what to do next with their flagship spreadsheet and graphics product, Lotus 1-2-3. There two obvious ideas: first, they could add more features. Word processing, say. This product was called Symphony. Another idea which seemed obvious was to make a 3-D spreadsheet. That became 1-2-3 version 3.0.

Both ideas ran head-first into a serious problem: the old DOS 640K memory limitation. IBM was starting to ship a few computers with 80286 chips, which could address more memory, but Lotus didn’t think there was a big enough market for software that needed a $10,000 computer to run. So they squeezed and squeezed. They spent 18 months cramming 1-2-3 for DOS into 640K, and eventually, after a lot of wasted time, had to give up the 3D feature to get it to fit. In the case of Symphony, they just chopped features left and right.

Neither strategy was right. By the time 123 3.0 was shipping, everybody had 80386s with 2M or 4M of RAM. And Symphony had an inadequate spreadsheet, an inadequate word processor, and some other inadequate bits.

“That’s nice, old man,” you say. “Who gives a fart about some old character mode software?”

Humor me for a minute, because history is repeating itself, in three different ways, and the smart strategy is to bet on the same results.

Limited-memory, limited-CPU environments

From the beginning of time until about, say, 1989, programmers were extremely concerned with efficiency. There just wasn’t that much memory and there just weren’t that many CPU cycles.

In the late 90s a couple of companies, including Microsoft and Apple, noticed (just a little bit sooner than anyone else) that Moore’s Law meant that they shouldn’t think too hard about performance and memory usage… just build cool stuff, and wait for the hardware to catch up. Microsoft first shipped Excel for Windows when 80386s were too expensive to buy, but they were patient. Within a couple of years, the 80386SX came out, and anybody who could afford a $1500 clone could run Excel.

As a programmer, thanks to plummeting memory prices, and CPU speeds doubling every year, you had a choice. You could spend six months rewriting your inner loops in Assembler, or take six months off to play drums in a rock and roll band, and in either case, your program would run faster. Assembler programmers don’t have groupies.

So, we don’t care about performance or optimization much anymore.

Except in one place: JavaScript running on browsers in AJAX applications. And since that’s the direction almost all software development is moving, that’s a big deal.

A lot of today’s AJAX applications have a meg or more of client side code. This time, it’s not the RAM or CPU cycles that are scarce: it’s the download bandwidth and the compile time. Either way, you really have to squeeze to get complex AJAX apps to perform well.

History, though, is repeating itself. Bandwidth is getting cheaper. People are figuring out how to precompile JavaScript.

The developers who put a lot of effort into optimizing things and making them tight and fast will wake up to discover that effort was, more or less, wasted, or, at the very least, you could say that it “conferred no long term competitive advantage,” if you’re the kind of person who talks like an economist.

The developers who ignored performance and blasted ahead adding cool features to their applications will, in the long run, have better applications.

A portable programming language

The C programming language was invented with the explicit goal of making it easy to port applications from one instruction set to another. And it did a fine job, but wasn’t really 100% portable, so we got Java, which was even more portable than C. Mmmhmm.

Right now the big hole in the portability story is — tada! — client-side JavaScript, and especially the DOM in web browsers. Writing applications that work in all different browsers is a friggin’ nightmare. There is simply no alternative but to test exhaustively on Firefox, IE6, IE7, Safari, and Opera, and guess what? I don’t have time to test on Opera. Sucks to be Opera. Startup web browsers don’t stand a chance.

What’s going to happen? Well, you can try begging Microsoft and Firefox to be more compatible. Good luck with that. You can follow the p-code/Java model and build a little sandbox on top of the underlying system. But sandboxes are penalty boxes; they’re slow and they suck, which is why Java Applets are dead, dead, dead. To build a sandbox you pretty much doom yourself to running at 1/10th the speed of the underlying platform, and you doom yourself to never supporting any of the cool features that show up on one of the platforms but not the others. (I’m still waiting for someone to show me a Java applet for phones that can access any of the phone’s features, like the camera, the contacts list, the SMS messages, or the GPS receiver.)

Sandboxes didn’t work then and they’re not working now.

What’s going to happen? The winners are going to do what worked at Bell Labs in 1978: build a programming language, like C, that’s portable and efficient. It should compile down to “native” code (native code being JavaScript and DOMs) with different backends for different target platforms, where the compiler writers obsess about performance so you don’t have to. It’ll have all the same performance as native JavaScript with full access to the DOM in a consistent fashion, and it’ll compile down to IE native and Firefox native portably and automatically. And, yes, it’ll go into your CSS and muck around with it in some frightening but provably-correct way so you never have to think about CSS incompatibilities ever again. Ever. Oh joyous day that will be.

High interactivity and UI standards

The IBM 360 mainframe computer system used a user interface called CICS, which you can still see at the airport if you lean over the checkin counter. There’s an 80 character by 24 character green screen, character mode only, of course. The mainframe sends down a form to the “client” (the client being a 3270 smart terminal). The terminal is smart; it knows how to present the form to you and let you input data into the form without talking to the mainframe at all. This was one reason mainframes were so much more powerful than Unix: the CPU didn’t have to handle your line editing; it was offloaded to a smart terminal. (If you couldn’t afford smart terminals for everyone, you bought a System/1 minicomputer to sit between the dumb terminals and the mainframe and handle the form editing for you).

Anyhoo, after you filled out your form, you pressed SEND, and all your answers were sent back to the server to process. Then it sent you another form. And on and on.

Awful. How do you make a word processor in that kind of environment? (You really can’t. There never was a decent word processor for mainframes).

That was the first stage. It corresponds precisely to the HTML phase of the Internet. HTML is CICS with fonts.

In the second stage, everybody bought PCs for their desks, and suddenly, programmers could poke text anywhere on the screen wily-nily, anywhere they wanted, any time they wanted, and you could actually read every keystroke from the users as they typed, so you could make a nice fast application that didn’t have to wait for you to hit SEND before the CPU could get involved. So, for example, you could make a word processor that automatically wrapped, moving a word down to the next line when the current line filled up. Right away. Oh my god. You can do that?

The trouble with the second stage was that there were no clear UI standards… the programmers almost had too much flexibility, so everybody did things in different ways, which made it hard, if you knew how to use program X, to also use program Y. WordPerfect and Lotus 1-2-3 had completely different menu systems, keyboard interfaces, and command structures. And copying data between them was out of the question.

And that’s exactly where we are with Ajax development today. Sure, yeah, the usability is much better than the first generation DOS apps, because we’ve learned some things since then. But Ajax apps can be inconsistent, and have a lot of trouble working together — you can’t really cut and paste objects from one Ajax app to another, for example, so I’m not sure how you get a picture from Gmail to Flickr. Come on guys, Cut and Paste was invented 25 years ago.

The third phase with PCs was Macintosh and Windows. A standard, consistent user interface with features like multiple windows and the Clipboard designed so that applications could work together. The increased usability and power we got out of the new GUIs made personal computing explode.

So if history repeats itself, we can expect some standardization of Ajax user interfaces to happen in the same way we got Microsoft Windows. Somebody is going to write a compelling SDK that you can use to make powerful Ajax applications with common user interface elements that work together. And whichever SDK wins the most developer mindshare will have the same kind of competitive stronghold as Microsoft had with their Windows API.

If you’re a web app developer, and you don’t want to support the SDK everybody else is supporting, you’ll increasingly find that people won’t use your web app, because it doesn’t, you know, cut and paste and support address book synchronization and whatever weird new interop features we’ll want in 2010.

Imagine, for example, that you’re Google with GMail, and you’re feeling rather smug. But then somebody you’ve never heard of, some bratty Y Combinator startup, maybe, is gaining ridiculous traction selling NewSDK, which combines a great portable programming language that compiles to JavaScript, and even better, a huge Ajaxy library that includes all kinds of clever interop features. Not just cut ‘n’ paste: cool mashup features like synchronization and single-point identity management (so you don’t have to tell Facebook and Twitter what you’re doing, you can just enter it in one place). And you laugh at them, for their NewSDK is a honking 232 megabytes … 232 megabytes! … of JavaScript, and it takes 76 seconds to load a page. And your app, GMail, doesn’t lose any customers.

But then, while you’re sitting on your googlechair in the googleplex sipping googleccinos and feeling smuggy smug smug smug, new versions of the browsers come out that support cached, compiled JavaScript. And suddenly NewSDK is really fast. And Paul Graham gives them another 6000 boxes of instant noodles to eat, so they stay in business another three years perfecting things.

And your programmers are like, jeez louise, GMail is huge, we can’t port GMail to this stupid NewSDK. We’d have to change every line of code. Heck it’d be a complete rewrite; the whole programming model is upside down and recursive and the portable programming language has more parentheses than even Google can buy. The last line of almost every function consists of a string of 3,296 right parentheses. You have to buy a special editor to count them.

And the NewSDK people ship a pretty decent word processor and a pretty decent email app and a killer Facebook/Twitter event publisher that synchronizes with everything, so people start using it.

And while you’re not paying attention, everybody starts writing NewSDK apps, and they’re really good, and suddenly businesses ONLY want NewSDK apps, and all those old-school Plain Ajax apps look pathetic and won’t cut and paste and mash and sync and play drums nicely with one another. And Gmail becomes a legacy. The WordPerfect of Email. And you’ll tell your children how excited you were to get 2GB to store email, and they’ll laugh at you. Their nail polish has more than 2GB.

Crazy story? Substitute “Google Gmail” with “Lotus 1-2-3”. The NewSDK will be the second coming of Microsoft Windows; this is exactly how Lotus lost control of the spreadsheet market. And it’s going to happen again on the web because all the same dynamics and forces are in place. The only thing we don’t know yet are the particulars, but it’ll happen.

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

Back home in New York. We had about 75-100 people come to the New York demo yesterday, along with an army of Fog Creek technical staff in matching sky blue kiwi polo shirts.

When I got back to my desk on Monday afternoon, I turned into the prototypical bastard client from hell. Our web designers probably hate me. I did the one thing that drives web design firms completely crazy: I suddenly took a look at the new web design they've done for us, which I've been approving every step of the way, and didn't like it any more, so I told them we had to start over.

In one of Gerald Weinberg's books, probably The Secrets of Consulting, there's the apocryphal story of the giant multinational hamburger chain where some bright MBA figured out that eliminating just three sesame seeds from a sesame-seed bun would be completely unnoticeable by anyone yet would save the company $126,000 per year. So they do it, and time passes, and another bushy-tailed MBA comes along, and does another study, and concludes that removing another five sesame seeds wouldn't hurt either, and would save even more money, and so on and so forth, every year or two, the new management trainee looking for ways to save money proposes removing a sesame seed or two, until eventually, they're shipping hamburger buns with exactly three sesame seeds artfully arranged in a triangle, and nobody buys their hamburgers any more.

This is sort of what happened with our new web design. We've been tweaking it and polishing it and changing things carefully, and the firm we hired to design it has been taking us step-by-step through information architecture, site maps, wireframes, initial designs, and several rounds of design. All with a carefully-designed process to get our buy-in at every step along the way. And so far every step I thought the design was converging and we'd get a nice web design out of it.

And then I came back after a week on the road, took one look at it, and thought, oh crap. We can't go public with that.

And they said, "but wait, look here, it's right in Basecamp, you said that this design was 'excellent work' and you were 'elated' to have the 'best web design ever in the history of the universe.'"

True that. I did say that. I even thought that.

But a week later, the same basic design just looked terrible. We've been removing sesame seeds from the initial design they did in hopes of making things better, and, lo and behold, at some point the design flipped from being good to being bad. Links had sprouted up all over the place, making it hard to tell where to go next and where you've already been. Most of the elegant whitespace in the original design was lost when we went from the original 1024 pixel wide design to an 800 pixel design. The web designers had presumably been working on Macs and showing us bitmaps, but since the antialiasing technology is different, when we finally got the HTML, the page just felt completely different and had crossed into the realm of plain and, subjectively, ugly.

Ah, well. We'll start over. It's better to have something we're both proud off than to try and salvage the work done so far. Sometimes you have to go all the way through the design process before you realize that you've built the wrong thing, but it's ok, it's a learning experience, it's not the end of the world to take a deep breath and go back to step 1.

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

Occasionally I think about organizing my own conference, but I would just invite people like Eric Sink and Rick Chapman, anyway, so the Business of Software 2007 conference has eliminated the need to do that.

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

Chicago; about 70 local software developers turned out. Chicago is a great city for architecture. Much better than New York. Here they build skyscrapers just because they love them.

Can I talk about hotels for a minute? There are about a million different ways to rate hotels. That makes user review sites, like TripAdvisor, somewhat hit-or-miss. One person’s hovel may be another person’s palace.

I’m sad to say that the Congress Plaza Hotel where we did the event at this morning does not qualify as anyone’s palace. The usual nice words you might use to describe such a hotel would be “threadbare” or “shabby.” Other words (“maccabre,” “Barton Fink,” and “scuzzy”) come to mind. This was entirely my fault; I set a target budget for hotels in each city and didn’t do the research to make sure the hotels would be entirely nice.

I’ll bet you can tell almost everything you need to know about the quality of a hotel based on how often they replace the sheets and towels. Another good indicator, for some bizarre reason, is plasma TVs. The nice chains, inexplicably, have old fashioned big-ass tube TVs. The ancient rotting edifices have 32” plasmas. I don’t know why this is. Maybe they think that having a plasma TV they can advertise on their web site will make them seem fancy.

Anyway, the Congress Plaza Hotel is the kind of 850 room monstrosity that Lot Polish Airlines would fill up with a 777 full of passengers, on their way to Warsaw, if the plane was stuck in Chicago overnight due to mechanical failure.

Oh. And the staff was actually on strike. So people coming to the demo had to cross a picket line. I’m sorry about that. I never thought to ask if there was a strike at the time we booked. I guess shabby hotels just treat their employees shabbily. Apparently Chicago considered passing a law requiring hotels to tell people about these strikes when they booked rooms and meetings. It didn't pass.

The lesson from Chicago is that using cheap hotels is not a good idea for business meetings. Psychologically, I think that people tend to associate the environment they’re in with the presentation. When a demo is in a modern, new, shiny business hotel, it’s like a little one hour vacation in luxuryland. You go to the bathroom and it’s marble everywhere and individual cloth hand towels. And you think nice things about the demo. But when you go to a demo at the Congress Plaza and the rug is stained and there are fluorescent lights everywhere and the bathroom looks like LaGuardia airport, some of that general depressing aura of shabbiness will rub off on the product being presented.

After the demo was over I walked two blocks south to the Hilton where the Inc. 500 Conference was in progress. There I spoke to a bunch of small companies about how we hire people at Fog Creek. A lot of the material I talked about is available on this site, as a series of articles I wrote about a year ago:

• Finding Great Developers
• A Field Guide to Developers
• Sorting Resumes
• The Phone Screen
• The Guerrilla Guide to Interviewing

You can get the whole series plus one bonus chapter in book form, as well.

I’m flying back to New York now, on We’ve Pretty Much Just Given Up Airlines. On Monday morning, if you’re in the city, please join me and the FogBugz development team at 9:00am for the FogBugz 6.0 World Launch, at the New York Marriott East Side Hotel. It’s free, but you have to register to reserve a space.

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

About 200 attendees came in Seattle; this will be one of our biggest scheduled demos except, maybe, London. I apologize that the screen wasn’t so easy to read throughout the room. We made a point of asking the hotel before we booked the room if the screen would be clearly visible to all attendees. It wasn’t, because the ceiling wasn’t high enough, and the hotel didn’t have the ability to raise the ceilings, so of course they lied and told us it would be fine, and it wasn’t. This is a problem at every tech conference I’ve ever been to; these rooms were built long before PowerPoint.

A bunch of Microsoftees had to cancel because they are having their annual company meeting today in a gigantic football stadium. The size of that company is insane. Can you imagine Safeco Field filled to the brim with software developers? And that’s just the Vista Shutdown Menu Team.

Something that came up in the demo today that I wanted to share. Software development is a cycle with three distinct phases: design, development, and debugging. It doesn’t really matter whether you’re doing Ultra Extreme Elite Programming, in which case you do all three phases in one week, or The Ancient OS/360 Waterfall Method, in which case you do them over the course of a year or two. You still have to design what you’re going to build, build it, and then debug it.

The three phases have to be scheduled in a very different way.

• Design is the art phase, where you’re doing new, creative work. Even though what you’re doing is completely new, after you’ve gone through a few software development cycles you’ll start to get a pretty good idea of how much time it takes to design a new version of your software. I’ve usually worked with relatively long development cycles of 12-18 months, and it’s always taken me about two months to get a detailed, first-draft spec containing enough detail for the development team to create very granular estimates.

  That said, when you’re building something brand new from scratch, you really can’t estimate the design phase at all, and that’s OK. Today I met somebody from a company in Seattle that’s working on a project headed up by one of the world’s great programmers, Charles Simonyi. Near as I can tell, they have been in the design phase for 16 years.

• Development is the engineering phase. It’s a construction project. As long as you start with a detailed blueprint, which, of course, can change over time, but which is really your best guess for what you’re building, this phase can be scheduled with great precision. FogBugz 6.0 has a spiffy new feature called Evidence-based scheduling, which uses a variation on the Monte Carlo method for making your schedules remarkably reliable during the development phase. When I get a chance, I’ll you about it in more detail.

• Debugging is the science phase. Science is difficult to schedule because you’re looking for things, and predicting when you’re going to find them is remarkably difficult. Unless you know in advance how many bugs you’re going to find, you don’t have an ice cube’s chance in the Sahara to work out a detailed estimate of how long this phase will take. Here at Fog Creek we’ve learned that for a new release of FogBugz, this phase takes at least 12 weeks, sometimes a little more, and we just leave it at that.

  Some people are more ambitious and try to track the rate at which bugs are being found and being fixed, and try to extrapolate to see when you’ll ship; in practice I’ve found that the rate of finding bugs is way too messy to be able to extrapolate from. If you have a fixed size test team and infinite bugs, they’ll find bugs at a constant rate, simply because they’re all spending 8 hours a day entering bugs and then stopping, but this flat line doesn’t mean you’re ever going to ship. You’re not. You have infinite bugs, remember? Sorry. On the other hand you may have just released a new beta to a new batch of beta testers, and it’s the best beta yet, but it’s the biggest group of beta testers, so there will be a big spurt of new bugs found, which doesn’t mean the code is getting buggier—it just means you have a bigger group of beta testers.

Now, there are various ways to get in trouble. If you don’t like writing functional specifications or doing up-front design, what happens is that you’re burdening the design phase with the development phase. If you ever started a new project by writing code, and you thought you’d “design as you went along,” what you’re doing is driving around with the handbrakes on. Here’s why. Designing a feature by writing a thoughtful spec takes about 1/10th as much time as writing the code for that feature—or less. If you try to code as you design, then you’re interrupting your short spurts of design with long spurts of coding. Now, if you’re the kind of person who designs everything perfectly the first time, that’s fine. But I don’t think you are. I think that your first designs are pretty good, but when you see them, you get ideas for even better designs. And if you already coded up the first draft, bad design, well, that’s coding time wasted. Your product’s design can only get better at 1/10th the speed that my product’s design can.

I’m now en-route to Chicago, where I have to do two speeches tomorrow: the morning FogBugz demo, and, in the afternoon, I’ll be talking about hiring to a bunch of startup CEOs at the Inc. 500 conference. I’m editing this in the e text editor, a Windows clone of TextMate, which is coming along nicely but could still use some polish before I’m ready to switch to it full time.

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

Vancouver, BC: Day one of the FogBugz World Tour. About 120 people showed up to see the first public demo of FogBugz 6.0, which will officially launch next Monday.

Vancouver is, without a doubt, one of North America's most beautiful cities. Sparkling, clean, everything works well, nothing can possibly go wrong, people are friendly, and with the new weakened US dollar it's really quite a prosperous place to live. Brett and I had dinner at Joe Fortes, where you get a choice of 4 different local species of salmon, maybe 20 other kinds of fresh fish, or about 10 different type of oysters, and there's a beautiful rooftop deck where you can enjoy the usually pleasant Vancouver weather.

The demo went relatively smoothly, despite a few first-time kinks. At some point I was fiddling around so much with the report generator that I queued up a backlog of lengthy Monte Carlo simulations on the web server which made FogBugz lose interest in continuing with the demo; this not the kind of thing that happens in production web servers (part of the problem is that the laptop is running XP which has a kind of 3/4-baked implementation of IIS, version 5.1, which is not what anyone would run on a real server). Anyway, I had to restart IIS in the middle of the demo. Ooops. Hopefully that won't happen again.

It couldn't be that bad. Here's some email feedback we already received from the demo:

“Thank you for the entertaining and informative talk. We will be buying FogBugz as a result.” Thank you! OK, we just broke even in Vancouver.

“Do you intend to provide free versions of FogBugz for open source projects, non-profits, or small teams of 2 people (like Perforce does with their products)?”. Yes, it's called the Student and Startup Edition. We'll announce it soon, but it's available now.

“Well, I expected to be bored about FogBugz and enraptured by fascinating tidbits of Joel Spolsky wisdom. In reality, the opposite was true. I now believe FogBugz to be a pretty interesting looking app, whereas an hour before your spiel I think I described it to someone as a "glorified Excel spreadsheet" (it's amazing what I can come up with when I am not encumbered by facts).” My tidbits aren't that fascinating.

“ Having just attended Joel's FogBugz 6.0 demonstration in Vancouver, we were very impressed with its capabilities.  My boss wants to go ahead and use FogBugz, however Joel mentioned that the Linux/Unix version was still in Beta.” For Unix we're still on 5.0 while we debug the PHP port, which I hope won't take long. While you're waiting for Unix FogBugz 6.0, you can either run 5.0 -- and upgrade for free -- or run a free trial on our server, and download the data when we're shipping.

The night before, in rehearsals, I discovered that the Fn+F7 trick that is supposed to turn on external monitors on this Thinkpad was actually freezing the computer solid, due to some kind of buggy interaction between the Intel 965 graphics chip software and the IBM/Lenovo Presentation Director software. I never did solve that problem, so I learned to use the Intel software to turn on the external monitor instead of pressing Fn+F7.

Flying in September after Labor Day is really not that bad, despite the scare stories you might have heard in the press; once everyone gets home from summer vacations the number of passengers in airports and on flights drops quite dramatically and flights start operating closer to schedule with much shorter lines. So far I don't think we've waited in one line at an airport. Here are my favorite tricks for planning air travel to avoid chaos, delays, and cancellations:

1. The ideal time to fly is around 10 am. Usually delays pile up throughout the day, so the earlier you fly, the less likely you are to suffer delays. The very early flights are popular with people who want to get a full day in, so the midmorning flights tend to be the most civilized.
2. Always check the OAG before booking to see what flights are available. The OAG includes JetBlue and Southwest flights which the online travel agencies can't show you.
3. Make sure you're never on the last flight of the day if you really need to get somewhere on schedule. If something happens to the last flight, you're in trouble. As a general principle, while planning for this trip, I always checked that there was at least one alternative flight that would get me to my next destination on time. Since we fly first class at Fog Creek, if one of our flights got cancelled, the airline will work hard to reaccomodate us while the coach passengers might have to wait forever for a rebooking.
4. Fly out of smaller airports whenever possible. My favorite alternative airports: John Wayne or Burbank instead of LAX, Ft. Lauderdale instead of Miami, Love instead of DFW.
5. If the flight you're booked on is cancelled, don't wait in line with the crowds for the single, overworked airline representative. Get on the phone to your airline's frequent flyer priority number. They can rebook you just as well.
6. The American Express Platinum card pays for itself just from the free membership in Continental, Northwest, and Delta's lounges... not only because the lounges are quiet and pleasant, but because the lounges have unharried and experienced airline agents who are happy to help you with complicated problems, rebookings, and upgrades.
7. Final trick: never schedule an important flight during the last few days of the month, especially on Northwest. Pilots are only allowed to fly a certain number of hours per calendar month and by the end of the month they're running out of hours, especially on the more awfully-managed airlines like Northwest, so flights galore get cancelled in the last few days of every calendar month.

See you tomorrow in Seattle! There are still five seats available if you haven't registered yet.

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

Updated web site: A couple of months ago I attended The Server Side's Java Symposium in Barcelona. Neal Ford paired with me on a keynote on Domain Specific Languages and related beasties and this talk is now available on video (and transcript). Neal will be joining me for a half-day tutorial on this topic at QCon in San Francisco later this year.

I bought a retail copy of Office 2007 today (I'm loading up the new laptop I got for the world tour, which is a Thinkpad X61s), and I must be a complete spaz, but I simply could not figure out how to open the bizarre new packaging.

It's a hard plastic case, sealed in two different places by plastic stickies. It represents a complete failure of industrial design; an utter F in the school of Donald Norman's Design of Everyday Things. To be technical about it, it has no true affordances and actually has some false affordances: visual clues as to how to open it that turn out to be wrong.

This is the same box that Vista comes in. Nick White over at Microsoft seems proud of the novel design, but from the comments on the web it seems I'm not the only one who couldn't figure out how to open it. It seems like even rudimentary usability testing would have revealed the problem. A box that many people can't figure out how to open without a Google search is an unusually pathetic failure of design. As the line goes from Billy Madison: "I award you no points, and may God have mercy on your soul."

Wasting five minutes trying to get the goddamned box open is just the first of many ways that Office 2007 and Vista's gratuitous redesign of things that worked perfectly well shows utter disregard for all the time you spent learning the previous versions.

I've tested Office 2007 extensively, and find it a tolerable replacement for the previous version, although it's extremely frustrating every time I have to spend several minutes finding something that I knew exactly how to find in the previous version. Even though there's no reason to upgrade to Office 2007, if you're setting up a new system, it's just as good as the previous version, even a little better in some places. But Vista is another story.

I've been using Vista on my home laptop since it shipped, and can say with some conviction that nobody should be using it as their primary operating system -- it simply has no redeeming merits to overcome the compatibility headaches it causes. Whenever anyone asks, my advice is to stay with Windows XP (and to purchase new systems with XP preinstalled).

PS The FogBugz 6.0 World Tour is filling up fast. Austin is already full. Vancouver, Seattle, Kitchener, and Irvine have just a few seats left. Sign up now!

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

What's the point, exactly, of physically going to 21 different cities to demo FogBugz 6.0 in person? Why not just put a video up on the web and be done with it?

A long long time ago when Windows 3.0 first came out, Microsoft organized a huge, permanent "Windows Seminar" team of intelligent, charismatic young people who went from city to city giving demos of Windows, Word, and Excel. Back then, showing someone cut and paste from one GUI window to another was astonishing.

Since then, I haven't seen a lot of demo tours like the one we're planning. Travel just costs too much. Business hotels all charge ridiculous amounts for catering ($15 per person for coffee), audio visual equipment ($500 to rent a projector), and as many additional gougy-charges as they can think of. $50 for sneezing. $120 to have the window shut. And $92 to have the lights on.

When we did up our budget, including hotel rooms, meeting space, rental cars, airfare, food, a projector, coffee and tea for attendees, and a printed brochure to hand out, our estimate is that it's going to cost us somewhere between $60 and $70 per person.

So, OK, maybe I don't know what I'm doing.

The reason we finally decided to pull the trigger is that we don't have a sales force. None. We have inside sales people who handle incoming phone calls (very rare) and incoming email (we get a lot of it), but our software is just priced too cheaply for the traditional enterprise software sales system, where you have a bunch of commission-based suits flying around the country, staying in nice hotels, and taking clients out to lunch. That kind of sales force costs $50,000 to make one sale. With prices in the $21/month range, that model just doesn't work for us.

When you think about it this way, $60 to reach one person is cheap!

Here's what you can expect if you show up. There will be some really, really expensive coffee and maybe a muffin or something if the hotel in particularly cheap. If you come a bit early, it'll be a great chance to meet some interesting software people in your city.

I'll show off some of my favorite features in FogBugz 6.0, but mostly, I'll talk about the software development process in general. I'll leave a lot of time for Q&A. There will be a member of the FogBugz development team with me and we'll also leave plenty of time afterwards for one-on-one questions if you have something to ask us that isn't of general interest. Then we'll pack our projector and rush off to the airport and on to the next city.

So far, we've planned 21 cities in North America. If you're in the US or Canada, you can register right now, in fact, you should, because we expect to fill up really quickly. If you register and can't make it, please cancel (you'll get a cancellation link in the confirmation email) so we can take someone else from the waiting list.

We are planning to visit international cities, but we haven't figured that bit out yet. We'll probably hit Europe, Australia, and New Zealand in November. If you're outside the US and want to attend, make sure you've filled out the survey with your correct email address so we can figure out where to go next and let you know when we're coming.

If you can't make it, we'll have a video of the New York City world launch of FogBugz 6.0 on our website sometime in September.

So. Click on the cute kiwi to register:

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

Updated web site: Continuous Integration is a topic that often brings people to my site. As a result it seems appropriate that I can now announce that Paul Duvall's book on the subject (part of my signature series) is now out.

OneLanguage design 28 July 2007 Reactions

Should we strive to only have one language in our development efforts?

Throughout much of the last decade the fashion in the enterprise software world has been to focus on one standard language for software development efforts. Many development organizations strive to do all their work in Java (or C#/VB).

The rationale for this is that developers find it hard to be proficient in more than one language. Sticking to a single language lowers the learning burden, particularly when hiring new people.

There's some truth to this, but also much that's missing. The programming environment is partly language, but also about languages and frameworks. Larger frameworks, Hibernate, Struts, ADO, present as much of a challenge to learn as a language even if you program them in a single host language. Often the difficulty of expressing what you need in the host language is sufficiently awkward that many frameworks resort to configuration files, which are effectively external DomainSpecificLanguages written in XML - which adds a jigger of 80 proof ugliness to them.

For many developers, the one-language notion is a sign of lack of professionalism. This is best exemplified by the Pragmatic Programmers' advice to learn a new language every year. The point here is that programming languages do affect the way you think about programming, and learning new languages can do a lot to help you think about solving problems in different ways. (It's important to learn languages that are quite different in order to get the benefit of this. Java and C# are too similar to count.)

I agree with prags' advice here, as in most things. But I also sympathize with the overhead of learning a new language. My personal scripting is pretty much all done with Ruby and I've been loath to any more than play with reasonable alternatives like Python, Groovy, or PowerShell. It's not that it's hard to use the alternatives, but with Ruby I know too much that I'd have to look up with the alternatives.

The important point here is that when I'm writing these scripts I'm not manipulating new abstractions. Much of what I do is fiddling with text, the file system, and hunks of XML or YAML. If I need to take on a sizable new abstraction, the cost of learning it as a library isn't really much less than the cost of learning a language to manipulate it. If I want to specify a directed graph structure for display, learning Graphviz's Dot language is hardly more work than learning a new Ruby library.

Using a DSL instead of a library can offer us better ways of manipulating our abstractions. This makes it easier to see what we've written and to reveal our intentions. An API is like declaring a vocabulary, a DSL adds a grammar which allows you to write coherent sentences.

This argument is strong for DSLs, but does it also apply to general purpose languages? If you are working in Java (or soon C#) does it make sense to use Ruby now it's available on your platform?

The last decade has seen the rise of memory managed C-based languages. People saw that, despite many years of skepticism, memory management makes life sufficiently better that it was worth stepping away from C and C++ in the enterprise world. A common platform and language also pulled people away from proprietary walled gardens like Powerbuilder and Delphi.

Now there is a similar question. Are modern scripting languages another step forwards? Do we prefer their well-chosen terseness? Time and time again I hear experienced Java and C# developers report they are more effective in Ruby - which is why I've been encouraging Ruby. It wouldn't surprise me if similar reports appear in the next few years about other languages too.

A decade ago I was talking to my old friend Tom Hadfield at OOPSLA 96. Java's rise was apparent and it was clear that Smalltalk's future was doomed. Despite my love of Smalltalk I was pretty sanguine. I felt that Java gave people enough of what they needed; while it wasn't quite as nice as Smalltalk it was enough of an improvement over C++, particularly with memory management, for me to be happy with it. Tom disagreed, he felt there was something fundamentally different about the expressiveness of Smalltalk, the way you could better capture the intention of what you were doing directly in your code - closing the gap between domain knowledge and programming.

In the intervening years I've come to the view that Tom was right after all. After several years in curly brace land, Ruby reminded me of what I was missing. There's a clarity to reading Ruby code that just makes it an easier medium to work with, despite the inferior tooling. I'm way more sympathetic to the Smalltalk holdouts than I felt then, even though I haven't felt inclined to open an image in anger for a long time.

So are we returning to the language cacophony of the late 80's and early 90's? I think we will see multiple languages blathering away, but there will be an important difference. In the late 80's it was hard to get languages to inter-operate closely. These days there's a lot of attention to making environments that allow different language to co-exist closely. Scripting languages have traditionally had an intimate relationship with C. There's much effort to inter-operation on the JVM and CLR platforms. Too much has been invested in libraries for a language to ignore them.

So my sense is that we will see multiple languages used in projects with people choosing a language for what it can do in the same way that people choose frameworks now. I agree with Neal that we are entering a period of Polyglot Programming.

InstallingOpenArchitectureWare dsl 27 July 2007 Reactions

There are few things more frustrating than spending hours trying to install a piece of software and then having to delete everything and start again. Today at 9.30 I began installing openArchitectureWare, I finally had it installed (I think) at 15.30. So I thought I'd write this to help someone else do it more quickly.

OpenArchitectureWare is a set of tools, based on Eclipse, to support Model Driven Development. I'm interested in exploring some of its tools that are oriented towards DomainSpecificLanguages. (Xtext - which helps you develop textual languages - is something that's specifically been pointed out to me as worth looking at.) I don't know how worthwhile these tools are yet, after all it took me most of the day just to install the dratted thing, but we'll see.

One of my problems with the installation was that I'm not an Eclipse user - my usual Java IDE is IntelliJ. To install openArchitectureWare you need to know how to deal with the plugin system in Eclipse - and I'd never done anything with Eclipse before so that was new to me.

The first step was the easiest one - install Eclipse. I installed it on my Ubuntu machine, so all I had to do was wajig install eclipse (wajig is a unified command-line for various debian packaging and sysadmin tools). Then all hell broke loose. Rather than go through my miserable morning, I'll explain what I would do now.

The trouble with OpenArchitectureWare is that it has dependencies, other eclipse plugins that need to be installed before it can work. As anyone with experience in these things knows, sorting out dependencies can be a right pain without a good tool. apt-get for Debian and gem for ruby are examples of a good tool that resolves dependencies. When I installed eclipse, apt-get knew it had to pull down a whole host of dependencies and installed them for me. The situation in Eclipse is not so good.

To install openArchitectureWare you need a bunch of plugins: EMF, UML2, ATL, and GMF. I couldn't see from the web pages exactly how to get these things, or if they had their own dependencies.

There are several ways of installing plugins in Eclipse, although I had to hunt a bit for instructions. The easiest way is a menu option in Eclipse itself. In the menus pick [Help -> Software Updates -> Find and Install] (no I don't know why it's on the help menu). With a bit of button pushing you can get it to download a list of packages - the relevant source is the Callisto Discovery Site. Once you have that list downloaded look in the Models and Model Development section and select Eclipse Modeling Framework (EMF) and Graphical Modeling Framework (GMF). You'll get an error message saying that these have an unresolved dependency. Take note of the button on the right that says 'select required'. Hit it and it will find the dependency to GEF and its dependency on Batik. If you don't see that button and hit it you'll have a frustrating time trying to find them (believe me, I know).

That gets two of openArchitectureWare's dependencies. The others, and openArchitectureWare itself need to be done the harder way. Digging around the eclipse site I found the relevant web pages for UML2 and ATL. These need to be downloaded as zip files as does openArchitectureWare itself.

When you unzip the UML2 and openArchitectureWare folders they unzip into a folder called eclipse that contains subfolders for plugins and features. You can take the contents of these folders and put them into corresponding folders on your load environment (in my case /usr/local/lib/eclipse). As that didn't work for me when I tried it first, I found another way.

The way to tell if stuff has installed properly is to go to [Help -> Software Updates -> Manage Configuration]. When you open that you have the option of "Add an Extension Location". An extension location is (almost) any directory that contains an eclipse folder with subfolder for plugins and features. I say almost because the eclipse folder also needs a file called .eclipseextension. This is just an empty file so you can create it with touch .eclipseextension. What I did is created folders in /usr/local/lib for openArchitectureWare and uml2-eclipse, moved the unzipped eclipse folders in there, did touch .eclipseextension inside each of them and then added them using "Add an Extension Location". ATL just produces a plugin directory so I copied the contents of it into the plugin directory for openArchitectureWare.

It's important that you do this after you use the Find and Install tool because if you do it first, the Find and Install tool will tell you have an unresolved dependency and refuse to do anything until you fix it. When I was all installed it tells me "UML2 End-User Features (2.1.1.v200707181556) requires plug-in "org.eclipse.emf.ecore.xmi (2.3.0)". I don't know how to fix this and I have a bunch of emf.ecore jars present in EMF. However the rest of eclipse seems to work so far, so I'm carrying on regardless.

Liz and I have been working to get ready for the FogBugz 6.0 World Tour, in which I travel from city to city (accompanied by a programmer) giving demos of all the cool features in the upcoming version. This is basically about the same logistical complexity as planning a family trip to DisneyLand, times 32.

Very little has been finalized, but it looks like the first round will hit Vancouver, Seattle, Chicago, New York, Princeton, Philadelphia, Boston, Toronto, Waterloo, Washington, Atlanta, Dallas, Austin, Denver, San Francisco, Berkeley, Newark (CA), Mountain View, Santa Monica, Irvine, and San Diego. The second round will hit Auckland, Christchurch, Brisbane, Melbourne, Sydney, Munich, Copenhagen, Amsterdam, London, Cambridge, and Dublin. Whenever my squirt nephew decides to have his bar mitzvah, we'll come to Tel Aviv.

Given the cost of flights, hotels, rental cars, meals, catering (we'll serve refreshments), printing (we'll pass out brochures), FogBugz polo shirts, etc., it looks like the cost for this trip is going to come to about $64 per attendee... that is, it's costing us $64 to put on a live demo of FogBugz for each person who attends. Phew. No wonder people do those crappy webinar things instead. I still think it's worth it, though.

Not loving your job? Visit the Joel on Software Job Board: Great software jobs, great people.

I taught my Pragmatic Project Management workshop in Israel last week. I was talking about defect charts and what they mean and how I use them. (No, I don't include priority or severity data on defect charts; just # opened and closed by week and # remaining open each week.

One of the participants explained that he also tracks # Fixed Private Defects. What are those? Fixed defects in a developer's sandbox, not checked into the mainline. Why are they not checked into the mainline? Because the organization does staged integration of several highly complex sub-projects, each of which has interdependencies with the other. And the build takes over a day.

This participant's defect data is not about defects. This data is about the cost of the build and possibly the product's architecture. When you can't build the whole product every day (this organization builds a couple of times a week), you create a bottleneck of fixes waiting to be integrated into the build. You slow down development--dramatically at the end of a project, badly enough during the project--to keep the build going.

Let's run a few numbers and see what it costs them to continue staged integration. Let's assume it costs just one person-day to create a fix. (That's optimistic, but not unrealistic.) Now, assume that for three months of a project, they have a weekly average of 50 "private-fixes" waiting to be integrated. (I'm assuming the 50 here, I have no data. But I think it's not far off reality.)

To keep staged integration going, they spend 50 person-days every week for 12 weeks. (Remember, this is a large project, really a program.) That's 600 person-days. I wonder if they could spend 600 person-days and fix the build so that they could build every day, without the need for staged integration. I don't know, but I suspect they could.

Watch for what your data is telling you. You might think you're measuring one thing, but you're really measuring something else. In this case, the defect data is not about defects; it's about the cost of an inadequate build system. What else are your defects telling you about your system?