Out of all the talks and discussions at Agila Sverige 2009, the most useful tidbit I took with me was this:

Evolution = Variation + Selection

The equation is obvious, and yet it was useful to have it spelled out like that. The moment I saw it on the screen, I knew it was just what our Agile project needed right now.

I’d been thinking for some weeks now that our project felt a bit stagnant. Everything was going well, we were delivering good stuff, the process was flowing smoothly, but there was something missing: change. And if there’s no change, there’s no improvement, and I get restless.

We’d had our regular retrospectives, and we’d always come up with some points for improvement. But I felt that we were often saying the same thing every month, and that any adjustments we made were small, on the margin. Looking back (with that equation in front of me) the reason is obvious. We had a good deal of selection (critical evaluation) but far from enough variation. We need to experiment more, change our routines and our processes more (more often, more extensively, more deeply, more creatively) and see what happens.

The first thing we’re going to change (after we all get back from our vacations) is try 1-week sprints/iterations instead of 4-week ones. And I’m going to spend a lot of time during the summer doldrums (when we’re between iterations) thinking up more variations to try.

I spent Monday and Tuesday at Agila Sverige (Agile Sweden). A lightweight and agile event, as befits the topic, somewhere in the continuum between the ALT.NET unconferences and “real” conferences. There was a small attendance fee, and meals were provided, and there were printed programmes, but the structure was like an unconference – lots of scheduled lightning talks in the mornings, and Open Space sessions in afternoons.

As with the ALT.NET events, I came home not so much with facts or techniques or tips, but with feelings of inspiration and energy. While it would have been nice to find answers, it is also nice to at least know that others are struggling with the same questions (such as fitting testing into short iterations, or adjusting Agile & Scrum for use in a very small team). The term mutual admiration society may be derogatory, but sometimes a bit of mutual admiration is what we need to energize us.

The most interesting discussion was a philosophical-political one. I cannot recall the original label for the session, but one of the topics we ended up discussing was the parallels between some dynamics of the Agile movement and “real-world” revolutionary movements. Liberation movements themselves are often very illiberal, even autocratic, and it can be hard to shift from leading a liberation movement to leading a liberated population. Very different qualities and behaviours are required.

The Agile movement is a kind of liberation movement, aiming to free developer teams from the burden of excessive management, procedure, planning and documentation. But when the message trickles through to the masses, the original leaders think the message has been distorted, lament the degradation that their lofty ideas have undergone, and start talking about the need to restart the process under a new name and a new manifesto. And the masses get upset that their leaders seem to be abandoning them.

The Agile movement is a principles-based movement. It’s not so much a method as a philosophy. (Aren’t real-world revolutions the same?) And for the leaders of the movement, principles suffice. But when the idea spreads (or when you try to spread the idea) from innovators and early adopters on to the early majority, it starts to seem that principles are no longer enough. The target audience now includes people who do not think much about principles – because if they did, they would be leaders or early adopters themselves. So processes and methods and rules start to agglomerate. “If you’re not doing TDD then you’re not doing Agile,” and so on. So there is an inherent conflict between the foundation of Agile software development (trusting people), and the desire to spread it to the bulk of the developer population (whom you don’t trust to do the right thing on their own).

(This, of course, raises a new question: should we even try? Would it be desirable to have the majority of projects run in an Agile manner? Or, as another speaker put it, are we running out of Agile developers? Can any developer work in an Agile manner, or does it take a certain kind of developer, of which there is a finite number?)

The issue of leaders abandoning a movement once it reaches some level of maturity is also an interesting one. (Lean development is one of the new new things that people orient towards when they get enough of Agile.) There are at least two forces conspiring to make it so. Firstly, thought leaders are rebels and/or innovators. In order to feel happy they need to feel that what they’re doing is new and fresh, and not mainstream. When their idea becomes mainstream, they must find something even newer. Secondly, the leaders will naturally continue to develop and refine their own ideas. They won’t stop just to wait for the rest of us to catch up. So by the time we get to Agile, the people who started Agile will again be ahead of us.

The Rails learning project continues. I’ve got an actual application to work on, one that will be useful when it’s done: a web-based version of our expense tracking application. The old one was done in Access and has served us well over many years. But file sharing has always been a pain since we often have different OS versions (and now different OSes) on our two computers, and can’t be bothered with hooking up external hard drives etc. So we want a solution that we can both access more easily. Thus, web, and Rails.

It’s fun and educational but it is also painful. The trivial tasks are trivial, but the moment I try to do something other than plain CRUD with strings and numbers, I get stuck. Every task takes me a whole evening. At this rate the application won’t be done until Christmas.

Today’s stuckness: time zones. I want to be able to migrate the data from our existing database, and CSV files seemed like a sensible medium. Exporting from Access to CSV went easily. Importing text went well enough, once I figured out that I needed to open and resave the file with different encoding. But importing dates did not. My data file had an entry for 1998-01-01 and after importing it somehow became 1997-12-31 23:00. When I got that fixed (by changing the application’s time zone to UTC) the data looked right but whenever I re-imported the data file, the dates were not recognized as matching what was already in the database, so the records were appended rather than updated. It turns out that in SQLite a date without time is not the same as a datetime with time = 0. Who comes up with this stuff?

In a way, Working Effectively with Legacy Code does what it says on the tin: gives advice on how to work with legacy code. But while the title is technically correct, it is also misleading enough to be harmful.

After all, I have been working with legacy code without any help, and making progress cleaning it up, too. So why would I need a book? What can it possibly do to help, other than commiserate and advise me to refactor? For a long time I dismissed the book as “probably won’t help me much in practice”.

Well, it turns out that the book is not really about how to work with legacy code. Michael Feathers has his own, very specific, definition of legacy code: for him it means code without tests. And he also has a very specific definition of working with legacy code: for him it means adding no more untested code. Whenever a change or a new feature is needed, the affected parts of the code base need to be made testable first.

Redefining the problem from “working with legacy code” to “making untestable code testable” suddenly makes the topic appear very useful indeed, and the book turned out to be excellent. Like numerous other reviewers, I wish I had read it a year ago!

It’s full of practical advice, with lots of great examples, rather than high-level principles. Very pragmatic, where other software design veer off into idealism: “you often need to make your design worse before you can make it better” was a recurring theme here that I really sympathise with. It was a very inspiring rbook, making my fingers itch to refactor. On top of that it’s interesting, well-written, and very obviously written by an experienced, knowledgeable expert.

The book is structured sort of like a cookbook, question-answer style: “I need to make a change. What methods should I test?” I cannot imagine using the book to look up answers to such specific, disjointed questions. No, the book needs to be read and digested, and then re-read and re-digested, repeating until you’ve fully internalized all its ideas. I’ve read it once from cover to cover (skipping only the C/C++ specific advice) and I know I will be browsing it for inspiration many times. The numerous cross-references make it a very browsable book.

However. I like the general idea of making all legacy code testable, but in practice it’s not always practicable. GUI code is the most obvious example: it’s hard to test in the best of cases, and in a legacy code base, making it testable is a hopeless battle. I’m not going to refactor an entire 1500-line aspx page so that I can test my one-line change which makes a certain element invisible in certain circumstances. And more generally, making legacy code “non-legacy” is a worthwhile goal, but sometimes the change really is small and safe enough, and entangled enough, that making it testable is just not worth it. But I guess the book needs to be more categorical than reality: aim for the stars, get to the treetops.

Amazon US, Amazon UK.

Started learning RoR with the help of Rails for .NET Developers. Spent well over an hour just getting through the first two pages of creating my first app. After typing in the 3rd instruction as per the book, I got errors.

C:\dev> rails book_tracker
C:\dev> CD book_tracker
C:\dev> ruby script/generate scaffold book title:string author:string on_loan:boolean
  wrong constant name Title:stringController

Fine, I’ll google. Via Google the Ruby community tells me that I have an old version of Rails and need to upgrade – while Mac OS X comes with Ruby and Rails pre-installed, it’s not exactly the latest version.

I update Ruby and I update Rails and still I get the error. I uninstall and reinstall rails, confirm that I have the right version, reboot the computer, and still I get the error.

Back to Google. Perhaps I somehow still have the old version somewhere. I google for “multiple versions of rails” and find a brief blog post about Using Multiple Versions Of Rails that gives me this useful piece of information:

your individual rails projects store the version number they expect

Aha! I performed steps 1 and 2 of the tutorial before upgrading Rails, so step 3 was still hitting the old version rather than the new version. I delete the entire folder, redo steps 1 2 and 3 and bingo, it works.

Over an hour to perform a 5-minute task. Couldn’t rails -v report all versions of rails that I have, rather than just the topmost one in the list?

Another Open Spaces event today. This time it was officially not organized by the ALT.NET community because some of the people behind the ALT.NET movement have had a bit of a disagreement (“the movement is stagnating”, “don’t be so elitist”) about what ALT.NET is and should be. And since Scott Bellware, who’s on the “ALT.NET has degraded and we need a fresh start” side of the battle, joined our discussions, we were officially using his preferred name, which was “progressive software development”. In the discussions we kept slipping back to using ALT.NET since most of us, I think, never felt that we are part of any movement, and knew very little about the controversy. We just want to meet developers who think like we do and work with the same technologies, and really don’t care what they call it.

Today’s event was a bit different from the previous one. The presence of Scott Bellware (who’s not exactly a superstar in the developer world but still rather well-known in the .NET circles) changed the character and the focus of our discussions. Scott’s got a forceful personality (as revolutionaries and firebrands tend to) so some of the sessions veered from discussion into lecture. But they were good lectures, inspiring and interesting. A lot of talk about the meaning of quality, and what is needed to change the world of software development to focus more on quality and learning. I came home with a whole reading list about lean thinking and lean development, and a renewed desire to improve myself and my work. I’ve sort of relaxed for the last half-year and stagnated, not learned as much as I could. Time to pick up the pace again.

As a first step, I will make a new effort to learn Ruby. Why Ruby? Because the other thing I came home with was a book titled Rails for .NET Developers. Scott told us about a project he’s planning, which involves developing a Rails-based web site, and invited us to contribute. I liked his idea, but had to admit I would be of little use as a developer since I knew no Ruby. That’s no excuse! he said and offered me the book. What the heck, said I, took the book and decided to have a go. So not only have I met a celeb developer up close, I now also own a book that used to belong to a celeb developer, ha ha.

The code base I work with is in a pretty bad state. Much better than it was a year ago, but that’s still faint praise. At times I can avoid thinking about it, especially when we’re focusing on some new piece of functionality that is mostly independent of the legacy code. Other times I have to wade through all this old muck to get something done. That really makes me want to clean up, go in with a big broom and sweep all the dirt away. Whenever we’re ahead of schedule I take a day or so to focus on refactoring.

A day is a very short time to spend cleaning a big code base, so I want to make sure I spend it well. But the code is so messy that it’s hard to decide where to begin. It’s like when you have a house that hasn’t been cleaned for years: whichever part you start cleaning, something else grabs your attention because from another angle that looks even worse than the part you’re working on.

Do I start from the parts I need to touch most often?
From the most important parts?
From the dirtiest parts?
From the parts where I can get the most done in the shortest time?
From the riskiest, most complex parts?

Every time I face this decision I make a different choice, but in general I tend to pick the first of these strategies. The files I need to work with regularly get cleaner, but there are many places that I have never even looked at, and I’m sort of hoping that I never will, although I’m sure the day will come.

Our head of development (let’s call him T), who was also 33% of the development team, left us in March, which makes me the new head of development. It’s not something I was aiming for at all (I’ve always been more drawn towards the technical specialist career track rather than management) but since there was no one else available, I’m now the manager of our two-person team.

In a way it’s nice to be manager, as it does give me greater say in how we develop (and to some extent what we develop). There will be even greater focus on design and architecture, on code quality and automated testing, and on paying down our technical debt.

The handover was very undramatic. I just slipped into the vacated shoes. We had three weeks of handover time, during which I was getting used to thinking and acting as head of development, and T worked part-time as an ordinary developer. Very smooth.

Last week was our first week without T, and it was as if the place was jinxed. All kinds of things kept going wrong, and today was no better.

Monday started with login problems at our intranet site. I inherited the responsibility of supporting the intranet from T. The site is based on Drupal + phpBB, and we use an LDAP integration module for user authentication. The LDAP module needs the credentials of one user in order to log in and get the details for all other users. We’d used T’s credentials in the past but now that he wasn’t an employee any longer, his login didn’t work. Easy-peasy, we’ll switch his credentials for mine and off we go. Except that when we did that, everything stopped working. We spent several hours on it on Monday and got nowhere. It didn’t help that we had only basic knowledge of PHP, knew nothing about Drupal development, and even less about LDAP. By noon we’d gotten rid of one detailed error message, only to see it replaced with the not-so-useful “invalid credentials”. It was release week, which meant that the intranet was not high priority. But of course it couldn’t stay down for too long, so on Thursday I spent another half-day digging. It turned out that another Drupal module was interfering with the LDAP module, by transparently inserting <p> tags around the settings we had entered. What a stupid error to lose time for!

On Friday, a few minutes past midnight, our production site went down for 15 minutes, for no known reason. The web host has so far ignored our requests for log files.

Our monthly release was due on Friday afternoon. Ingrid woke up with a fever and I had to stay at home, and Eric couldn’t get home before 5. T came in for part of the release process, and I had to phone in when Eric got home, to support my junior colleague during the remaining testing tasks. Luckily everything went well, even though it was a semi-complicated release. Less fortunately our testing showed that the release included several avoidable bugs, but nothing too severe.

Saturday morning I got a call telling me that it was impossible to log in to our site, even though the site was available, and had worked perfectly well on Friday. Luckily it was easy to diagnose with the help of Google, and almost as easy to fix. It turned out that we’d missed a quirk about web.config files, which for some still uknown reason hadn’t caused any problems in our test environment. Every time the ASP.NET worker process was restarted (i.e. every morning, because of the overnight idle period) the application effectively got corrupted. My first task on both Sunday and Monday morning, before any user had time to log on, was to restart the application in just the right way to avoid that corruption, and when I got in the office today we put in a proper fix.

Then, yesterday evening, our office network collapsed because of the server (or firewall, or some other piece of hardware in that stack) overheating. Turns out there is no AC in the server room, and the server cabinet is of the wrong type, and someone (probably the cleaners) had closed the door during the weekend. Network access was restored quickly, but we still spent half the day without access to incoming email. This was one problem I didn’t have to fix, or even feel any responsibility for, but it meant we were flying blind as far as production support goes – much of our monitoring is, unfortunately, based on email notifications. This was doubly unfortunate since today was the first weekday after release, and it’s not uncommon for a few bugs to arise. Luckily we had no urgent issues today.

It’s all been a flood of unfortunate events, which were all resolved in the end before causing any major issues. But it was a precarious balance, and now I feel all exhausted after a week of firefighting. Stress doesn’t arise from having lots of work, but from feeling of having no control over the situation.

Our dev/build server at work ground to a halt yesterday. When I logged on to look at it, the cause was immediately obvious: the second hard drive, the one we use for our builds, was 100% full. And I mean 100.0000% full, with exactly 0 bytes free space. That was a bit of a surprise since I had a distinct memory of seeing 40 or 50GB free space there some months ago, and none of us could remember installing anything particularly voracious on that machine.

Further investigation showed that the main culprit was our NCover output folders, which hold coverage reports for our two regular builds. One of the folders had about 425,000 files, ranging from 12KB to 150KB in size. The second folder was so full that I didn’t even try to list all its contents, it would have taken forever. Just deleting all these files took most of the day (and that was from the command line – when I made the mistake of browsing to the smaller folder in Windows Explorer, Explorer keeled over). Emptying both folders freed the expected ~50GB and quickly got the computer back on its feet.

Where did all this stuff come from? It turned out that we had misunderstood how NCoverExplorer generated its files. We thought it would reuse its file names, so the coverage report for a given class would always be found in the same file, and each new report would overwrite the old one, if saved in the same folder. Instead it generates random file names every time. We’d inadvertently been saving all our detailed coverage reports, with our entire source code nicely rendered in html markup, and generating several new copies every day. No wonder we were running out of space.

After three months of Mac use, I’ve more or less settled in.

I stuck to Opera for email, web browsing and RSS. I know that most of Opera’s goodies (such as mouse gestures and Speed Dial) are available in Firefox as extensions, and I’m a happy Firefox user at work, but I feel at home in Opera. I like the convenience of having web, email and RSS all in the same application, and I like the feeling of supporting the underdog.

The bundled text editor in OS X is almost as primitive as Notepad. I’ve been evaluating TextWrangler for a while, but I’m not completely happy with it. It’s got some small shortcomings that, taken together, really annoy me. Why did they have to come up with their own shortcut for switching between documents? Why does the Find & Replace dialog never remember my settings? And why is there no keyboard shortcut for Replace & Find Again, when all other search commands have one? If you have a better text editor to recommend, let me know. All I really need is a smooth multi-document interface, syntax highlighting, and a user-friendly search with regexp support.

For organizing photos, I’m going with Picasa. It is fast, easy and convenient to work with, has good tagging support, doesn’t copy my photos into a black hole like iPhoto.

I have a bunch of Office documents that I need to keep working with. For these, I tried NeoOffice. But my initial conclusions (clunky, ugly, slow, and no support for mdb files) remain valid, and I just couldn’t stand working with something so clunky.

So I downloaded Parallels Desktop, and went back to Windows and Office for my spreadsheet and database needs. It felt GOOD. Parallels is a joy to work with: fast, easy to install, convenient to run, and I have full access to all the OS X documents from Windows. I almost forget it’s there: it just feels like I’m using two operating systems in parallel, seamlessly.

And I don’t miss Windows at all.