Microsoft’s advice regarding the latest Windows vulnerability is so far removed from reality that it’s laughable:

Users should take care not to visit unfamiliar or un-trusted Web sites that could potentially host the malicious code.

Have they never seen a real user surf the web? The whole point of the web is that you can click on links which take you to previously unvisited, potentially interesting sites!

Actually the vulnerability itself is interesting too – opening an image is just about the most innocuous action I can think of. If something that basic can open my computer to malicious code then there’s really no code in Windows that I can trust to be secure.

News of the week: Sony’s DRM software uses rootkit techniques

Recap
DRM-protected CDs from Sony surreptitiously install software on users’ computers. The software is hidden using rootkit techniques normally associated with malware, isn’t mentioned by the EULA, and cannot be uninstalled safely by the average user. The software is also badly written, wasting resources, opening up the computer for other malware, and potentially corrupting the system if it’s removed by an uninformed user.

And as if that wasn’t enough to make a good story, Sony first resists public requests for an uninstaller, then makes available an uninstaller that is written as badly as the original program (if not worse), and then makes the users jump through various hoops to get it, including downloading and installing an ActiveX component of uncertain provenance…

It took a few days for the story to spread through all mainstream media (it’s a bit technical after all). And of course it took only a short moment before someone wrote a Trojan exploiting this.

Sony, blind and oblivious to all the uproar, says, “Most people, I think, don’t even know what a rootkit is, so why should they care about it?” This is nothing, not malicious at all, and you shouldn’t bother your pretty little heads with all this grownup stuff.

And naturally Sony is now getting sued and boycotted by all and sundry.

The story is fabulous, better than fiction. I’ve really enjoyed following it. (It helps, of course, that the original articles at SysInternals blog make for a good read – thoroughly detailed, understandable, well written and well illustrated.)The whole situation is such a spectacular combination of evil, stupidity and incompetence. It is so absurd that it ceases to be annoying and becomes entertaining.

First, the strategically incorrect decision to build something like this must have been based on a grossly mistaken understanding of how their customers would react if they ever found out. Second, they misjudged the skills of their coders vs. the skills of spyware hunters. Third, they lied in the EULA. Fourth, they completely misread and mishandled the public reaction by first denying the problem, and then trying to avoid responsibility.

So, the business development people are incompetent, as are the people in charge of software project outsourcing, and their lawyers, and their PR department. Hmm… that’s about the whole firm… excepting HR and finance.

It’s difficult to imagine what more Sony could have done to make this worse. Perhaps they could have tracked users who use the software? They could have, actually, since the software phones home as well… Oh, the one thing they haven’t tried yet is to enforce the EULA and countersue users who remove the malware.

What does one learn from this?
That when it comes to software, you can’t trust anybody, and you can’t be too careful. A large well-known company is not necessarily more trustworthy than a small unknown one.

This week has seen a lot of SAS work, so that’s what my head is full of right now.

• After several weeks of intensive SAS usage, I’m finally reaching what would be called “conversational” for a human language. (This came after a lull of over a month where I was mostly working in Excel and VBA). I can express any basic concept, and no longer need to search for words every time I open my mouth – “speaking SAS” has gone from frustrating to enjoyable. Now I can start working on getting rid of my accent, and increasing my vocabulary so that I can express myself better. In programming terms, rather than just get things done, I can look for cleaner, faster, more efficient solutions.

• I have scrolled through and inspected a dataset of ca 121,000 rows of data. Manually. Twice. Good thing there’s a Find command in the dataset viewer (no quick filter, though…)

• I’ve found myself entering a paragraph break after each sentence, and finishing each paragraph with a semicolon in an e-mail.

• I’ve started using SQL more seriously. It’s SAS’s own version of SQL, but that appears to be (a subset of) ANSI SQL with some SAS-specific additions – so hopefully most of what I learn by doing this will be useful elsewhere as well. I’ve advanced from simple “SELECT * FROM table WHERE” to queries that include joins, summary functions, groups etc. It is very powerful (which is no surprise) and some of it is a lot faster than the “pure SAS code” equivalent – and a lot more elegant.

Hmm… perhaps it’s time I got myself a SAS book. This feels like a great reason to go and do some bookshopping this weekend!

My job description doesn’t say (that is, if I actually had a job description then it wouldn’t say) a word about software development. It would probably say a fair bit about modelling, data management and data analysis, though. And once those pass a certain level in terms of complexity, you really end up in the realm of programming anyway. So while Excel is the main tool for me and my colleagues, we spend a fair amount of our time developing small applications in VBA, using VBA to augment our Excel models, and now also coding in SAS.

But we’re not programmers. And that shows.

In my case, programming has been my main hobby for the past 2 years or so, so I consider myself fairly proficient. I don’t have the level of experience that a professional developer does, but I have come to think like a developer. The other two “modellers” in our group don’t. They are very intelligent people, but when it comes to programming, they just don’t think the right way. In fact I think their general intelligence is making the problem worse, because it makes them too confident: they assume that they know what they’re doing.

Case in point:
This past Friday, I found a small but very important bug in code that colleague A had written, in a project that we are working on together. A number had been hard-coded as a constant, and in order to get the right results, that number needs to be changed manually once a month. Turns out that this number had been off by one for the last 3 months. And the output from this model is used to make million-dollar decisions.

(To make things more interesting, colleague A happened to be travelling on Friday, so I couldn’t take this up with him, but had take this directly to our manager, because he was about to use the output. The rest of the day was spent in a state of moderate panic, estimating the effects of the bug, and trying to minimise the effects.)

The bug was in no way subtle or hard to find. The report output wasn’t behaving the way I expected, and once I’d noticed that, it took less than 10 minutes to find the error.

Hard-coding a number that needs to change is dangerous and stupid in itself. But I think it’s only a symptom of a larger problem.
This really comes down to looking at code with the right mindset – and a big part of “right” is “defensive”.

1) Don’t assume that your code is correct.

The starting point should be that your code may be broken. Even if it runs, assume that it is not doing the right thing.

Every time I finish a piece of code, I ask myself – where could this go wrong? How might this output, which looks OK at first glance, be wrong? How can I check it? In the case of financial models, I usually track at least one data point in each result set all the way back to the input data, manually, using pen and paper and a calculator.

Colleague A would have found the problem if he had looked, but he didn’t – because he assumed that if it doesn’t look obviously wrong, it must be right. (It’s not the first time I find such problems in his work, either.)

It helps that I know what can go wrong. I know to look for edge cases, off-by-one errors, misaligned data, mismatched units / currencies. But these are less obvious to those who have little programming experience.

2) Failures should be visible.

Checks and double-checks and triple-checks. In environments that support it, tests and checks should be automatic. If there are two routes that should get you to the same point, take both and show both results. If there are conditions that the results should satisfy, those should be checked automatically, and very loud bells should start ringing if they don’t match.

Produce far more output than needed. Display your assumptions, and save intermediate results. Show inputs side by side with the output.

In this case, showing the key inputs would have made the problem very visible and we’d have discovered it immediately.

All of this doesn’t mean that my code is bug-free. It will never be. But it will have fewer bugs than code written by someone who doesn’t approach things this way.

Things will improve as my colleagues gain experience – but in the meantime, mistakes like this might cost us a lot of money. So I think I need to start taking more responsibility for what they do in their code.

The question is, how? How do you teach someone to think the right way – especially if they don’t even think of their job as programming?

“Leading by example” will help somewhat, but so much of this is quiet and invisible… It’s hard to make my assumptions visible. Thinking out loud while discussing our code (his and mine) might be one way. Politely adding checks and balances to the parts of his code that I review is another. But it doesn’t seem quite sufficient.

And that’s assuming that this is even “learnable” – which it might not be, it might just be a more general mindset that you either have and value, or don’t…

Finally!

I’ve made four (4) attempts to order the Visual Studio 2005 Beta DVDs, starting in January I think. Three times I got nothing. This fourth time I had more or less given up hope, and wasn’t expecting much, but since it didn’t cost me anything I clicked again. And it worked!

I wonder if MS just has really appallingly bad shipping systems, or if some Royal Mail employees steal everything that is labelled Microsoft. Both seem quite possible. Royal Mail hasn’t exactly excelled in the past. (Once they left me a note that I had a package to collect, and when I turned up at the post office with the notice slip in hand, they couldn’t locate the package…)

I’ve had the VB Express beta installed for a while now, and enjoyed playing with it, but haven’t touched C# or ILDASM or any of the other tools. Lots of fun ahead now!

I am determined not to let this blog degenerate into another link collection, which is what many blogs out there have become nowadays. One blogger (or non-blog site) says something interesting, and then everybody else links to that, and posts a sentence or two, but doesn’t add anything new. Quite an incestuous club.

However, there are some sites out there that I just find too interesting to be ignored.

The blog about the upcoming new version of Excel is one of them. It’s a thorough overview of new features in Excel 12, with lots of detail and lots of screenshots. The blog is also quite well written, and the guy himself is very responsive. The blog is a pleasure to read and participate in.

For the first time ever, I find myself looking forward to a new version of Office, and feeling excited about what is coming. So many annoyances are getting fixed, and so many nice new features are coming!

Excel is my #1 tool at work – I use it for maybe 75% of my projects & tasks, so any shortcomings have ample opportunity to make themselves felt. Indeed, with many of them I’ve progressed through all 5 stages of frustration – ignorance, annoyance, denial, damning them to the nethermost hells, and acceptance. Finally I get so used to them that they become fixtures in my life, and I no longer question them. In fact I’ve almost grown fond of them and the workarounds and tweaks that I’ve learned to use in order to get around them, and I’ve long since given up expecting them to be fixed.

One doesn’t expect Microsoft to change any of the things that matter, after all – upgrades mostly just make minor tweaks to existing features, and change the looks of all toolbars. (This version is no different, of course – the toolbars get a new design again.) There hasn’t been anything really new for about 10 years, since Excel 97.

It is interesting to see that this time, Microsoft is actually fixing many of these annoyances – formula length limits, IF() nesting limits, array formula limits, the awful interface for conditional formatting. As one of the commenters at that blog jokingly said, Microsoft “are putting us developers out of business… In the old days, those of us who knew how to do Conditional Formatting wielded great power over less competent users.”

At the same time, they’re adding a lot of new features that really look very useful. And I’m sure we’ll discover that all sorts of new unexpected things will become possible as well. It’s an exciting time to be an Excel developer.

The blog is also interesting in and of itself. Well before the product is released, a Microsoft employee is sharing the firm’s plans and designs with users, engaging in a discussion with them, and soliciting their comments and feedback. A blog is an excellent medium for such a discussion. I’ve never submitted comments on any of Microsoft’s web sites or surveys – there’s always the feeling that they’ll just disappear into a black hole, get chewed up by the Microsoft Machine and never emerge again. Here, there’s a person on the other side who actually responds to my comments. A novel feeling!