Heaps and heapsort

Like most programmers, quite some time ago I'd read about heaps and the heapsort. Being a dutiful student, I could quote the standard line, saying that a heap was an implicit tree where node N was less than (or greater than, depending on the type of heap) nodes 2N and 2N+1. Like (I think) most, I'd also implemented a heapsort -- but to be honest, most of really came down to a line by line translation from the description in Knuth Volume 3 into the syntax of the programming language I was using at the time (Pascal, the first time around, and later the same again in C).

For years, given that I'd written a Heap sort that worked, I considered my knowledge of heaps and the heap sort entirely adequate. To be entirely honest, however, during most of that time I never really fully understood heaps. In particular, I never really understood the 2N and 2N+1 -- why those particular numbers/nodes? What was the significance of 2N and 2N+1? Why not 3N or 1.5N+65 or something? In short, that "implicit tree" had never really gelled for me -- I'd read the words, but never truly understood what they meant or how the "tree" worked.

For some reason a few months ago my lack of understanding on that particular subject bubbled to the top of the heap (I know: puns are the death of real humor) and I decided to implement a heap (and heap sort) entirely on my own, based only on my recollection about 2N and 2N+1. Instead of going to the book and translating syntax, I sat down with a supply of paper and pencils (the most underrated programming tools ever) and re-invented the heap and the algorithms to create and maintain a heap from first principles.

So, what did I do to finally and truly understand a heap? It ultimately came down to one piece of paper:

And from there came true understanding at last. If we arranged the nodes by layers, it "just happened" that the children of every node N were nodes 2N and 2N+1. Better still, after three layers of the tree, it was pretty apparent that as/if I added more to the tree, the same relationships would hold. Only 30 years (or so) after I'd first written a heap sort, I finally understood what I was doing!

I finally also understood why people who understood heaps thought they were cool: since the "links" in the tree are all based on (relatively simple) math, it's a tree you can walk almost any direction you'd care to think about by simply doing the right math. Just for example, in a normal binary tree, the only way to get from a node to its parent is to traverse down the tree, and remember the parent on the way there so when you need it back, you can recall it from wherever you have it stored. Likewise, getting to the sibling of a node requires that you get back to the parent (see above) and then take the other branch downward to get to the sibling.

With a heap, however, both of these are simply trivial math: the children of node N are nodes 2N and 2N+1, so the parent of node M is node M/2 (using truncating/integer math). Likewise, the sibling of node N is either node N+1 or N-1 -- if N is even, the sibling is N+1, and if N is odd, the sibling is N-1.

So, for any given node, we can walk through the tree up, down, or sideways (literally) with a single, simple math operation. Better still, all the nodes are contiguous in memory, with no space used up on links, so it's also pretty cache-friendly in general.

There was one other point I ran into repeatedly as I worked at understanding what I was doing well enough to produce working code: the math all depends on 1-based indexing. Making it work right with 0-based indexing like most currently languages use was not trivial. In fact, I finally decided that for this purpose it was better to maintain the illusion of 1-based addressing for most of the code, and since I was writing this in C++, have a truly trivial class to handle the translation from 1-base to 0-based addressing (i.e., subtract one from every index I used). As simple of an operation as that seems like it should be, keeping it in one place still simplified the rest of the code tremendously (and made it much easier to see the relationship between the code and that fundamental 2N/2N+1 relationship that defines the heap).

Anyway, in case anybody cares to look at the resulting code, I've decided that trying to post significant amounts of code directly here is too painful to be worthwhile, so here is a link to it on ideone.com. I think this is long enough for now, but sometime soon I'll probably write a post going through that code in more detail, explaining how it works and why I did things the way I did.

Adventures in monitors

Friday I broke with my usual habit of staying home on the big shopping days, and went to Best Buy. I had some store credit, and was (and should still be) working on a project where I'd get a pretty major benefit from having another monitor attached to my computer. Besides, while my current monitor was quite high-end when it was new (a LaCie 321), it was getting quite old -- something like 7 or 8 years, if I recall correctly. Over the last few years, nearly anytime somebody heard about my ancient monitor, they'd tell me about how much better monitors had gotten since then, so even a really cheap, entry-level monitor would be a huge improvement.

In terms of specs, almost an new monitor should be a pretty serious upgrade. Just for one obvious example, the LaCie is only rated at 500:1 contrast ratio. Definitely pretty unimpressive compared to the ratios claimed by nearly all new monitors (anymore, the bare minimum seems to be at least a few million to one, and five or even ten million to one isn't particularly hard to find). Granted, those are "dynamic" contrast ratios, but the static ratios are still better than my ancient LaCie

After looking around a bit, I found an HP W2071d, which is a 20" wide-screen monitor, that was marked down to only $79.95 (from a list price of $149.95, for whatever that's worth). Seemed like a decent enough deal, so I picked one up, brought it home, and connected it up. As usual for me, just about as soon as it was connected it up, I pulled out my trusty Eye-one display 2 and calibrated it so the brightness, contrast, color (etc.) should match up with the LaCie (in case anybody cares: 5200K whitepoint, 110 cd/m² luminance, gamma of 2.2).

Anyway, let's get to the meat of things: just how big of an improvement did the new monitor bring? Well, let me show you a couple of pictures. The HP is on the left, the LaCie on the right:

Here, the two don't look drastically different -- the background in both is reasonably black (though in person, the background of the LaCie is definitely blacker). The picture on the HP is a bit dark, but not too terrible. But now look what happens when I quite crouching down quite as far, and aim the camera down at the screen at something like a 30 degree angle:

Now, especially toward the bottom right corner, the HP's background isn't even close to black. Despite a higher claimed contrast ratio (600:1 static, 5000000:1 dynamic) the contrast between the picture and the background on the HP is looking pretty minimal. Worse, while the picture looks pretty much the same from either angle on the LaCie, on the HP it's gone from a bit dark and shadowy to completely washed out -- to the point that the stripes on the sweater have almost completely disappeared!

My conclusion: both the specifications and people claiming current entry-level monitors have surpassed high-end monitors from yesteryear are basically full of crap¹. The new monitor undoubtedly works for spreadsheets or word processing, but the LaCie retains a significant edge in contrast, color accuracy and ability to view from different angles.

¹ No, I'm not accusing HP of lying in the specifications -- I'm sure under the right circumstances, it meets its specification. Nonetheless, even after years of regular use, the blacks on the LaCie look deep and rich, while those of the brand new HP look weak and wimpy at best.

Software patents, part II.

In my last post, I promised a post about what I think needs changing in the patent system. This necessarily gets considerably more detailed, so it's likely to be harder to apply it to different patent systems. Nonetheless, when people talk about problems with software patents, the US seems to be (by far) the most often cited culprit, so I doubt that being specific to it is really a major problem. Anyway, on with the show.

The first problem I see is that the US patent office is basically run as a business -- it's profitable to the government, and most of the profit comes from issued patents. Worse, what's really intended to be the primary job of the patent office (searching through prior art to figure out whether something is really original) is apparently run at a loss. The profit comes from "maintenance fees" on patents after they're issued.

The patent office has suggested a way to try to streamline the patent examination process that would involve the inventor either citing only a few sources of related art, or (if they know of too many) specifically pointing to those they think are most relevant, including specific citations of the parts they consider relevant.

At least to me, this seems utterly insane. First of all, it's simply attempting to shift the burden of the patent office's job from the patent office to the inventor. Second, it's asking the inventor to predict what the patent examiner will find the most useful. If, for example, the patent examiner looks at some other reference and finds it more relevant than the ones the applicant pointed to as most relevant, would the patent become invalid? If so, the patent system basically becomes a lottery. But, if there's no penalty, what's to prevent an applicant from citing something utterly irrelevant as the most relevant art related to his patent?

The simple fact is fairly simple: as it stands right now, the patent office simply isn't complying with the requirements of patent law. The law says the fees they charge should cover the cost of examining the patent. If their fees aren't high enough to do that, then the fees need to be raised. Another point that's routinely cited is that an examiner has only about 20 hours to examine a given application (there's some disagreement about the exact number, but at least it's at what I'd consider the noise level -- there's little question that it's between  15 and 25 in any case). Another common complaint is that patent examiners simply aren't competent -- that they often just don't know enough about the technology to sort out (for one example) truly new techniques from simply new and different terminology for existing techniques.

These have a simple solution though: simply increase the fees for a patent application to the point that they can/will cover the costs of hiring competent examiners, and giving them enough time to do a good job. Along with funding the process properly, higher fees, all by themselves, would probably do quite a bit to encourage companies to look more carefully at their applications before filing, to ensure that what they're filing is truly novel and useful, before wasting others' time finding prior art to show that it's invalid.

The second major problem I see is that it's much easier to get a patent granted than to get that same patent invalidated after it's been granted. The patent office has been taking some steps that attempt to balance this a bit better, but at least personally, I don't think they've done quite enough yet.

Classically, the situation was that the inventor filed his application, and the patent examiner (usually only one, though sometimes two) tried to find prior art. Under most circumstances, almost nobody else participated in the process at all. The patent office does now publish (most) patent applications, so normal people can review them, but it's still basically provided as a one-way communication -- if you look at an application on the patent office's web site, there's not (for example) a button on the page to let you send an email to that examiner citing what you think is prior art on that patent application. Right now, however, instead of treating prior art submissions as what they really are (normal people doing them a favor to make their job easier) they seem to treat it as if they're doing me a big favor by allowing me to submit anything at all, and it's perfectly reasonable to make me jump through flaming hoops before I can be granted such a privilege.

Even so, the patent office follows a "preponderance of the evidence" rule in deciding whether to issue the patent. That basically means only a simple majority of the evidence needs to point toward its being novel for the application to be accepted, and the patent to issue. If 51% of the evidence says it's original, and 49% says it's not, the patent can be issued. Keep in mind that in most cases, it's up to one person to find any and all evidence of invalidity (in roughly 20 hours of work or less), but a large company may easily have a dozen people working hundreds of hours (if necessary) to get the patent issued, and it's quickly apparent that the system is heavily loaded in favor of issuing the patent unless it's quite obviously bad (and sometimes even if it is quite obviously bad).

Once the patent issues, however, the courts are required to give the patent a presumption of validity. To invalidate the patent, you don't simply need to provide a preponderance of evidence in the other direction. It's not enough at that point to show that 51% of the evidence points toward invalidity, and only 49% toward validity -- instead, the courts require "clear and convincing" evidence that the patent is invalid before they will declare it invalid. I've never seen a court attempt to specify a percentage of the evidence necessary to meet the clear and convincing hurdle, but it's clearly a lot higher than a mere preponderance of the evidence. If I had to characterize it, I'd probably put the number at around 90% -- still somewhat short of the "beyond a reasonable doubt" (that's supposed to be) used in criminal courts, but much higher than than 51% used as the basis for issuing the patent in the first place.

We end up with a system that deprives most people of much ability to provide evidence that a patent application isn't valid, issuing the patent even when/if there's a substantial chance that it's not valid, and maintaining it as valid even when most of the evidence indicates that it's really not.

Though I don't think those are (probably) the only two problems with the patent system as it stands today, I think addressing those two points would go a long ways toward redressing much of the current imbalance in the patent system. Interestingly, neither of these requires any change to the actual patent laws at all, only changes in procedures about how the existing law is enforced. Over the past several years I've seen many proposals that included what initially seem like much larger changes (nearly all including substantial changes to the patent law itself). I believe most of these would do little to cure existing problems, and quite a few would add many more problems of their own as well.

From a programmer's point of view, most of the proposals strike me as doing nothing to try to find or fix actual bugs, and instead advocate changing indentation and naming conventions throughout the code. They'll change how the code looks, and might (conceivably) make it more readable, to at least some degree, but they're almost certain to introduce at least a few new bugs in the editing, but fail (except, perhaps incidentally) to fix existing bugs. Just as with code, to do real good, you need to identify the real problems, prioritize them, and (usually) attempt to make the smallest changes that fix those problems.

I'd also note that almost none of this is specific to software patents at all. Most of the problems I see apply at least equally to areas outside of software. As noted in my previous post on the subject, I think most of the arguments attempting to separate software patents from other patents lack merit.

Why most programmers are wrong about software patents.

Many (probably most) computer programmers favor elimination of patents that apply to software. I believe, as is often the case, that they're well on their way to repeating a dark history they've blithely ignored.

Patents were invented as a service to society -- the basic intent was (and remains) fairly simple: give an inventor exclusive rights to his/her invention for some limited period of time in exchange for agreeing that as soon as that limited period of time expires, the invention goes into the public domain -- anybody can freely use it. By itself, that wouldn't mean much: after all, if the patent didn't exist to start with, anybody would be able to make free use of the invention immediately, instead of waiting for the patent to expire.

The benefit to society comes from another requirement for a patent: that the patent describe the invention so others can use it. The exact phrasing varies by country, but in the US the requirement is that the patent contain a description of the invention sufficient for a person of ordinary skill in the art to implement the patented invention.

Going a step beyond that, US patent law also includes a "best mode" clause that says the inventor must reveal what he believes to be the best method of implementing/deploying/using the invention. For example, if he's thought up a new type of oil for lubricating automobile engines, he can't patent it, but in the patent claim that it's intended to be used as cooking oil (which could deprive society of the benefit, because they ignore it after finding that it makes food taste terrible). Obviously the example of a cooking vs. lubricating oil is a bit extreme, and probably easily seen through -- most people would probably think something was a bit odd about Exxon patenting what they claimed was a cooking oil, for example.

Anyway, let's take at least a quick glance at that dark history they're ignoring. Before patents were invented, most crafts were governed by guilds. To learn how to carry out a particular craft, you had to be admitted into the proper guild. The guild guarded the secrets of its craft quite carefully in most cases -- to the point of sometimes murdering people who were suspected of revealing guild secrets, and refusing admittance to others because they were seen as too dangerous. In other cases, there were literally wars between competing guilds. These not only get people killed, but in some cases when a guild was destroyed, its secrets died with it. Archaeologists have found artifacts showing techniques we don't quite know how to duplicate to this day.

As many problems as they had, guilds were a substantial improvement over the previous system: one of individual craftsmen keeping secrets entirely to themselves, to be passed onto their children, sometimes on their deathbed -- or taking their secrets to the grave, if their children weren't handy at the right time, or were estranged, etc.

Most arguments programmers level against this fall into three broad groups. The first is that programming techniques (algorithms, protocols, etc.) get published even when there are no patents. The second is rarely quite so clearly articulated, but basically comes down to the notion that if one person invented something, they can invent something at least equally good when and if they need to. The third is that software moves so quickly that by the time a patent has expired, it's no longer of any value.

The first is partly true. They can point, for example, to the fact that European companies continue to publish despite the European policy against software patents. This, however, ignores reality. What's happening right now is that Europe is basically "freeloading" off of the patent systems of other countries (US, Japan, etc.) If, for example, Seimens or Philips wants a patent in the US and/or Japan, they need to publish all the details of that invention in their US/Japanese patent application. That means the invention is no longer secret -- and given current communications, the minor detail that it's published in the US/Japan instead of somewhere in Europe adds (at most) a few milliseconds for a European citizen to download the patent application. As such, Europeans enjoy the benefit of inventions being published, without (themselves) giving the inventor anything in return.

The second strikes me as simple conceit. Yes, there are patents running the gamut from utter stupidity to simply being something nobody else thought was important or original enough to bother patenting. Being at all honest, those constitute only a fairly small percentage of patents overall though. Much of this stems from a problem I've seen many times: somebody will draw a badly mistaken conclusion about what the patent covers, often based on looking at nothing other than the patent's title. They see a title that says something like "A method for run-length compression", and they think something like "good lord what insanity -- everybody's known about RLE for ages -- obviously the patent office is a bunch of idiots!" Though it's barely possible they're (at least sort of) correct, and this really is a patent on something everybody's known about for decades, chances are it's something else entirely -- some (possibly minor) improvement that allows the job to be done a little faster, or using a little less memory, etc. In a typical case, it's not earth-shaking, but is original and at least mildly useful.

The third argument may sometimes have some degree of validity as well, but often doesn't. This argument, however, ignores that fact that if the invention becomes obsolete quickly, the cost of the patent (to the people) is also low -- if an invention is only meaningful for a year or two, the cost to society of the fact that an inventor (theoretically) gets exclusive use of it for years afterwards costs society essentially nothing. At the same time, people have invented and patented algorithms that retain value well after the patent expires. An obvious case in point would be RSA encryption. The US patent on RSA expired well over a decade ago, but it's still almost certainly the most widely used from of PK cryptography. RSA remains (roughly) as valuable today as ever, although there are alternatives that could be used if RSA were not available.

Now, don't get me wrong: I don't think the US patent system (or probably any other) is anywhere close to perfect. Some time soon, I'll write a post about what I see as problems in the system, and at least some ideas about how to fix those problems. To give fair warning: I believe many of the problems can be fixed with much less drastic changes than most people seem to think are necessary. In fact, some of the changes are (or at least initially seem) so minor that they're probably almost invisible except to attorneys who specialize in intellectual property law. Nonetheless, I think some seemingly trivial modifications can have fairly serious consequences.

C++ dynamic arrays

I've seen dozens (at least) of questions and proposed answers about how to create a dynamic array in C++. The typical reply recommends using the array form of the `new` operator to allocate space, something on this order (but usually less complete):


template
class dynamic_array {
    T *data;
    size_t size;
public:
    dynamic_array() :data(0), size(0) {}

    dynamic_array(T *init, size_t i_size) :size(i_size) {
        data = new T[size];
        for (size_t i=0; i<size; i++)
            data[i] = init[i];
   }

    bool add(T const &new_item) {
        T *temp = new T[size+1];
        for (size_t i=0; i<size; i++)
            temp[i] = data[i];
        ++size;
        delete [] data;
        data = temp;
    }
     
    T &operator[](size_t index) {
         assert(index < size);
         return data[index];
    }
    // We also want a const version of operator[].
    // We also probably want something like get()
    // in std::vector, that throws if index is out of bounds.

    size_t get_size() { return size; }

    void clear() { size=0; delete [] data; }

    ~dynamic_array() { delete [] data; }
};

Now, it's certainly true that you could do worse than this. I believe it manages memory correctly (doesn't leak memory or create dangling pointers), and it is dynamic (to some degree -- it allows adding, but not currently removing, elements dynamically). While it's not currently written to be exception safe, that's mostly to keep it from getting too long -- making it exception safe wouldn't be particularly difficult.

It still has some pretty serious problems though. First of all, it requires that T be default constructible and assignable. Assignable isn't so bad (though in C++11, we'd prefer movable), but default constructible is kind of a pain, especially since we never actually *use* any default objects -- but when we expand the allocation, we have no real choice but to default construct the objects, then assign each value from the old array into the new one. That can be a pretty serious burden if creating a default object is slow, and worse still if assigning objects is slow as well. Since it's a template, we don't know much about T. Either or both might be true.

That points to a much bigger problem: adding an item to the array is slower than we'd like, and the bigger it gets, the slower adding items gets. In computer science terms, adding a single item has complexity that's linear on the current size of the array.

Unfortunately, we can't just do a minor rewrite inside of add() to improve the situation. In fact, the basic premise of how this dynamic array works turns out to be wrong. The only way to fix the biggest problem is to start over with something that works a bit differently from the beginning.

Our big problem here is that when/if we use `new whatever[size]`, we not only get enough space to store the specified number of objects, but code is inserted to create that number of objects in the storage. Generally speaking, we don't want that. Unless we change the interface at least a little bit, there's never a circumstance in which we really want objects in that memory. Even if they're created, immediately after allocating those objects, we're going to assign some other object to that position anyway. We'd really prefer to copy construct the new objects from the old objects (or, in C++11, move-construct). This way a new object is created with the correct value instead of being created with a value we know is wrong, then immediately assigning the correct value to the new object.

We'd also prefer to avoid re-allocating space every time we add a new item. We really want to allocate a chunk of raw memory, turn that raw memory into new objects as they're added, and only allocate more space once in a while, when our memory is all used up.

Using new T[size] won't let us do either of those though. Instead, we need to use something (usually operator new, but malloc would also work) that lets us allocate raw memory, then use placement new to create objects in that raw memory.


template
class dynamic_array {
    T *data;
    size_t allocated;
size_t in_use;

size_t round_up(size_t input) {
size_t ret = 1;
while (ret < input)
ret <<= 1;
return ret;
}

void alloc(T *init, size_t current_size, size_t new_size) {
char *temp = (char *)operator new(new_size * sizeof(T));
allocated = new_size;
for (size_t i=0; i<current_size; i++)
new(temp+i*sizeof(T)) T(init[i]);
in_use = current_size;
data = (T *)temp;
}

void destroy(size_t b, size_t e) {
for ( ;b!=e; ++b)
data[b].~T();
}

public:

    dynamic_array() :data(0), allocated(0), in_use(0) {
alloc(0, 0, 10);
}

    dynamic_array(T *init, size_t i_size) {
alloc(init, i_size, round_up(i_size));
}

    void add(T new_item) {
if (in_use == allocated)
alloc(data, allocated, allocated * 3 / 2);
new ((char *)data + in_use++ * sizeof(T)) T(new_item);
    }

    ~dynamic_array() {
destroy(0, in_use);
operator delete(data);
}

    T &operator[](size_t index) {
        assert(index < in_use);
        return data[index];
    }

    size_t size() { return in_use; }

    void clear() {
destroy(0, in_use);
in_use = 0;
}
};

This is starting to run pretty long, so I'll leave further embellishment of this class for a later post. This is still quite a ways short of what std::vector already provides though -- thus the standard advice to just use a standard container if at all possible. Even for something as simple as a dynamic array, the amount of code involved becomes fairly substantial. For a more complex container like std::set or std::map, the savings is greater still.

Herb's wrong, I'm right, and here's why.

Microsoft's MSDN Channel 9 recently released a video of Herb Sutter, Andrei Alexandrescu, and Scott Meyers from C++ and Beyond 2012. In this video Herb (in particular) argues that further support for meta-programming should not be added to C++.

He points out (quite rightly) that meta-programming is basically used to add features to the language. He argues that rather than supporting meta-programming, we should look at what features meta-programming is used to implement, and add those features directly to the language itself.

In my opinion, there is really only one reasonable answer to this: wrong!

This is utterly and absolutely wrong for a number of reasons, most of which (interestingly enough) have been expounded many times before. One of the mantras of C++ development for years has been that the committee does not and cannot know all the purposes to which the language might be put. Rather than adding features to the language specifically for every individual purpose and each end user, there has been a concentration on adding features to support things like writing libraries so that adding a relatively small number of features can support a wide variety of uses.

Trying to look at all uses of meta-programming and find the features being created with it, then trying to add those features to the language would be a 180 degree reversal of this decision. The problems already cited would become all too real and all too problematic essentially immediately.

Consider what would happen if I wanted to write a library using features not currently included in the base language. My choices would be to implement that code using meta-programming, or else petition the committee to please, please consider my proposal for a new language feature. For the sake of argument, let's assume that the C++ committee starts to produce revisions twice as fast as they have in the past -- or than essentially any other ISO standard committee for any other programming language has either -- and produces an update every five years instead of roughly once every ten years.

That means, on average, the programmer has to wait 2.5 years before he can even get started on his library. Given that the standard is typically frozen for a year or so before being approved, it's really a bit longer than that, but let's just leave it at 2.5 for the sake of argument. I hope I don't have to spend any time convincing anybody that a two and a half year delay in getting started would be unacceptable for many projects.

That's probably the least of the problems though. Much worse is the fact that accepting any more than a minuscule number of such proposals would quickly lead to C++ becoming substantially larger and more complex. Virtually everybody agrees that C++ is already larger and more complex than anybody would like, and going this direction would unavoidably lead to amplifying that problem.

I do agree with Herb that the current meta-programming facilities have serious problems. The code itself, the compile times, and especially the error messages can all be truly horrific. More importantly, these aren't just remote possibilities; they happen on a regular and routine basis.

That does not, however, mean that template meta-programming in general is a problem and should be avoided. It means that the current facility has problems, most of which stem from the fact that its intent was never to support meta-programming at all.

These problems should not lead us to conclude that meta-programming is evil in general and should, therefore, simply be avoided entirely. They should convince us that the current meta-programming facilities are inadequate and in need of replacement. What we should be doing now is looking for a simple, small, relatively easy to understand set of features that will support meta-programming more sanely. I won't try to propound here exactly what features are necessary, much less what form those features should take. This is, however a basic engineering problem. It's the kind of thing that's been solved before, and can be solved again -- and while the result undoubtedly won't be perfect, it will be good enough to fix the problems that are now obvious (and probably quite a few nobody's thought of yet).

Early in the history of programming, many thought of it as a priesthood. It was liberated largely by improvements in tools and education. At least in my opinion, the situation here is similar.
I guess if I were a great leader of men, at this point I'd come up with some ringing phrases about "when faced with such a challenge, do we shrink in fear, or do we rise up and vanquish the foes of free computing everywhere?"

Since I'm an engineer, though, I'll simply reiterate that this type of problem has been met and overcome before, and can be again. What's necessary isn't vilifying template meta-programming or its current users, but to make sound engineering decisions to fix a problem in the most reasonable fashion possible to make meta-programming dependable, accessible, and understandable, so it becomes the tool that can fulfill the promise shown by the current uses of template meta-programming.

A rant on operating systems and development.

I guess I'm overdue on playing my role as an Andy Rooney imitator, whining about crap that should be taken out behind the barn and put out of everybody's misery.

This time I'm going to talk about operating systems. I think the operating system situation has gotten completely out of hand. Every one of them seems to be competing at being the worst it can be. I'm still using Windows, but only because the people writing OS/X, Linux, *BSD, etc., seem to be doing everything in their power to make them even less attractive than Windows, so I can't reasonably switch to anything else, no matter how much I'd love to.

Let's start with the fact that I'm a developer. Every operating system seems to be intent on making itself as hostile to developers as humanly possible. The new stuff in Windows 8 is (almost?) all COM based, and COM is one of those things pretty much guaranteed to make any developer with even a shred of self respect and/or sanity wretch uncontrollably every time he gets close to it. Think of the people who refuse to drink some particular liquor because of the time in college (or whenever) that they got horribly drunk on it, and then even more horribly hung over, and have never touched it since.

Well, COM is pretty much like that, except you don't even get that one night of having a little fun acting stupid first. COM is just a horrible hangover that starts from the first time you see it, and apparently won't ever go away! To make it worse, not only is COM itself badly designed, but virtually everything Microsoft does with COM seems to be even worse. You end up with a full page of code to do what should have taken one simple function call. Basing all that's shiny and new in Windows 8 around COM has to be just about the most colossally stupid blunder yet, from a company that seems to have adopted the military attitude toward blunders ("we must plan with exquisite care to ensure against any minor problems getting in the way of committing the most massive blunder in human history.")

"Okay", you say, "why don't you just switch to OS/X?" The answer is simple: in its own, entirely different way (yes, we all know, "different is good" -- but not this time) Apple has done even worse. Microsoft makes you deal with COM -- but Apple makes you deal with Objective-C. If COM is the hangover that never goes away, Objective C is the nightmare that never ends. Where COM is all very "real" and far to nitty-gritty, Objective-C has a bit of a surreal feeling (at least to me). Every time I write anything for MacOS, I get a little of that nightmarish "I just can't believe this" feeling. The problem is that in a nightmare, you eventually wake up, but with OS/X and Objective C, you get your face rubbed in the fact that yes, it really and truly is this badly messed up. The infuriating part is that they come close enough to doing things right that it seems like it must almost be an intentional bait and switch.

Well, perhaps Linux would really be the way to go. In a lot of ways, this *should* be the best choice. Unfortunately, for at least some of what I do, I wrote OpenGL code. On Linux, pretty much the only choice there is Mesa3D. The Mesa project has done some pretty amazing things, and accomplished a lot. Despite that, Mesa3D basically only implements OpenGL 2.1, and I really want support for OpenGL 4. Unfortunately, the gap from 2.1 to 4 is big enough that I can't very well just look the other way, or make do with what it provides.

If OpenGL was the only problem, I could probably manage to live with it, but that's a long ways from reality. Any programmer who's been around for more than a week or two knows about the holy wars between advocates of vi and advocates of emacs that have been going on since "eight megs and constantly swapping" was intended to refer to a simply massive amount of memory, rather than so little that modern script kiddies think "well, of course with only eight megs you couldn't hope to do anything!" In any case, having used both I'm fully prepared to say this: they're both right. Vi and emacs are both awful. The world would clearly be a lot better place if neither of these horrible pieces of garbage had ever been invented.

Now, for any non-programmer who might happen upon this (hunh? Why would a non-programmer read this?) I'll point out that programmers aren't merely religious about text editors. Many people who consider themselves religious might only go to church once or twice a week. Even people who devote their lives to religion rarely pray (or meditate, etc.) for more than 8 or 10 hours a day. A typical programmer spends about 16-18 hours per average day with his text editor (and may easily go 30 hours straight at times). Worse, unlike religious people, programmers absolutely demand immediate, tangible results from that time.

In short, a programmer who doesn't like the editor (or IDE) he's forced to use, is likely to become a bit like some crazed middle-ages crusader, starting to hate life, and trying to make everybody around him do the same. While there are alternatives to Vi and emacs available for Linux, most seem to be in about the same position as I'm dealing with right now about operating systems. They all suck so badly that everybody sticks with what they know, because everything else is too horrible to even contemplate.

Oh, before I leave off on Linux I should add a bit of a rant on one more thing: the other tool that programmers are stuck with using at times is a debugger. Pretty nearly the only debugger available for Linux is gdb. This horrendous piece of garbage is the one thing that can make the editors and IDEs available for Linux almost seem bearable. Yes, they're awful, but it's many times worse. Consider this: back in the early 1980's I did some programming on a Control Data mainframe, mostly using Fortran (and when I could, Pascal). This was roughly three decades ago, on hardware that was pretty much obsolete even then, using an operating system that was mostly a relic from at least a couple of decades before. Even so, its debugger was still better than gdb. It was bad (bordering on awful), but gdb is even worse. This particularly rankles because there were at least reasons for most of the shortcoming in Control Data's debugger -- but gdb doesn't have to fit (along with the program being debugged) into a machine with less than a megabyte of RAM, or somehow protect itself from the target program without benefit of hardware memory protection.

So, that takes us pretty much back to where we started. Much like with editors/IDEs, I find the very notion of continuing to deal with COM loathsome and horrible -- but all the alternatives seem to be even (quite a bit) worse. I'm still left wondering, though, why it is that everybody involved seems so intent on doing the job so badly. At least as far as I can see, Microsoft came to dominance by being the one company that did just a mediocre job instead of a truly awful one, but nowadays they seem to be doing things just as badly as everybody else. If just one OS could rise to the level of just being mediocre, I'm pretty sure every programmer around would flock to it, because even mediocre would be so many times better than the messes we deal with now.