Software patents, part II.

In my last post, I promised a post about what I think needs changing in the patent system. This necessarily gets considerably more detailed, so it's likely to be harder to apply it to different patent systems. Nonetheless, when people talk about problems with software patents, the US seems to be (by far) the most often cited culprit, so I doubt that being specific to it is really a major problem. Anyway, on with the show.

The first problem I see is that the US patent office is basically run as a business -- it's profitable to the government, and most of the profit comes from issued patents. Worse, what's really intended to be the primary job of the patent office (searching through prior art to figure out whether something is really original) is apparently run at a loss. The profit comes from "maintenance fees" on patents after they're issued.

The patent office has suggested a way to try to streamline the patent examination process that would involve the inventor either citing only a few sources of related art, or (if they know of too many) specifically pointing to those they think are most relevant, including specific citations of the parts they consider relevant.

At least to me, this seems utterly insane. First of all, it's simply attempting to shift the burden of the patent office's job from the patent office to the inventor. Second, it's asking the inventor to predict what the patent examiner will find the most useful. If, for example, the patent examiner looks at some other reference and finds it more relevant than the ones the applicant pointed to as most relevant, would the patent become invalid? If so, the patent system basically becomes a lottery. But, if there's no penalty, what's to prevent an applicant from citing something utterly irrelevant as the most relevant art related to his patent?

The simple fact is fairly simple: as it stands right now, the patent office simply isn't complying with the requirements of patent law. The law says the fees they charge should cover the cost of examining the patent. If their fees aren't high enough to do that, then the fees need to be raised. Another point that's routinely cited is that an examiner has only about 20 hours to examine a given application (there's some disagreement about the exact number, but at least it's at what I'd consider the noise level -- there's little question that it's between  15 and 25 in any case). Another common complaint is that patent examiners simply aren't competent -- that they often just don't know enough about the technology to sort out (for one example) truly new techniques from simply new and different terminology for existing techniques.

These have a simple solution though: simply increase the fees for a patent application to the point that they can/will cover the costs of hiring competent examiners, and giving them enough time to do a good job. Along with funding the process properly, higher fees, all by themselves, would probably do quite a bit to encourage companies to look more carefully at their applications before filing, to ensure that what they're filing is truly novel and useful, before wasting others' time finding prior art to show that it's invalid.

The second major problem I see is that it's much easier to get a patent granted than to get that same patent invalidated after it's been granted. The patent office has been taking some steps that attempt to balance this a bit better, but at least personally, I don't think they've done quite enough yet.

Classically, the situation was that the inventor filed his application, and the patent examiner (usually only one, though sometimes two) tried to find prior art. Under most circumstances, almost nobody else participated in the process at all. The patent office does now publish (most) patent applications, so normal people can review them, but it's still basically provided as a one-way communication -- if you look at an application on the patent office's web site, there's not (for example) a button on the page to let you send an email to that examiner citing what you think is prior art on that patent application. Right now, however, instead of treating prior art submissions as what they really are (normal people doing them a favor to make their job easier) they seem to treat it as if they're doing me a big favor by allowing me to submit anything at all, and it's perfectly reasonable to make me jump through flaming hoops before I can be granted such a privilege.

Even so, the patent office follows a "preponderance of the evidence" rule in deciding whether to issue the patent. That basically means only a simple majority of the evidence needs to point toward its being novel for the application to be accepted, and the patent to issue. If 51% of the evidence says it's original, and 49% says it's not, the patent can be issued. Keep in mind that in most cases, it's up to one person to find any and all evidence of invalidity (in roughly 20 hours of work or less), but a large company may easily have a dozen people working hundreds of hours (if necessary) to get the patent issued, and it's quickly apparent that the system is heavily loaded in favor of issuing the patent unless it's quite obviously bad (and sometimes even if it is quite obviously bad).

Once the patent issues, however, the courts are required to give the patent a presumption of validity. To invalidate the patent, you don't simply need to provide a preponderance of evidence in the other direction. It's not enough at that point to show that 51% of the evidence points toward invalidity, and only 49% toward validity -- instead, the courts require "clear and convincing" evidence that the patent is invalid before they will declare it invalid. I've never seen a court attempt to specify a percentage of the evidence necessary to meet the clear and convincing hurdle, but it's clearly a lot higher than a mere preponderance of the evidence. If I had to characterize it, I'd probably put the number at around 90% -- still somewhat short of the "beyond a reasonable doubt" (that's supposed to be) used in criminal courts, but much higher than than 51% used as the basis for issuing the patent in the first place.

We end up with a system that deprives most people of much ability to provide evidence that a patent application isn't valid, issuing the patent even when/if there's a substantial chance that it's not valid, and maintaining it as valid even when most of the evidence indicates that it's really not.

Though I don't think those are (probably) the only two problems with the patent system as it stands today, I think addressing those two points would go a long ways toward redressing much of the current imbalance in the patent system. Interestingly, neither of these requires any change to the actual patent laws at all, only changes in procedures about how the existing law is enforced. Over the past several years I've seen many proposals that included what initially seem like much larger changes (nearly all including substantial changes to the patent law itself). I believe most of these would do little to cure existing problems, and quite a few would add many more problems of their own as well.

From a programmer's point of view, most of the proposals strike me as doing nothing to try to find or fix actual bugs, and instead advocate changing indentation and naming conventions throughout the code. They'll change how the code looks, and might (conceivably) make it more readable, to at least some degree, but they're almost certain to introduce at least a few new bugs in the editing, but fail (except, perhaps incidentally) to fix existing bugs. Just as with code, to do real good, you need to identify the real problems, prioritize them, and (usually) attempt to make the smallest changes that fix those problems.

I'd also note that almost none of this is specific to software patents at all. Most of the problems I see apply at least equally to areas outside of software. As noted in my previous post on the subject, I think most of the arguments attempting to separate software patents from other patents lack merit.

Why most programmers are wrong about software patents.

Many (probably most) computer programmers favor elimination of patents that apply to software. I believe, as is often the case, that they're well on their way to repeating a dark history they've blithely ignored.

Patents were invented as a service to society -- the basic intent was (and remains) fairly simple: give an inventor exclusive rights to his/her invention for some limited period of time in exchange for agreeing that as soon as that limited period of time expires, the invention goes into the public domain -- anybody can freely use it. By itself, that wouldn't mean much: after all, if the patent didn't exist to start with, anybody would be able to make free use of the invention immediately, instead of waiting for the patent to expire.

The benefit to society comes from another requirement for a patent: that the patent describe the invention so others can use it. The exact phrasing varies by country, but in the US the requirement is that the patent contain a description of the invention sufficient for a person of ordinary skill in the art to implement the patented invention.

Going a step beyond that, US patent law also includes a "best mode" clause that says the inventor must reveal what he believes to be the best method of implementing/deploying/using the invention. For example, if he's thought up a new type of oil for lubricating automobile engines, he can't patent it, but in the patent claim that it's intended to be used as cooking oil (which could deprive society of the benefit, because they ignore it after finding that it makes food taste terrible). Obviously the example of a cooking vs. lubricating oil is a bit extreme, and probably easily seen through -- most people would probably think something was a bit odd about Exxon patenting what they claimed was a cooking oil, for example.

Anyway, let's take at least a quick glance at that dark history they're ignoring. Before patents were invented, most crafts were governed by guilds. To learn how to carry out a particular craft, you had to be admitted into the proper guild. The guild guarded the secrets of its craft quite carefully in most cases -- to the point of sometimes murdering people who were suspected of revealing guild secrets, and refusing admittance to others because they were seen as too dangerous. In other cases, there were literally wars between competing guilds. These not only get people killed, but in some cases when a guild was destroyed, its secrets died with it. Archaeologists have found artifacts showing techniques we don't quite know how to duplicate to this day.

As many problems as they had, guilds were a substantial improvement over the previous system: one of individual craftsmen keeping secrets entirely to themselves, to be passed onto their children, sometimes on their deathbed -- or taking their secrets to the grave, if their children weren't handy at the right time, or were estranged, etc.

Most arguments programmers level against this fall into three broad groups. The first is that programming techniques (algorithms, protocols, etc.) get published even when there are no patents. The second is rarely quite so clearly articulated, but basically comes down to the notion that if one person invented something, they can invent something at least equally good when and if they need to. The third is that software moves so quickly that by the time a patent has expired, it's no longer of any value.

The first is partly true. They can point, for example, to the fact that European companies continue to publish despite the European policy against software patents. This, however, ignores reality. What's happening right now is that Europe is basically "freeloading" off of the patent systems of other countries (US, Japan, etc.) If, for example, Seimens or Philips wants a patent in the US and/or Japan, they need to publish all the details of that invention in their US/Japanese patent application. That means the invention is no longer secret -- and given current communications, the minor detail that it's published in the US/Japan instead of somewhere in Europe adds (at most) a few milliseconds for a European citizen to download the patent application. As such, Europeans enjoy the benefit of inventions being published, without (themselves) giving the inventor anything in return.

The second strikes me as simple conceit. Yes, there are patents running the gamut from utter stupidity to simply being something nobody else thought was important or original enough to bother patenting. Being at all honest, those constitute only a fairly small percentage of patents overall though. Much of this stems from a problem I've seen many times: somebody will draw a badly mistaken conclusion about what the patent covers, often based on looking at nothing other than the patent's title. They see a title that says something like "A method for run-length compression", and they think something like "good lord what insanity -- everybody's known about RLE for ages -- obviously the patent office is a bunch of idiots!" Though it's barely possible they're (at least sort of) correct, and this really is a patent on something everybody's known about for decades, chances are it's something else entirely -- some (possibly minor) improvement that allows the job to be done a little faster, or using a little less memory, etc. In a typical case, it's not earth-shaking, but is original and at least mildly useful.

The third argument may sometimes have some degree of validity as well, but often doesn't. This argument, however, ignores that fact that if the invention becomes obsolete quickly, the cost of the patent (to the people) is also low -- if an invention is only meaningful for a year or two, the cost to society of the fact that an inventor (theoretically) gets exclusive use of it for years afterwards costs society essentially nothing. At the same time, people have invented and patented algorithms that retain value well after the patent expires. An obvious case in point would be RSA encryption. The US patent on RSA expired well over a decade ago, but it's still almost certainly the most widely used from of PK cryptography. RSA remains (roughly) as valuable today as ever, although there are alternatives that could be used if RSA were not available.

Now, don't get me wrong: I don't think the US patent system (or probably any other) is anywhere close to perfect. Some time soon, I'll write a post about what I see as problems in the system, and at least some ideas about how to fix those problems. To give fair warning: I believe many of the problems can be fixed with much less drastic changes than most people seem to think are necessary. In fact, some of the changes are (or at least initially seem) so minor that they're probably almost invisible except to attorneys who specialize in intellectual property law. Nonetheless, I think some seemingly trivial modifications can have fairly serious consequences.

C++ dynamic arrays

I've seen dozens (at least) of questions and proposed answers about how to create a dynamic array in C++. The typical reply recommends using the array form of the `new` operator to allocate space, something on this order (but usually less complete):


template
class dynamic_array {
    T *data;
    size_t size;
public:
    dynamic_array() :data(0), size(0) {}

    dynamic_array(T *init, size_t i_size) :size(i_size) {
        data = new T[size];
        for (size_t i=0; i<size; i++)
            data[i] = init[i];
   }

    bool add(T const &new_item) {
        T *temp = new T[size+1];
        for (size_t i=0; i<size; i++)
            temp[i] = data[i];
        ++size;
        delete [] data;
        data = temp;
    }
     
    T &operator[](size_t index) {
         assert(index < size);
         return data[index];
    }
    // We also want a const version of operator[].
    // We also probably want something like get()
    // in std::vector, that throws if index is out of bounds.

    size_t get_size() { return size; }

    void clear() { size=0; delete [] data; }

    ~dynamic_array() { delete [] data; }
};

Now, it's certainly true that you could do worse than this. I believe it manages memory correctly (doesn't leak memory or create dangling pointers), and it is dynamic (to some degree -- it allows adding, but not currently removing, elements dynamically). While it's not currently written to be exception safe, that's mostly to keep it from getting too long -- making it exception safe wouldn't be particularly difficult.

It still has some pretty serious problems though. First of all, it requires that T be default constructible and assignable. Assignable isn't so bad (though in C++11, we'd prefer movable), but default constructible is kind of a pain, especially since we never actually *use* any default objects -- but when we expand the allocation, we have no real choice but to default construct the objects, then assign each value from the old array into the new one. That can be a pretty serious burden if creating a default object is slow, and worse still if assigning objects is slow as well. Since it's a template, we don't know much about T. Either or both might be true.

That points to a much bigger problem: adding an item to the array is slower than we'd like, and the bigger it gets, the slower adding items gets. In computer science terms, adding a single item has complexity that's linear on the current size of the array.

Unfortunately, we can't just do a minor rewrite inside of add() to improve the situation. In fact, the basic premise of how this dynamic array works turns out to be wrong. The only way to fix the biggest problem is to start over with something that works a bit differently from the beginning.

Our big problem here is that when/if we use `new whatever[size]`, we not only get enough space to store the specified number of objects, but code is inserted to create that number of objects in the storage. Generally speaking, we don't want that. Unless we change the interface at least a little bit, there's never a circumstance in which we really want objects in that memory. Even if they're created, immediately after allocating those objects, we're going to assign some other object to that position anyway. We'd really prefer to copy construct the new objects from the old objects (or, in C++11, move-construct). This way a new object is created with the correct value instead of being created with a value we know is wrong, then immediately assigning the correct value to the new object.

We'd also prefer to avoid re-allocating space every time we add a new item. We really want to allocate a chunk of raw memory, turn that raw memory into new objects as they're added, and only allocate more space once in a while, when our memory is all used up.

Using new T[size] won't let us do either of those though. Instead, we need to use something (usually operator new, but malloc would also work) that lets us allocate raw memory, then use placement new to create objects in that raw memory.


template
class dynamic_array {
    T *data;
    size_t allocated;
size_t in_use;

size_t round_up(size_t input) {
size_t ret = 1;
while (ret < input)
ret <<= 1;
return ret;
}

void alloc(T *init, size_t current_size, size_t new_size) {
char *temp = (char *)operator new(new_size * sizeof(T));
allocated = new_size;
for (size_t i=0; i<current_size; i++)
new(temp+i*sizeof(T)) T(init[i]);
in_use = current_size;
data = (T *)temp;
}

void destroy(size_t b, size_t e) {
for ( ;b!=e; ++b)
data[b].~T();
}

public:

    dynamic_array() :data(0), allocated(0), in_use(0) {
alloc(0, 0, 10);
}

    dynamic_array(T *init, size_t i_size) {
alloc(init, i_size, round_up(i_size));
}

    void add(T new_item) {
if (in_use == allocated)
alloc(data, allocated, allocated * 3 / 2);
new ((char *)data + in_use++ * sizeof(T)) T(new_item);
    }

    ~dynamic_array() {
destroy(0, in_use);
operator delete(data);
}

    T &operator[](size_t index) {
        assert(index < in_use);
        return data[index];
    }

    size_t size() { return in_use; }

    void clear() {
destroy(0, in_use);
in_use = 0;
}
};

This is starting to run pretty long, so I'll leave further embellishment of this class for a later post. This is still quite a ways short of what std::vector already provides though -- thus the standard advice to just use a standard container if at all possible. Even for something as simple as a dynamic array, the amount of code involved becomes fairly substantial. For a more complex container like std::set or std::map, the savings is greater still.

Herb's wrong, I'm right, and here's why.

Microsoft's MSDN Channel 9 recently released a video of Herb Sutter, Andrei Alexandrescu, and Scott Meyers from C++ and Beyond 2012. In this video Herb (in particular) argues that further support for meta-programming should not be added to C++.

He points out (quite rightly) that meta-programming is basically used to add features to the language. He argues that rather than supporting meta-programming, we should look at what features meta-programming is used to implement, and add those features directly to the language itself.

In my opinion, there is really only one reasonable answer to this: wrong!

This is utterly and absolutely wrong for a number of reasons, most of which (interestingly enough) have been expounded many times before. One of the mantras of C++ development for years has been that the committee does not and cannot know all the purposes to which the language might be put. Rather than adding features to the language specifically for every individual purpose and each end user, there has been a concentration on adding features to support things like writing libraries so that adding a relatively small number of features can support a wide variety of uses.

Trying to look at all uses of meta-programming and find the features being created with it, then trying to add those features to the language would be a 180 degree reversal of this decision. The problems already cited would become all too real and all too problematic essentially immediately.

Consider what would happen if I wanted to write a library using features not currently included in the base language. My choices would be to implement that code using meta-programming, or else petition the committee to please, please consider my proposal for a new language feature. For the sake of argument, let's assume that the C++ committee starts to produce revisions twice as fast as they have in the past -- or than essentially any other ISO standard committee for any other programming language has either -- and produces an update every five years instead of roughly once every ten years.

That means, on average, the programmer has to wait 2.5 years before he can even get started on his library. Given that the standard is typically frozen for a year or so before being approved, it's really a bit longer than that, but let's just leave it at 2.5 for the sake of argument. I hope I don't have to spend any time convincing anybody that a two and a half year delay in getting started would be unacceptable for many projects.

That's probably the least of the problems though. Much worse is the fact that accepting any more than a minuscule number of such proposals would quickly lead to C++ becoming substantially larger and more complex. Virtually everybody agrees that C++ is already larger and more complex than anybody would like, and going this direction would unavoidably lead to amplifying that problem.

I do agree with Herb that the current meta-programming facilities have serious problems. The code itself, the compile times, and especially the error messages can all be truly horrific. More importantly, these aren't just remote possibilities; they happen on a regular and routine basis.

That does not, however, mean that template meta-programming in general is a problem and should be avoided. It means that the current facility has problems, most of which stem from the fact that its intent was never to support meta-programming at all.

These problems should not lead us to conclude that meta-programming is evil in general and should, therefore, simply be avoided entirely. They should convince us that the current meta-programming facilities are inadequate and in need of replacement. What we should be doing now is looking for a simple, small, relatively easy to understand set of features that will support meta-programming more sanely. I won't try to propound here exactly what features are necessary, much less what form those features should take. This is, however a basic engineering problem. It's the kind of thing that's been solved before, and can be solved again -- and while the result undoubtedly won't be perfect, it will be good enough to fix the problems that are now obvious (and probably quite a few nobody's thought of yet).

Early in the history of programming, many thought of it as a priesthood. It was liberated largely by improvements in tools and education. At least in my opinion, the situation here is similar.
I guess if I were a great leader of men, at this point I'd come up with some ringing phrases about "when faced with such a challenge, do we shrink in fear, or do we rise up and vanquish the foes of free computing everywhere?"

Since I'm an engineer, though, I'll simply reiterate that this type of problem has been met and overcome before, and can be again. What's necessary isn't vilifying template meta-programming or its current users, but to make sound engineering decisions to fix a problem in the most reasonable fashion possible to make meta-programming dependable, accessible, and understandable, so it becomes the tool that can fulfill the promise shown by the current uses of template meta-programming.

A rant on operating systems and development.

I guess I'm overdue on playing my role as an Andy Rooney imitator, whining about crap that should be taken out behind the barn and put out of everybody's misery.

This time I'm going to talk about operating systems. I think the operating system situation has gotten completely out of hand. Every one of them seems to be competing at being the worst it can be. I'm still using Windows, but only because the people writing OS/X, Linux, *BSD, etc., seem to be doing everything in their power to make them even less attractive than Windows, so I can't reasonably switch to anything else, no matter how much I'd love to.

Let's start with the fact that I'm a developer. Every operating system seems to be intent on making itself as hostile to developers as humanly possible. The new stuff in Windows 8 is (almost?) all COM based, and COM is one of those things pretty much guaranteed to make any developer with even a shred of self respect and/or sanity wretch uncontrollably every time he gets close to it. Think of the people who refuse to drink some particular liquor because of the time in college (or whenever) that they got horribly drunk on it, and then even more horribly hung over, and have never touched it since.

Well, COM is pretty much like that, except you don't even get that one night of having a little fun acting stupid first. COM is just a horrible hangover that starts from the first time you see it, and apparently won't ever go away! To make it worse, not only is COM itself badly designed, but virtually everything Microsoft does with COM seems to be even worse. You end up with a full page of code to do what should have taken one simple function call. Basing all that's shiny and new in Windows 8 around COM has to be just about the most colossally stupid blunder yet, from a company that seems to have adopted the military attitude toward blunders ("we must plan with exquisite care to ensure against any minor problems getting in the way of committing the most massive blunder in human history.")

"Okay", you say, "why don't you just switch to OS/X?" The answer is simple: in its own, entirely different way (yes, we all know, "different is good" -- but not this time) Apple has done even worse. Microsoft makes you deal with COM -- but Apple makes you deal with Objective-C. If COM is the hangover that never goes away, Objective C is the nightmare that never ends. Where COM is all very "real" and far to nitty-gritty, Objective-C has a bit of a surreal feeling (at least to me). Every time I write anything for MacOS, I get a little of that nightmarish "I just can't believe this" feeling. The problem is that in a nightmare, you eventually wake up, but with OS/X and Objective C, you get your face rubbed in the fact that yes, it really and truly is this badly messed up. The infuriating part is that they come close enough to doing things right that it seems like it must almost be an intentional bait and switch.

Well, perhaps Linux would really be the way to go. In a lot of ways, this *should* be the best choice. Unfortunately, for at least some of what I do, I wrote OpenGL code. On Linux, pretty much the only choice there is Mesa3D. The Mesa project has done some pretty amazing things, and accomplished a lot. Despite that, Mesa3D basically only implements OpenGL 2.1, and I really want support for OpenGL 4. Unfortunately, the gap from 2.1 to 4 is big enough that I can't very well just look the other way, or make do with what it provides.

If OpenGL was the only problem, I could probably manage to live with it, but that's a long ways from reality. Any programmer who's been around for more than a week or two knows about the holy wars between advocates of vi and advocates of emacs that have been going on since "eight megs and constantly swapping" was intended to refer to a simply massive amount of memory, rather than so little that modern script kiddies think "well, of course with only eight megs you couldn't hope to do anything!" In any case, having used both I'm fully prepared to say this: they're both right. Vi and emacs are both awful. The world would clearly be a lot better place if neither of these horrible pieces of garbage had ever been invented.

Now, for any non-programmer who might happen upon this (hunh? Why would a non-programmer read this?) I'll point out that programmers aren't merely religious about text editors. Many people who consider themselves religious might only go to church once or twice a week. Even people who devote their lives to religion rarely pray (or meditate, etc.) for more than 8 or 10 hours a day. A typical programmer spends about 16-18 hours per average day with his text editor (and may easily go 30 hours straight at times). Worse, unlike religious people, programmers absolutely demand immediate, tangible results from that time.

In short, a programmer who doesn't like the editor (or IDE) he's forced to use, is likely to become a bit like some crazed middle-ages crusader, starting to hate life, and trying to make everybody around him do the same. While there are alternatives to Vi and emacs available for Linux, most seem to be in about the same position as I'm dealing with right now about operating systems. They all suck so badly that everybody sticks with what they know, because everything else is too horrible to even contemplate.

Oh, before I leave off on Linux I should add a bit of a rant on one more thing: the other tool that programmers are stuck with using at times is a debugger. Pretty nearly the only debugger available for Linux is gdb. This horrendous piece of garbage is the one thing that can make the editors and IDEs available for Linux almost seem bearable. Yes, they're awful, but it's many times worse. Consider this: back in the early 1980's I did some programming on a Control Data mainframe, mostly using Fortran (and when I could, Pascal). This was roughly three decades ago, on hardware that was pretty much obsolete even then, using an operating system that was mostly a relic from at least a couple of decades before. Even so, its debugger was still better than gdb. It was bad (bordering on awful), but gdb is even worse. This particularly rankles because there were at least reasons for most of the shortcoming in Control Data's debugger -- but gdb doesn't have to fit (along with the program being debugged) into a machine with less than a megabyte of RAM, or somehow protect itself from the target program without benefit of hardware memory protection.

So, that takes us pretty much back to where we started. Much like with editors/IDEs, I find the very notion of continuing to deal with COM loathsome and horrible -- but all the alternatives seem to be even (quite a bit) worse. I'm still left wondering, though, why it is that everybody involved seems so intent on doing the job so badly. At least as far as I can see, Microsoft came to dominance by being the one company that did just a mediocre job instead of a truly awful one, but nowadays they seem to be doing things just as badly as everybody else. If just one OS could rise to the level of just being mediocre, I'm pretty sure every programmer around would flock to it, because even mediocre would be so many times better than the messes we deal with now.

Still more about version control

This is (more or less) a counterpoint to my last post. For those who found it forgettable (or unreadable), the primary point of my last post was that version control systems combine too many separate (and mostly unrelated) components into a single package. Instead of being built as a single package, a number of components should be specified, to give users the ability to combine the pieces as necessary to fit their needs.

This post is mostly about why that hasn't happened (not only with version control, but with many similar tools as well), and even if it did, the result would probably be pointless and useless. The short summary of the reason is that nobody has designed a usable way to specify/define interfaces between components like this that's independent of both language and underlying operating system.

There are certainly a number of candidates, including CORBA, COM and CLI, Java Beans, Linux D-Bus, etc. Each of these, however, has problems, at least from my perspective.

CORBA is almost a dead issue -- it's been around for decades now, and never achieved enough market share to really matter to more people. It also imposes a fair amount of overhead in many cases, ORBs are often fairly expensive, etc. While it's sometimes used for really big systems, for something like version control it's probably overkill even at best.

COM might be a (halfway) reasonable answer on Windows, but it's really only native and (even reasonably) well supported on Windows. On some other systems there's some support, but it's generally only to support some sort of interoperation with Windows or Windows applications. CLI was (to some extent) intended to be a successor to COM, and to some extent it is. CLI does make it relatively easy to design interfaces between components that (for example) are written in different .NET languages. That is an important qualifier though -- while it's easy to support components in different .NET languages, supporting a component in a language that isn't directly supported on .NET is doubly difficult.

Java Beans are obviously Java-only. At least in my opinion, they're a lousy design anyway.

Linux D-Bus has pretty much the same problems as COM except that its native system has a much smaller market share.

One of the big problems (as I see it) is that nearly every time somebody tries to define something like this, they look at data being marshalled between components, and realize that this data could just about as easily be sent over a network connection as directly from one component to another. Even though the observation is true in itself, I will posit that this is a mistake for at least two reasons.

The first reason is that even though it's true that you can send the data over a network connection just about as easily as between components on one machine, supporting such a distributed system in a meaningful way requires a lot more than just that. Suddenly, you add in requirements for authentication/security, locating a server for a particular service, etc. Solving these, in turn adds still more problems. Just for example, security often involves keys that are only valid for a limited period of time. That, in turn, requires synchronizing all the clocks in the network.

The second reason is that even though from a purely conceptual viewpoint, it doesn't matter whether a function is being called on machine A or machine B, from a practical viewpoint it often does. A network connection imposes much greater overhead than a normal function call -- enough more that you nearly need to take the difference in overhead into account when designing a system. If you're going to invoke a service on some machine that may be halfway across the country, connected by a dozen hops with an overhead in the range of tens or hundreds of milliseconds, you need to ensure that you make a *few* function calls, each of which accomplishes a great deal. Even via some neutral interface, if the caller and callee are on the same computer, we can expect the overhead of calling a "remote" function to be *much* lower -- nanoseconds to (possibly) low microseconds, not tens to quite possibly hundreds of milliseconds.

These existing systems are poorly suited to the relatively small, simple, types of systems I have in mind. I do think something better could be done (or maybe just popularized -- D-Bus is actually quite good). I'm a lot less certain that anybody's putting much (if any) real effort into doing it though. Bottom line: for most practical purposes, you can't do it now, and probably won't be able to very soon either.

More about version control

Over the last few weeks, I've been thinking a little more deeply about version control, and why I see it as a problematic situation at the present time.

After some thought, I've decided that the problem is that version control has been (and currently is) primarily oriented toward (what are intended to be) complete products. I think, instead, development in version control should be oriented toward components, with some defined protocols for those components to use when they talk to each other.

Defining these protocols would be non-trivial, but I think doing so would go a long ways toward curing what I see as problems with version control at the present time. It take virtually no looking at all to find multitudes of opinions about the strengths and weaknesses of various version control systems -- Git has a great merge engine, but a very expert-oriented UI and is really only usable on Linux. Mercurial is more friendly and portable, but doesn't have as good of a merge engine. Depending on the kinds of changes involved, one might be substantially faster than the other at propagating changes from one repository to another. Windowing systems for either (to the extent they're available at all) work by generating command lines and spawning the appropriate commands. Virtually every reasonably modern version control system defines at least one protocol to make repositories available via a network. Even though almost everybody agrees that distributed version control is generally a better long-term solution, many still argue that Subversion is enough simpler to justify its use in many smaller shops.

IMO, this leads to a problematic situation: a version control system includes so many parts with such minimal relationships with each other, that virtually every one of them has quite a few shortcomings for nearly every real situation (nearly the only obvious possible exception being Git for Linux kernel development).

Now, as I see it, a version control system is primarily a file system. Where a "normal" file system sees each file as a single item, in a version control system each file may actually be multiple items. In some cases, those items form essentially a single line over time, so there's a single current file at any given point in time. In other cases, "a file" may include multiple "threads", each of which has a current version at any given point in time.

At the very least, this means a version control system needs not only a way of specifying a file, but a point along the timeline for that file. The versions that support multiple branches of a file also need a way of specifying which branch is desired. For convenience, we want some way to do one or both for an entire (often quite large) group of files.

To support the file system, you need at least a few components: the most basic and universal is a differencing engine -- something to look at two versions of a file, and figure out some (hopefully minimal) set of changes to transform one file into the other. This lets us store multiple versions of a file as a single file with multiple (usually fairly small) sets of changes instead of lots of copies of the whole file.

Another that's increasingly common and important is more or less the opposite: a merge engine. Given some starting file and some sets of changes, an ability to merge those changes into a single "current" file automatically finds all the changes that don't conflict with each other, and includes them all, and makes it easy for a person to select which changes to include when there are conflicts.

As mentioned above, for a reasonably modern system we also normally want some way to make a repository available over a network -- possibly only an internal network, but for many purposes, making it directly available to external users via the Internet is useful as well.

In the current situation, all of those (mostly unrelated) parts are embodied in a single product. Somebody who wants to specialize in text merging has to spend a fair amount of time getting to know one code base and how its merge engine talks to the rest of the system before he can make his ideas usable with one system. Having done that his ideas remain useful only for that one system, so if somebody wants to do the same cool stuff with another system, it's probably going to take a lot of work learning both systems well enough to figure out how that code works, and how it'll need to work differently in the new system, and translate from one to the other. It's enough work that even if somebody does it once, chances are pretty good that it'll lead to forked development, with the two having to be maintained in parallel from that point forward. The same, of course, is true of all the other pieces and parts -- the diff engine, the network protocols, etc.

When looking at actual code, most programmers agree on things like "single responsibility principle", "Don't Repeat Yourself (DRY)", etc. Within a single code base, most of us apply these (albeit with varying degrees of rigor). Strangely, however, applying these same principles at a larger scale seems quite unusual -- and that's exactly what I see here. Practices we wouldn't even consider tolerating within a code base, are completely normal and expected across code bases.

Now, it is certainly true that version control is not unique in this respect. Early in the history of (almost?) any kind of tool, it's common to see more or less exploratory programming, where people with all sorts of different ideas do all sorts of radically different things. Over time, however, the field tends to narrow to the point that it's reasonable to standardize many of those -- and explicitly state what parts are being treated as abstractions with standardized interfaces but implementation open to variation. Looking at the "field", however, I think version control has reached the point that it's ripe for standardization.

Given the degree to which version control is taken as the sine qua non of software engineering in general (e.g., one of the questions on the "Joel Test" is "Do you use source control?") it seems...interesting that little has been done to apply the principles of software engineering not only to the code for a specific version control product, but to version control as a whole.