Reenigne blog

If you've got a big, complex application programming interface there are generally two ways to do it. You can assign an integer to each function (or method if it's an object-oriented interface) and share the mapping between integers and method names between the callee and caller, or you can provide a single function or method which takes a string.

The first way is generally used in older and more speed-sensitive code, since it has generally been a bit faster in the past (though for most purposes probably isn't significantly so now, especially for out-of-process calls - the context switch time will dwarf the string-lookup time).

The second way is used on the internet (where everything is text based and the network dominates most timings). It's also used in Windows DLLs (though this is more of a hybrid - the names are resolved to locations at DLL load time). COM, on the other hand, uses the first method (and has a lot of complicated registry/interface/type-library goo to make sure the numbers don't get out of sync).

I think that any time you have an API which is sufficiently complicated that it may change with future software or hardware releases, the string-based method is the way to go. Especially for something like this. Keeping track of all the possible variations of the mapping between method names and integers is always going to get sufficiently complicated that you might as well have the computer do it.

I was thinking some more about this recently. Specifically, APIs are generally implemented as DLLs or shared libraries which you load into your process and which expose functions that can simply be called. But in a world with no DLLs, how do you do API calls?

At a fundamental level, there are two things you can do - write to some memory shared by another process and make system calls. The former is useful for setting up calls but not for actually making them, as the process implementing the API has no way to notice when the memory changes (unless it polls, which is not a good solution). But system calls are expensive - the context switch alone could be many thousands of CPU cycles.

I mentioned in the linked post that bandwidth for inter-process communication should not be a problem but I neglected to mention latency - if you can only make API calls a few tens of thousands of times per second, certain operations could become very slow.

However, the latency of any single call is still far below what could be noticed by a human being - the only perceived speed problems caused by this architecture would be if many thousands of API calls were made as a result of a single user action.

I think the answer is to design APIs that aren't "chatty". Such designs are already in common use in database situations, where high-latency APIs have been the rule for a long time. Instead of having getCount() and getItemAtIndex() calls to retrieve a potentially large array of data, you retrieve a "cursor" which can return many records at once.

Another possibility is for APIs themselves to be able to execute simple programs. Again this is an idea used by databases (SQL, the syntax usually used to access databases, is itself a language in its own right). Such programs should not be native code (since that gives all the same problems as DLLs, just backwards) but can be interpreted or written in some sort of verifiable bytecode (which could even be JIT compiled for some crazy scenarios). The language they are written in need not even be Turing-complete - this is just an optimization, not a programming environment.

If all else fails, there is still another way to get low-latency, high-bandwidth communication between chunks of code which don't know about each others' existence, and that is to create a whole new process. The involved APIs return a chunk of code (perhaps in the form described here) and the calling program compiles these (and some code of its own) into a (temporary) chunk of binary code which then gets an entirely new process to execute in. This moves potentially all of the IPC costs to a one-time startup phase. The resulting design resembles Java or .NET in some respects.

This is kind of like the opposite of the Singularity OS - reliability guarantees enforced by the compiler, but the OS allows you to use all the user-mode features of your CPU (not just the ones supported by your runtime) and processes are isolated by the MMU.

My first thought on reading this article was "what would an rm for the web look like?". It seems that many of the commenters on that post had the same thought.

One possible way this could work is to set up a proxy that people can set in their browser. Then if they don't want to see a particular website any more, they can visit the "rm" website, enter the address of the website that they don't want to see any more and then it would be blocked by the proxy. The "rm" website should of course also provide some way to remove a site from the blocked list.

There are a few reasons why such a thing might be useful. There are a few "shock websites" out there which most people would not want to visit knowingly. It could significantly cut down on the number of instances of rickrolling and spyware infections. It could enable organizations to take control of their own web filtering needs rather than relying on some off-the-shelf solution which is bound to block some sites that it shouldn't. Of course it would be easy to circumvent, but that's not the point - the idea isn't to prevent people from seeing stuff they really want to see, it's to keep them honest and keep them from accidentally stumbling onto unsavory areas of the web.

Since changing jobs I now mostly work with Unix machines rather than Windows ones. Mostly it's a not too unpleasant experience (once things are set up the way I like them) - Unix seems to be much more flexible and configurable than Windows, even if some things don't work quite so well out of the box.

One area I have run into difficulties with is editing text files. When I first started using DOS I used to use edlin, which is horrible. After a while I discovered QEdit on a magazine cover disk and loved it - very quick to load and scroll (even on an 8MHz 8086 machine), easy to use but with powerful macro facilities there when you need them.

I used that up until 2002 when I switched from using Windows 98 to Windows XP and QEdit stopped working so well. I bought TSE (the updated, renamed QEdit) and it was worth every penny - everything that was great about QEdit was the same or better in TSE and I've used it almost every day since for almost every text editing task (though I have been known to use the Visual Studio editor occasionally for its Intellisense goodness). Moving from QEdit to TSE I discovered that the Windows shortcuts for select, copy and paste (shift+navigation to select, Ctrl-X to cut, Ctrl-C to copy and Ctrl-V to paste) were much quicker and easier than the QEdit defaults (which I think originated in WordStar) Ctrl-K B, Ctrl-K K etc.

When I switched to Linux, TSE was one thing that I missed most. It works under Wine but not particularly well (it always defaults to full screen, copy/paste integration doesn't work and scrolling is painfully slow). A Linux version of TSE is in the works but doesn't show any sign of materializing soon.

Joe is the closest thing to TSE that Linux has to offer but it's a bit limited and just different enough to be annoying. It also isn't available on all of the machines that I need to use. The latter requirement basically means I have to make the big choice - vi or emacs?

I gravitated towards vi initially as it seemed to be a bit more lightweight and I had some experience with it from my brief stint using Debian sometime around 1999-2000. I relearnt how to do the basics, put it into syntax highlighting mode and fix the colours (dark blue on black is unreadable to me, and of course comments have to be in green). But it still drives me crazy - I never know which mode I'm so I'm always inserting ":q" into files or (worse) typing text which gets interpreted as random commands and then trying to figure out what I did and how to undo it.

I eventually found out that most people who develop GNU software use Emacs (I saw a lot of people using it at the GCC Summit) and that it has great features for writing C code according to the GNU guidelines. So I decided I had better learn Emacs. I had used it a bit at university (the Physics department gave us a brief tutorial in it as part of the Fortran course). I have also learnt a bit of Lisp in the intervening years and having Lisp as the macro language of an editor seemed to be a rather reasonable thing to do.

One big problem annoyance with all the Unix editors is that none of them support the Windows-style shortcuts I mentioned above (plus Ctrl-Z, Ctrl-Y, Ctrl-A and Ctrl-S for undo, redo, select-all and save) by default. I almost squealed with delight when I discovered the CUA mode in Emacs 22 which brings it significantly closer to being the perfect editor (at least once I figured out how to tweak various terminal settings to actually make it work - this helped. Unfortunately many of the machines at work still use Emacs 21, so there was some more fiddling required to install the CUA package.

Then I need to figure out if I can get the scrolling behavior to work like QEdit's, fix up some behaviors related to tabs and (once again) fix the syntax highlighting colours. "Eventually I'll have everything the way I like it" may be the mantra of a great many Linux users...

Part of the reason this is so difficult is that customizing editors is generally considered an advanced thing to do - you really have to learn the basics first. But if the first thing you want to do with a new editor is make it act like your old one, you first have to learn the default way of doing things, then do the customization, then unlearn the default way of doing things. My natural inclination is to take shortcuts - try to figure out the advanced stuff without really getting the hang of the basics first. This does seem to have a tendency to complicate things.