I've been volunteering as a mentor for FIRST robotics team 2809 (one of the many reasons for the deafening silence in these parts of late) and one of the things I'm working on at the moment is a better emulator for the FIRST C++ interfaces.
I got into the whole mentoring gig without thinking too hard about it, and it turned out to be both vastly more fun and vastly more work than I could have possibly imagined. 2809 is only two years old, and most of the high school students who were on the team graduated last year, so while we are extremely fortunately in our university mentors, it was a steep learning curve for a lot of people, myself included.
I have robotic programming experiencing going back a long way. I worked for a while as on a robotic weather station project as a grad student, and my last post-doctoral gig involved development for the robotic calibration source manipulator for the Sudbury Neutrino Observatory. Since then computer assisted/image guided surgery has been a big part of my professional life, so I've always been close to the hardware, and one of the first things I typically do when working with hardware is create an emulator: a software system that mimics the API of the hardware lib.
This has a number of advantages: it frees the software development process from the hardware development process, and as I've found that unlike software, hardware deliverables are sometimes quite badly late, this is a good thing. It also lets you examine software failures, which do happen from time to time, in a strictly reproducible environment, and it lets you study the effects of various non-idealities, hardware failures, and so on.
So all in all having an emulator is a Good Thing. I put one together for the FIRST system quite late in the game, but it was still useful for debugging and testing various coding ideas.
My goal at the moment is to beef up the emulator so it can be used by people who aren't guru-level C++ coders. The learning curve for the FIRST C++ environment is such that it is very tough for the kids to deal with practical implementation issues. Having a better emulator will let me teach them enough in the off-season that by the time the next competition roles around we'll be in vastly better shape to cope.
My original emulator was strictly command line, but I'd like a prettier front end on it, which in the current case I'm writing in wxPython, mostly because it reduces the amount of time I have to deal with Eclipse/CDT. I've thought about driving everything off of Visual Studio, which would certainly make life easier for the kids, and have to confess to still being a little tempted in that direction. But they'll need to deal with Eclipse during build season, and familiarity breeds fluency.
As of tonight, I have the wxPython front end talking to the emulator back-end, and am about to strip out all of this year's code from the robot with the exception of the low-level drive stuff. Our drive system doesn't change much from year-to-year, so I want to keep that base platform intact, and then once the kids are up to speed with it start throwing them some problems of sensors and effectors and seeing what they do with them.
Metaprogramming is writing code that writes code.
There are a lot of ways of going about it, and a lot of purposes to which it can be put. This series is intended to capture my thoughts on metaprogramming, based on over ten years of experience with various ways of doing it.
My first metaprogramming experience came as a user of UML-based code generation tools, which were big in the '90's when UML was the silver bullet du jour. These tools all suffered from some fairly problematic features. In particular:
1) They didn't think like developers
2) They didn't act like developers
3) They didn't write code like developers
4) They barely read code at all
The last point is the most important, and still the most challenging for metaprogramming tools.
By the late '90's I was writing my own code generators, and since then have accumulated a great deal of experience on what works and what doesn't. This experience has brought me to the point where I sincerely believe we should be aiming at tools that can replace junior developers, although that is a goal we haven't quite reached yet.
The Industrial Revolution was successful not because it made certain occupations more efficient, but because it elimiated many occupations entirely. To take a recent example, there used to be a job position called a "computer". These were people--mostly women--who worked in teams to do complex calculations. Looking backward, we can see each one as little more than a single-operation chip, like a multiplier, say. They would take two numbers in, multiply them, and pass the result on to the next person in line.
Mainframes and then desktop computation didn't make these jobs easier: they wiped them out. This is the way mechanization should work, but we aren't there yet.
This is in part because much of what humans do is hard. CAD programs face this challenge. A typical CAD program replaces the draughting table. What we want it to do is replace the draughtsman, and eventually the architect. After all, there are only so many ways to put together a building. There are algorithms for layout, and simulations to optimize against expected use patterns. There will still be a role for humans in setting the goals and deciding the priorities and maybe working up the decor, but a huge number of the choices architects make can be automated, and any task that can be automated, should be.
The same is true of software development. The goal of metaprogramming is to bring us as close as possible to the ideal world of Grady Booch's Rule 122: Never write code unless you absolutely, positively must. If we can write code that writes code, we can get a lot closer to this ideal.
The problem with automating software development is the problem with automating any human task: we need to understand in detail what the humans who perform that task are actually doing. Most of the first generation of metaprogramming tools failed miserably at this, and so the tools didn't really satisfy anyone. Except in a few narrowly defined domains, metaprogramming has failed to realize its promise.
Currently, most practical metaprogamming systems are specifically for generative programming, which is mostly what I'll be talking about. But those systems all focus on making certain tasks that every individual developer does easier. This is the same model that CAD systems use: make the draughtsman's job easier by automating parts of it. While a good thing, this isn't the goal. The goal is to eliminate the entire job category.
One of the things that makes it difficult to eliminate entire categories of development jobs is that software development as it is currently practised is still incredibly heterogenous. That makes it hard to capture invariants that can be automated. However, within a given development context, it is possible to use metaprogramming to ensure that no developer will ever do certain jobs, effectively eliminating the category.
This is a akin to a CAD program that eliminates line-drawing from an architect or draughtsman's job. Imagine an program that could take a specification in terms of patterns of use, say, and generate plans on that basis. Likewise, imagine a program that could take a software specification in terms of desired behaviours and generate the code on that basis. We aren't there by a long shot, but there is no reason to suppose that we won't be there someday, and sooner rather than later.
Code generation is a strangely rare art, possibly because people are trying to solve the wrong problem.
I've used code generators dating back to Visual Studio plugins from the mid-1990's, and eventually started writing my own around the turn of the millennium.
While most code generators are elegant, scriptable, table-driven applications that can be tweaked and customized to handle any number of special conditions, mine are messy, hand-coded affairs that only support the coding style I care about. Yet in my experience many of these elegant systems fail to represent quite enough real-world complexity to be practical aids to development, and more often than not they generate "write only" code.
I treat code generators as junior developers, with the added advantage that they don't want paying. They write only the boilerplate serialization and UI code that would otherwise take up large amounts of relatively unrewarding effort.
They produce code that is indistinguishable in style to the code I write, because they have the corporate coding-standard built in. The ones that generate C++, it is true, use comment-markers to indicate the 'safe' areas for developers to add their own code, which is a product of how hard it is to parse C++. The ones that generate Python take advantage of the language's build-in parser to figure out how not to step on custom additions.
The thing about how messy they are, though, with dozens of special cases hand-coded, is due to a curious fact: good, clean, real-world software is full of special cases and odd exceptions. That is one reason why software development is hard: there are a lot of rules to "good style", to make code readable. This is particuarly true for UI-generating code, where a whole range of different widgets require special conditions to make them behave sensibly and look good, which is surely the goal of all reasonable coding standards.
I'll be writing more here about generating code as time permits. My thinking on the topic has evolved a lot over the past decade, going from grandiose schemes that will change how software development is done, to more modest, practical and effective systems that take a great deal of the day-to-day burden of development off developers, letting them concentrate on design and the custom implementations that will make the application remarkable.
In putting together some Web services stuff for a client I've been
struck by the weird comparisons I've seen between JSON and XML.
Typically,
XML is fingered as a "heavyweight format" as opposed to the lightweight
simplicity of JSON. This is very hard to understand.
Here is a simple XML format for transfer of information:
<N v="V"/> (10 characters to send 2 bytes)
Here is the equivalent JSON:
["N", "V"] (10 characters to send 2 bytes)
It's
all a question of how strongly you are willing to make your assumptions
about data formats. Given the comparison is JSON, which is an "all
assumption, all the time" format, it seems odd that JSON was not
implemented as an XML format, and that the "examples" of XML formats
for data transfer that one sees in JSON comparisons look more like this:
<EgregiouslyLongWrapperName SomeIrrelevantVariable="VeryLongValue>
<NameTagNameMadeAsLongAsPossible AnotherIrrelevantVariable="MoreWaste">
<NameValue Wrap="YesSendAsCDATA">
N
</NameValue>
<NameTagName>
<ValueTagNameMadeAsLongAsPossible AnotherIrrelevantVariable="MoreWaste">
<ValueValue Wrap="YesSendAsCDATA">
V
</ValueValue>
<ValueTagName>
</EgregiouslyLongWrapperName>
Of
course, unlike JSON, XML scales up nicely to add more information,
which may indeed include type information. But XML is a compact,
efficient tool in the hands of anyone who knows how to use it well, and
one should always be highly suspicious of hand-rolled "efficient"
formats like JSON. One suspects they are generally created by people
who can't be bothered to spend a few days learning the
standard-compliant way of doing the job properly, and so have invented
yet another wheel that is only approximately round.
I've been swapping back and forth between C++ and Python a lot in the past month, which causes me to reflect on Russel's theory of types, which was an attempt to avoid certain kinds of conceptual recursion by distinguishing the type of the "two" in "two apples" from the "two" in "two oranges."
It didn't get a lot up uptake from the other philosophers in the playground, and is now considered not much more than a footnote.
Except in software.
The biggest distinction between C++ and Python, after you get over the "oh my god it's full of whitespace" thing, is the type systems, which I'm obviously obsessed with given the content of the postings on this blog. But it's an important question, both philosophically and from a practical software engineering point of view.
Rather than asking "which is better", it is more profitable to ask, "What are the strengths and weaknesses of strongly vs weakly typed languages?"
Advocates of strong typing believe that making the type part of the public interface of any object allows for stronger error checking. Advocates of weak typing believe that advocates of strong typing are smoking some pretty interesting stuff.
In practical terms, I've noticed that the (relatively) strong typing of C++ really does add a significant cognitive burden when writing code. At times it seems like about half the code is type declarations of some kind, often repeated ones. I saw this when writing my Python code generators: entire sub-systems of the C++ generator got sidelined because they were concerned with tracking type information that is irrelevant to Python.
But does the weak typing of Python produce more errors, and if so of what kind?
So far, I have to say: no. With reasonable testing--about which more later--and ordinary best practises in design and implementation, I don't see any errors in Python that can be pinned on not having a strong(er) type system.
I'm aware that Stroustrup once said that C's type system is "promiscuous and stupid", or words to that effect, but I've worked in languages with stronger type constraints and found no greater level of safety with them, either. And modern C++, while still labouring under the burden of C compatibility, is a lot tighter than it once was.
One sees the burden most severely in things like the size-types in the STL, where:
std::string::size_type
is different from:
std::vector<float>::size_type
is different from:
std::vector<std::set<int> >::size_type
even though they are all "really" unsigned ints under the hood in most implementations. Not being able to inter-change values with these types without generating warnings (at best) or errors (at worst) does not make for safer code. It makes for developers who spend some of the precious and limited cognitive capacity on remembering typical trivia, rather than concentrating on design and implementation issues that matter.
The one thing that strong typing does is move a vast range of run-time errors to compile time, and that's a good thing if you never test your code. For those of us who do test early and well, it isn't such a big deal.
The other thing strong types do is permit the compiled code to run more efficiently, because all those run-time checks don't have to happen. But I can't help thinking the concept of type as such is probably the biggest instance of premature optimization we have ever seen.
There are definite advantages to Python's typing policy, sometimes called "duck typing." If an object has methods "waddle" and "quack" then it's safe to treat it like a duck.
Or is it?
For duck-specific terms like "waddle" and "quack", it's fairly safe to assume that objects supporting methods with those names are sufficiently duck-like to be treated as such. But what about a method named "twist"? Does it belong to a dancer, a bartender, or does it have something to do with a 4D complex Weyl spinor representation?
That's a fabricated example, but the example I mentioned in the previous entry isn't. There is more to identity than syntax.
One solution to the problem of distinguishing lists from strings would be to check for the index operator, and then check for some string-supported method to decide if something was a string or a list. Although it would have worked in the context of the problem I was facing, it is in general a very bad idea. It makes some very strong assumptions about what kind of objects are out there, and reminds me of nothing so much as the penchant for Web developers to try to decide what browser their Javascript is running on by checking for a few well-known methods that only have single-browser support at the moment.
The other thing it reminds me of are the various "terrorist watch lists" floating around these days, like the FAA's "No Fly List." People sometimes say that the No Fly List has a million people on it, but this is false. It has a million (plus, and growing) names on it, but having the same name as a Maryland peace activist does not mean you share that person's beliefs.
So while I appreciate the enormous flexibility of the Python type system, to use it well requires sensitivity and taste, like so much else in software development. Or one might wind up sending entirely the wrong code to Gauntanamo.
Python has been on my horizon for a while, and I've written code in it for small projects since 2002 or so. Until recently it didn't seem to me to have sufficiently deep libraries to warrant making the change from Perl. And as a survivor of SGML's "ignorable whitespace" and other nightmares, the idea of parsing on whitespace was offputting. Whitespace is not actually evil--it is just misunderstood. To have your compiler depend on something so easily misunderstood seemed to me problematic at best.
Sometime in the past two years, though, Python's capabilities, and especially the libraries, have crossed the threshold into serious development territory, and a couple of current and prospective clients have projects where Python is very clearly the language of choice, whitespace or no. This created an interesting opportunity.
Predictive Patterns makes very heavy use of code generator technology. Our code generators are aimed at replacing junior developers: they write code from high-level specifications that is clean, conforms to our coding standard, and can be edited easily by humans. This is how all our serialization code is written, for example, using an XML-based technology that is loosely based on the idea of treating the application as a document.
Our code generators are written in Perl, and for some time there has been internal pressure to rewrite them in Python. The use of Python on active projects for clients makes this a necessity. It would just be wrong to have a Perl script generating Python code.
The first step in this project has been to convert the existing C++ generators to Python, which has produced some reflections on the relative strengths of the three languages involved: C++, Perl, and Python.
Like any serious C++ developer, Lakos' Large Scale C++ Software Design has an important place on my bookshelf, not too far from Design Patterns and Effective STL. When I started coding seriously in Python one of the most striking things was the complete irrelevance of physical design principles, which are the focus on Lakos' book. How the code is organized into translation units, how interfaces are used to insulate the underlying implementation... all of this just doesn't much matter, which is both refreshing and a little bit disorienting.
Python's notion of "duck typing" is both refreshing and disorienting too, particularly when it doesn't work. For example, many types support an index operator: []. But the semantics of this operator is completely different for two very important classes of type: containers, and strings.
In C++, standard strings are sometimes referred to as "almost containers", a nice bit of conceptual legerdemain that faces up to this problem: strings have a container-like interface, but at the end of the day are something we want to draw an edge around and call a unitary entity, not simply a container of more-or-less-unrelated characters that happened to get dumped into the same bucket. Strings have a conceptual unity that other container classes do not, despite their similar interfaces. If a C++ "vector" were actually the Cartesian tensor its name suggests we might have a similar problem there, but alas this is not the case.
This difference between containers and strings bit me while developing a Python script that generates SQL out of an XML specification. Trying to be a good Pythonista I distinguished between container types and everything else using a try/except around the index access. This worked fine until the code got hold of a string, declared it on the basis of the interface to be a container, and complacently went on to do something entirely inappropriate and quite embarrassing. At this point I learned that "is_instance" should not always be considered harmful.
The differences between Perl and Python are of course more striking. As I move through the process of translating the main code generator, which is about 3000 lines of Perl, I notice how much more compact the Python implementation is, and how much cleaner the code. With one major exception: regular expressions.
Although at the end of the day Perl regexes are probably too complex, they have an inner elegance, a deep beauty, that the Python re module just lacks. This is a small quibble when taken in context of the gains to be made from using Python, but I can't help thinking that Python 3.0 with regexes built in would be the best of all possible worlds...
