Larval Mode: a Linucon 2005 panel

by Posted on

The name Larval Mode comes from the Hacker’s Dictionary, where it means the state of being a novice programmer or techie. The panelists were Eric Raymond and “one of the unsung heroes of the internet” John Quarterman. They did this panel specifically for a group of high school students who traveled to Linucon.

Here are some topics they talked about.

I can’t guarantee that I quoted Eric Raymond or John Quarterman correctly, but I tried to capture the essence of what they said.

Eric Raymond starts by asking the audience what they want to hear and why did they come here. It turns out nobody has any specific questions or specific goals for coming here. They mumble things like “Just want your take on programming in general…”, or “To listen to you…”. Eric laughs and declares it completely unhelpful. So the panelists don’t know what to talk about. John Quarterman threatens to talk about risk management. Apparently that’s his favorite topic. This threat actually makes the audience to come up with some questions.

Somebody from the audience.   Is there a lot of hacking going on these days?

Eric Raymond. There’s a whole lot of hacking going on, if the word is used CORRECTLY! (Audience laughs.) I do it all the time. All of my friends do it all the time.

John Quarterman. We go to the annual invitation-only hackers’ convention!

Eric Raymond is adamant about a distinction between hackers and crackers. Crackers are a separate lower form of life, he says. He tries to estimate the size of the open source community (which, of course, more than anyone else, can claim the title of hackers) and says it grew way, way beyond the most optimistic estimation he had in 1980s. “I know the last time I tried to do a significant statistical assay is in 2001. I think we found 45 000 individual contributors on Sourceforge” (he goes on to give more statistics I could not understand from the recording).

Jay Maynard from the audience. So you are saying that approximately one out of 1500 people is a hacker.

  Hackers, crackers: what are they up to these days?

John Quarterman says it used to be that the computer break-ins were performed by script kiddies and such, and crackers who did not mean any harm, just wanted to do it as a proof of their abilities, but now it’s mostly criminals.

Eric Raymond. Yeah. There’s been an interesting shift in the demographics of spamming, cracking and phishing. One of the things I noticed is, I don’t see a lot of spam anymore. I used to get a lot of that. Nowadays it’s more actual attempts to zombie my machine so it could be used for flood [?] attacks. So there’s been a shift from sleazy advertising to actual attempts to take over large networks of machines to be used for spamming.

John Quarterman. Why did that happen? Because the net got big enough. When the net got big enough [to do legitimate business, it also got big enough for shady business, such as to steal information for blackmail].

Large companies have lost several million of digital records because they shipped it by UPS and UPS lost them. There is another case, Credit Card Systems case, where a number of merchants using credit cards [stored customers’ information with Credit Card Systems?] and it never occurred to them, hmm, somebody’s got to realize [that there’s a lot of stuff all in one place and is convenient to break in and steal].

Eric Raymond says he’s still thinking what the worst case scenario is for viruses of mass destruction. He mentions a story he wrote about it, The final Virus.

Eric Raymond. And I realized, right now, as bad as fishing and spamming problem is, we’re in a relatively benign environment, because those criminals don’t actually want to destroy machines or incapacitate them. They want to use them. But I’m starting to think about what would happen if somebody actually wanted to take down as many PCs as possible. And I realize it would be really easy. All you would have to do is, there are virus kits out there that you can insert a payload of your choice. [It will generate a virus that] waits until a certain date, waits several months, so that it would have time to spread; then pops up and reformats the disk. It would be trivially easy to write. Unfortunately, there are people in the world with a motive to do this. They have Arabic names.

John Quarterman. Most successful terrorists are Tamil Tigers. They have killed two heads of state.

  A related notion: open source warfare

Eric Raymond. The nightmare scenario I carry around is what would happen if Al Qaeda gest smart enough. They could pretty much destroy every PC that runs Windows in the United States. I’m thinking it’s probably a question of “when”.

John Quarterman recommends the Global Guerrilas blog for those interested in this kind of stuff.

John Quarterman. Some people operating in Iraq, and in places like Chechnya… What do terrorists know? Killing people provokes a reaction. [And by causing governments to overreact they get people to come over to their side.]

Eric Raymond. That’s not the only tactic. Al Qaeda is rather successful in using a second tactic, which is to recruit the media and intelligentsia to run the war for them.

John Quarterman. But what if you could attack systems instead, without killing anybody? And it’s what’s happening in Chechnya. They decided what if we attack petroleum pipelines? Suddenly Russia decided to have a referendum on independence. So they’re arguing smart.

So, what if they get control of botnets and viruses and use them for their purposes?

People may like to think of Bin Laden sitting there as an apex, [they may think of terrorism as a] top-down control thingy. [But it’s more like this:] Somebody goes out and does something, media picks it up, and the terrorists watch what reaction it would provoke. And other guys will modify it and make it more effective. What does it sound like? A lot like open source.

Eric Raymond. It is evolution under pressure…

  Why is open source software so poorly documented?

John Quarterman. Somebody needs to ask a question or I’ll start talking about risk management.

Eric Raymond and John Quarterman at Linucon 2005
CIMG0814 Eric Raymond and John Quarterman at the Larval Mode panel at Linucon 2005

Somebody from the audience. Last year you went crazy about CUPS, and … [did anything change since then]?

Eric Raymond. Maybe, but I don’t know. The point wasn’t to go crazy about CUPS, the point was to [express some thoughts on open source software design philosophy].

CUPS is a common UNIX printing system. I tried to set it up one day and it was horrible. It was a combination of mutually reinforcing blunders by […] and Red Hat. I got lead down blind alleys. So I started taking notes about what I was going through, and ended up writing a whole horrible saga as an example of what was wrong with open source user interfaces.

A guy in the audience. People don’t want to do un-fun things. They just want to get it working, and oh, it works now; working is a relative term. And once it works, it’s all the backend testing and QA stuff…

Eric Raymond. That’s too simple: these were design problems. They were not problems of not being sufficiently tested. These were problems with people simply not paying attention to what the system looks like from the end user point of view.

The guy. Well, I mean, it’s not fun to install it on a bunch of different systems and make sure all these weird things that someone could potentially have happen were fixed. Would you want us to do that, would you want…

Eric Raymond. Don’t those people have any professional pride? I never let anything I ship go out looking like that…

The guy. I think they have a need, and once the need is fulfilled, they […] [When I do stuff and I get something working, I share my results, I know this works, and I’m not gonna assume someone else doesn’t know as much as I do, and I don’t have experience with documenting.]

Eric Raymond. I actually like writing documentation, at least of my own stuff. I often write documentation before I code it. Which makes me sometimes a pervert among most programmers.

John Quarterman. I trained myself to write it as I program, but not before.

  Motivation, not details

John Quarterman brings up another important issue, discussed in this panel. In the early days of software engineering, documentation consisted of a list of options but it would provide no clue of how to accomplish what you’re trying to accomplish with this program. “No, it’s 40 different options you have to wade through to figure it out.”

Eric Raymond. The main things that’s missing, and this is a lesson for those of you who might go into programing some day, the main thing that’s missing from documentation that programmers write isn’t detail — there’s a lot of that. It’s motivation. Documentation is not sufficient to help people how to do thing X: you also need to talk about when, and why they may want to do this.

John Quarterman. A few examples go a very very long way. A few examples plus a rationale why they are useful examples.

John Quarterman at Linucon 2005
CIMG0817 John Quarterman at the Larval Mode panel at Linucon 2005, holding a drawing of himself. Could it have been a caricature by Howard Tayler, who was also a guest at Linucon 2005?

Eric Raymond. So, technical writers, don’t leave users in a lurch!

There is a systemic problem in software of all kinds, not just open source. On one hand, you have documentation that was written by somebody who’s afraid to scare any animals, so it doesn’t tell you enough, it’s very superficial and it covers a couple of trivial cases, and a lot of what you want to know isn’t there; on the other hand, you get documentation written by programmers, especially Unix programmers, who are obsessive about describing every single little detailed option, everything it could possibly do, and it’s too much information. And what’s missing in both kinds of documentation is motivation.

For every switch that you have, it’s not sufficient to say, this is what it does. User also needs to know, why would I want to use this? [When is it appropriate?]

John Quarterman. When you use no options whatsoever, it should do the most common thing that the users want it to do. You should not have to figure out 40 different options just to do the most common thing.

  Better than options: programs that configure themselves

Eric Raymond. And better than options are programs that configure themselves. Ooh, I can tell a good war story about this.

The single program I put the most hacking time into over the last year is called GPSD. It is a service daemon for modeling GPS’s global positioning system receivers [?] that are attached to serial USB devices on the computer. And the problem… there are two problems that this solved. One is that if you have two applications, both of which want to be aware of your location, and they’re both are trying to look at the same serial port, the USB port, it’s not good. They can tamper with each other. You need a multiplexer. You need some kind of program that is able to broadcast that location from where it’s picked up on GPS and make it available to multiple applications.

The other problem is that the standards for how GPS is reported to computers [?] are poorly designed, weak and crappy. Which is to say they were written by vendors. (Laughter.) They were written by GPS vendors.

A guy from the audience. GPS vendor committees.

Eric Raymond. Yeah, GPS vendor committees. The result is that there are about 60 mutually incompatible protocols for how GPS reports to a host. And to some a way to solve both the multiplexing problem and the protocol problem is to have a service daemon, whose job is to watch these devices, decode the bit stream coming in from the GPSes, and then publish the location information on a well-known […] port, in a simple format that’s documented, that applications can pick up. That’s what GPSD does.

  Eric Raymond’s challenge: write a self-configuring program

Eric Raymond. When I got my hands on GPSD, though, one of the things you needed to do was you needed to tell it: you need to talk to this device, and you need to do it at this baud rate, and this is the protocol you should expect. This is bad. This means that somebody who needs to use GPSD needs to know technical details about the device they’re plugging into their computer, that they would never ever use again and don’t care about! This is bad.

Programmers, especially Unix programmers, write a lot of user interfaces like this, because Unix programmers always have a mighty idea that hardware and other programs are dumb and I need to be able to second-guess them. And a lot of times this is appropriate. But when you’re writing something that an end user is gonna need to set up, it’s not appropriate. End users don’t wan to worry about administrative details. So the challenge I accepted in my own mind was, how do I get this thing to a point where you just plug the device into a machine, and it comes up, and it just works without the user having to tell them anything.

This was actually kind of tricky and interesting, making this work. What I ended up doing was writing a little state machine inside the GPSD itself, looking at the bit stream coming off of a serial device. And I exploited the fact that all these protocols have one thing in common: they all send well-formed packets with a checksum in it. So you can tell when you’re seeing real information and when you’re seeing binary garbage. So what I ended up writing is, when a […] opens a device, it watches the stream and asks, do I see something that looks like one of those six packet types? If it doesn’t see that, it says “well, I’ll try a different baud rate”. So it cycles through all the different baud rates until it sees something that looks like a well-formed packet. It looks at the packet types and says “oh, I’m looking at one of THOSE!” [And it gloms [?]] that packet type, and then it just happily […] for the rest of time when the device is unplugged. [?]

That only solves half a problem, though. [The user would not have to specify the baud rate in and the protocol type], [but he would still have to] specify the device. Fortunately, there is […] called “hotplug”. What hotplug lets you do is it lets you attach a script to a device in such a way that whenever a device goes active, the script pops up and does something for you. So I wrote a hotplug script so that when a USB or a serial device opened up, it sends a notification to daemon that says, “hello, hello, possible GPS device here, take a look at it”.

So, the upshot is, the user… the daemon isn’t even necessarily running on a machine to start with. You plug in a device, a hotplug script wakes up and says, “hmm, possible GPS device here. If a copy of GPSD isn’t running, I should start one. And once a copy of GPSD is running, I should open its control port and say, hello, hello, possible device over here, at which point the hotplug script goes away, and GPSD says, hmm, oh, possible GPS bitstream here, I think I’ll just snip it for a while and see if it looks like a real GPS, and if it does, then I’ll automatically start publishing information on port 2947.” This, people, is called zero configuration, and it is GOOOD. Better than options is auto-configuration.

John Quarterman. As the users become less programmers and more executives, the more important it becomes to write auto-configuring programs. The more users use your program, the better for you: you’ll get more feedback, and your effort is going to pay off, whether payoff means you get paid more, or a realization you did something good because everybody using it.

If the users going to be other programmers, they have things that they want to be working on, and if they can just plug in the GPS and it works, it’s gonna be less distraction from what they’re really trying to do.

  John Quarterman talks briefly about risk management. Mostly commonsense stuff.

Eric Raymond. So! Can we get another question, or another topic? Or does he have to start talking about risk management?

John Quarterman, who was rearing to talk about risk management the whole evening (or so it seems), mentions a few basics of risk management, but does not go all out. Somehow his enthusiasm evaporates soon, or perhaps he knows the audience is less than thrilled with this topic, so he wraps up quickly. The basics he mentions are insurance, diversity, reputation.

John Quarterman. Suppose we got firewalls, intrusion detection, we got scanners for spyware, every known security facility that you can think of, and then your ISP fails. Or the ISP three hops between you and the customer using that software that you just made. What do you do? Or hurricane Katrina comes in and takes out the server that you thought was in India. Which happened to the chair of this very conference. Turns out it was in Galveston. What do you do?

All that fancy software that you got, all those fancy systems you bought from vendors is no use to you whatever. That’s where risk management comes in. [I’m gonna talk more about this tomorrow,] but there’s a whole range of things you could do, starting with, if you’re big enough to afford it, insurance. And simpler things. Diversity. Have more than one server.

So there’s a whole range of things. Redundancy, diversity, insurance, and things called reputation systems, where you know how well stuff works, how other people are doing things.

Eric Raymond. Open source all runs on reputation. It’s a reputation-based community.

Well, thank you for listening. We wish you had more questions.

Somebody from the audience reminds him that the panel is supposed to go on for another hour.

Eric Raymond. Oh! Then we’ve got sixty more minutes!

John Quarterman. Fear this.

  How did they get where they are today?

A guy from audience asks about the speakers background and how they got where they are.

Eric Raymond. Oh dear. (Laughter)

Let’s see. Well, I was going to say a lot of this for the road show tomorrow, but you ask an interesting question, which is, how I got where I am today, and yes, there’s a handle [?] here that would be interesting to high school students.

I got where I am today… there are two kinds of important truths, there are two kinds of things that really changed the world when we figured out what’s going on. One kind is really complicated and difficult, and takes amazing amounts of skull sweat to get there. Let’s say, the structure of benzene. Or structure of DNA, a much more recent example. These people had to sweat and calculate and think for years, and then finally came up with this tremendously complex insight [that changed the world]. That’s one important kind of truth.

  The other important kind of truth is one that is sitting under everyone’s nose. And I’m here today because I noticed one of those.

The obvious thing that I noticed is what I encapsulated in my now already famous quote: given a sufficiently large number of eyeballs, all bugs are shallow. What I noticed was peer review works really well with software. After I thought about it for a while I realized it shouldn’t be a surprise, because this is the way we’ve been organizing science and engineering for 400 years. We know that when people do creative work, it can’t be mechanically checked, it can only be critiqued and mistakes found by [other people looking at their work]. And human beings are fallible: they make mistakes, they screw up, they miss their assumptions. And we’ve learned, starting from the beginnings of the scientific revolution around 1600s, that the only way to do creative work and have it bogon-checked [?] routinely is to make the entire process transparent. So that not only your results, but your experiment plans are visible and available to be criticized by everybody.

But somehow in the domain of software we lost this insight, we dropped it. And the error we made in dropping it got tied into a widespread adoption of business model in which essentially what software companies were selling was their secrecy. They would make big blobs of binary software that were secret, and they would essentially you have to pay us rent on our secrecy. And because this was convenient and spawned a big and apparently healthy looking industry, everybody thought this was fine for a long time. Well, everybody except a small band of ragtag [?] hackers.

But a funny thing happened on the way to the future. It turned out that development methods that were being used for proprietary software, the ways of verifying those blobs of bits, didn’t scale up very well. So in the early 1990s there was a lot of talk about software crisis. You remember that, I’m sure (he addresses John Quarterman). People were worried that, vow, the complexity of software is outstripping our ability to debug it. And what they didn’t realize was that there was a rebel band of ragtag hackers off on the internet who actually had the answer to this problem! But the thing is that the hackers didn’t know they had the answer.

They were simply continuing a community practice that they had developed over a period of many years, because it was fun! It worked. And the thing is… I was there, John was there. We didn’t realize how powerful what we were doing actually was. We never actually thought about the fact that there was this sort of a classic mode of software engineering, in which you build big, hierarchical, elaborately structured projects, and you control the objectives, and you control the working methods, and you manage the team really closely. The thing is, all of us in what’s now the open source community, we didn’t think this was a model for good software engineering. And the same time the way we were actually working a lot of time was completely different. Transparent, loosely organized. Development teams that were basically small groups of peer-to-peer connections. And none of us thought about the fact that what we were doing contradicted what we thought we believed about software engineering. We were sleepwalking! We were unconscious!

And in that kind of situation the most powerful thing you can do is to wake up. To wake up and see things from a different angle. I got lucky. I happened to get there first. And I’m not under any illusion: I know if it hadn’t been me, it would have been somebody else. Sometime in the time frame between 1994 and 2000, the lower bound being when the internet became generally available, somebody was going to have this insight. I just happened to get there first. And the lesson is, sometimes the really important things are the obvious ones. The things you look at and think, no, it can’t be this simple.

John Quarterman. E really does equal mc2 !

Eric Raymond. So that’s sort of the most general answer to how I got here. By noticing something really obvious. But there’s something else I did afterwards, which is that… well, I exploited that insight ruthlessly. And I did it for a reason. It was because I was part of this culture of internet hackers, and I got really tired… over the last 20 years or so I got really tired of watching this too beautiful software lose all the marketing balance. As a result, the glory and yes, the money that should have been ours, ended up going to good marketing.

I decided that it annoyed me a whole lot. And because it annoyed me a whole lot, I decided to exploit the accident of having had this insight become famous in a minor way, as ruthlessly as I could. And, well, that’s the other half of the reason I’m here. Because I basically, sort of, did deliberate self-promotion for a while. Not because I need to be famous — I don’t actually like being famous very much — but it is an instrument with which I can do things on behalf of a community I care about very much.

John Quarterman. The software community needed its Lenin.

(Audience laughs)

Eric Raymond. Lenin?

John Quarterman. He was a propagandist and…

Eric Raymond. But he was evil! I don’t wanna be evil! Actually, the analogy I…

John Quarterman. Tom Paine. How about Tom Paine?

Eric Raymond. I like that better.

John Quarterman addresses the audience. If you haven’t read Thomas Paine, I highly recommend him. You may be surprised how good that guy was.

Eric Raymond. Yeah. I have started […]

But that’s a good analogy. Actually, another analogy I’ve seen that I kinda like is somebody proposed an analogy between open source movement and protestant reformation. And in that analogy Linus is Martin Luther and I am Erasmus.

So that’s at a very broad level of detail. First, I noticed something obvious that probably should have been noticed a long time before, and […] It’s nice to be brilliant and improve [the life of?] humanity, but often you don’t get to do that unless you get yourself noticed. Sad but true.

John Quarterman. I’m here because of Samuel R. Delany, John W. Campbell and Joseph Campbell.

Eric Raymond. Rock’n’roll, John! I didn’t know that!

John Quarterman asks the audience: You recognize any of those names? No?

Well, Samuel R. Delany is actually a very famous science fiction writer, and for a long time he was one of the dozens of New Wave authors. He wrote a book called Nova. There is a character in there who went to Harvard. He’s talking about why he went to Harvard because it taught him to think. It sounds silly but it’s basically true. So I went to Harvard too. And there I accidentally went to a first ever Unix users group meeting. There I met people who had got associated with open source software.

Eric Raymond. This was what, ’75?

John Quarterman. Around that time.

John W. Campbell was the editor of “Analog”, a famous science fiction magazine, it’s still around in some form. And he published the thing called “Dune” by Frank Herbert. Back then it was radical, it was different from anything I’ve ever seen. Because it wasn’t about spaceships, but about evolution, and how society was changing. It encourages you to look at the big picture. Mythology too, teaches you to take a big picture view when thinking about things.

It’s the kind of thing that Eric Raymond has been talking about, it’s hard for an average programmer to do, because programmers are detail oriented. There’s nothing wrong with that. But there need to be also people who see how details fit together.

Eric Raymond. Anyway, most programmers are — this is a generalization, but obscure [?] enough to be useful — most programmers are shadow-autistic, shadow-obsessive-compulsive. They are very introverted, very detail-oriented. I am not. I am a shadow-Tourette syndrome case, all jittery, jagged creativity.

John Quarterman. Yeah. OK. So, Joseph Campbell, some things he was talking about sounded a lot like Star Wars, a small, ragtag group of revolutionaries. Many people who wrote software that produced images of Star Wars were early Unix hackers. [I worked in a BSD project in Berkeley. My main task on BSD was documentation. I did write some of the software, a very small amount of it. ]

The actual Computer Systems Research group in Berkeley never had more than five [?] people, ever. Bill Joy was the original head. Do you know who Bill Joy is? (Apparently the audience confirms they know.) OK. One of the founders.

And Berkeley Unix had alpha and beta test cycles, which involved people at the university and all over the world, using Arpanet to communicate before the internet formally started. And there were people in Australia, Japan, University of Texas, Massachusetts, people from Utah…

Eric Raymond. To give you a sense of scale, there were fewer people in the entire Unix community worldwide than there are at this convention now.

John Quarterman. Oh yeah. This convention is bigger than that first meeting that I missed and as big as some of the Unix conventions years and years later.

But does the technique sound familiar? This is basically the same techniques that Linus used later for Linux. So, I got involved with that, and that was one of the steps along the path to open source. I also […] the helped to promote the internet a little bit.

(He speaks about his role on February 19th 1983, as the University of Texas changed over to TCP/IP from the Arpanet protocols, but I’m not sure I understood what his role was. — E.)

 

John Quarterman. The reason Eric says other people knew what he’s talking about, because what he’s talking about, the way software can be written, is exactly what we were doing with Berkeley Unix.

John Quarterman, Eric Raymond and The Tron Guy at Linucon 2005
CIMG0811 Left to right: unidentified congoer with a puppy, John Quarterman, Eric Raymond and The Tron Guy hang out at the registration at Linucon 2005

Eric Raymond. That’s right. Well, there were some differences in scale and intensity. The… Linus was a child of the internet, a cultural escapee [?]. He took that to an extreme, where the Berkeley people not quite […]

And one of the consequences is the hacker community today is a hundred times the size that my most wildly optimistic projections predicted in the 1980s. It’s startling how big it’s gotten.

John Quarterman. I’m actually not that surprised, ’cause the internet itself was doubling in size year to year for a decade. In 1983 there were thousands of people on the internet; now it’s approaching a billion. It’s not growing as fast anymore, but when you’re talking […]

Eric Raymond. And there’s going to be a time, and it’s not gonna be very long, when everybody is going to be internet-connected all the time unless they voluntarily choose not to.

(They talk about internet access in South Korea, how they are much better connected than US is. Eric Raymond says: small country, a lot better organized.)

Eric Raymond. I don’t think we’re going to lag Korea for more than 18 months.

And the reasons for that, by the way, are not technical: they’re political. It’s because we’ve got to [get rid of?] telecom companies that are getting in the way. They like being in a high margin, low-volume voice traffic business, and they know that where the world is heading is, [there are going to be?] providers of high volume, very low margin bit transport business, and the fighting that transition tooth and nail. And the problem is, they can bribe the government to hold it up, and they do that a lot.

John Quarterman. There are 40,000 lobbyists in D.C. now. And a lot of them are lobbying for telcos and for certain high profile existing [content?] providers like Disney. If you are wondering why copyright […] are getting extended over and over: Disney is one of the major lobbyists behind copyright.

So, you’ve got to realize that on one hand, what we got on the internet, the open source community, is a fabulous thing; you also have to realize, it’s constantly under threat. The telcos would like to take it back to where they charge for everything that goes through the net. And it could happen. So, it would be worth your while to pay attention to what’s going on in Congress and state legislatures.

Some of you might even consider doing political work. Most of us hacker types like that even less than Eric claims to like his license thing (I think he speaks about Eric Raymond’s dislike of GPL — E.) but someone ought to do it.

There are some people out there doing it, people like Cory Doctorow with EFF, Larry Lessig, and […] Gilmore.

Eric Raymond laughs. Oh, I thought you were talking about John Gilmore.

John Quarterman. And John Gilmore. He’s doing some very useful work.

  How government interferes with deployment of new technologies

Somebody from the audience. Are you familiar with what is going on in Texas with some wireless house bill? They are trying to make it so that municipalities could not offer wireless.

John Quarterman. Naah, I have to admit I’m not familiar with that one. I should be, though.

John Quarterman talks about mistakes the State of Texas had made before. About how they try to apply some law to ISP, which was unwarranted or unjust, and suddenly the Texas ISP Association was born to oppose this thing.

Eric Raymond. I learned a thorough lesson in the early 1990s when I founded a free community internet where I live, in Chester County, Pennsylvania. And I what I wanted to do was to give people at least a dialup access to the internet hosts, so that they could surf the web, use the PPP or whatever. And one of the things I learned as I was putting it together in the early 1990s — well, it’s something I already knew in theory but I learned it in practice: if you’re trying to get technology deployed to help people, government intervention and government funding is not your friend.

John Quarterman. Mandating technology is almost always a mistake. The most famous case of that is GOSIP. It was an alternative suite of protocols to the TCP/IP protocol of the internet. The protocols themselves were not bad, [they did?] more or less the same thing…

Eric Raymond. They were promoted by a bunch of Europeans, while most of TCP/IP was developed in Pentagon. They didn’t want to use the evil, icky, American military protocols on their network.

John Quarterman. And I know this for a fact, because I chased down so-and-so in Sweden at a conference, and asked him. He said, yes, we prefer to use something that’s not developed in US. And the way they specified the protocols, the European so-called National Delegation Standards Committees wrote aaaallll the specs before trying to do any implementation.

Eric Raymond. No field experience.

John Quarterman. And then they farmed out the implementation in Europe: this piece stack of the stack is this country, this piece of the stack is this country, each being developed separately without interaction. The US meanwhile has this thing called [GOSIP?], a technology that says, every company that sells computers or software to the government has to implement this stuff.

Eric Raymond. There is a famous hippie word for that result, and that word is clusterfuck.

You know postscript? It turned out that that funny three-way handshake they wanted to implement like […], it turned out it was impossible to implement.

John Quarterman. [Of course] they didn’t know, because they didn’t try to implement it.

This thing held back Europe in internetworking for at least eight years. TCP/IP didn’t take off there until 1991, whereas in the States it’s been going since 1983.

Eric Raymond says something about IETF. Somebody asks what is IETF.

Eric Raymond. Internet Engineering Task Force. These are the people who write the standards that make the internet go. “No kings, no presidents, no voting, just rough consensus on running code”. But… which… I think that line was out of the 1992… that could have been a manifesto for the open source community as it was evolving that same year, because Linux had been launched a year earlier.

John Quarterman. It is possible to have heterogeneous internetworking. You can have completely different operating system platforms that talk to each other through the same protocols. Different users using the same stuff. Instead what we’ve got is 90% users using one platform. One platform that is known to be highly susceptible to bugs.

Eric Raymond. Yeah, that’s gonna be the one that will get wiped off by terrorists one day.

John Quarterman. Which is what I’m leading up to.

Eric Raymond laughs.

John Quarterman. Now, what would be the simplest thing to do to avoid that eventuality?

Eric Raymond. Same thing you have in biological systems: diversity of response.

John Quarterman. And there was a conference at Harvard where somebody presented a paper on economic analysis. The usual objection here from big companies to run something other than Windows is “well, it’s so hard to run multiple systems”. In fact, the paper showed, even if only a small minority of the systems you’re running are not the same as everything else, you win. With all those considerations.

  Economics of system administrators, and other bundled goods

Eric Raymond. People are talking about the total cost of ownership and saying, well, it’s more expensive to hire Linux guys. What they don’t tell you, what those people don’t talk about, is the fact that you get away with less hardware if you are system administrators, because Windows boxes are so unstable that you can only use them as single function servers; but Linux or Unix boxes, you can pile a dozen services on them, and the machines are reliable enough to run them all. The result is, you just replaced a dozen of Windows boxes with one Linux box. And the number of administrators you have to hire scales with the number of boxes, not with the number of services. So now you only have to pay one guy, where you used to pay a dozen.

John Quarterman. And the paper didn’t even take that into account. So it’s even better than the paper says.

Eric Raymond. By the way, don’t worry about the other eleven guys. They’re gonna have jobs. And the reason that’s true is because of an interesting thing about the economics of bundled goods. Have you heard this one?

John Quarterman. No, I haven’t.

Eric Raymond. This is good. I think you’ll like this. So (he addresses the audience), what happens to the hourly wages of automechanics when the price of cars goes down?

Audience mumbles something.

Eric Raymond. The price of cars goes down, the hourly wages of auto mechanics go up. Here’s the reason why. When people buy a bundled good that has several different components, and the price of one component goes down, and they can substitute that good for something else they use, they spend more money on the rest of the components. So what people are actually buying when they buy a car? People, aside from a few collectors, don’t actually want to have cars. They want to have a good quality transportation. And components of that bundle include the car itself, the gasoline, and mechanics to service it. So when price of automobiles goes down, people’s rational incentive is to spend more money on other pieces of the bundle: the gas and mechanics.

That’s why when the price of one service goes down, because you’ve bundled them up, even if you hire just one system administrator instead of a dozen, those other 11 people are actually going to find jobs, because information processing is also a good substitute for other things, and people will pour more money into it.

And there’s some unpleasant news related to that too, which is that increasingly it’s not enough just to know how to program. Increasingly as a programmer you have to develop some kind of business knowledge or technical knowledge that makes you a domain expert in an area that’s not just writing code. So what this means for you high school students is: yeah, it’s fun to sit there and code, but it’s not enough.

John Quarterman and Eric Raymond discuss what other occupations high school students should consider going into, that may be a little more “future-proof” than programming.

John Quarterman. Accounting is gonna be less useful, because that’s gonna be outsourced soon.

Eric Raymond seems to think economic knowledge is useful to have, because not nearly enough technical people understand economics.

John Quarterman. If you are ever thinking of working for a venture funding company, I highly recommend you pick up a book called “High Tech Ventures” by Gordon Bell. He’s the guy who designed… not designed the internet, but planned and engineered it. But what he says about how high tech ventures work — the funding, what people look for in the first stages [?], this is the best textbook that venture capitalists use, and this is how they think. A lot of things that go on in startup companies, that otherwise are inexplicable, would suddenly be like “whoa, so that’s what they are talking about”.

Eric Raymond.   I have another business book to recommend. Probably THE best business book I’ve ever read: Christensen’s “The Innovator’s Dilemma”.

Oh, this is great. It’s all about why large, well-established companies who dominate their markets are doing everything right, not making serious management mistakes, routinely get dry gulched by small garage outfits, who it seems they should be able to crush by the sheer weight of money [that they could use to hire talent?]. What Christensen identified is a repeated pattern of what he called “disruptive technology”. I’m gonna tell you a disruptive technology story. And then I’m gonna talk about the pattern.

In the 1950 American electronics industry was dominated by a handful of large radio and television manufacturers, who had learned how to do beautiful and effective things with vacuum tubes. They built huge, expensive radios and televisions that were in beautiful burled oak cabinets. And everything everything went fine for them until about 1964, at which point the bottom suddenly fell out of their market, and it was all Japanese transistor radios.

Here’s how this happened. The Japanese had some technical tricks they could do with transistors that enabled them to build radios that were very small, very low power, than ran on batteries. And the sound quality of these radios was very poor, was much worse than the tube radio at that time. And because that was the case, the American electronics companies didn’t care about this technology. New transistors existed as a theoretical possibility, but they thought, well, we can keep making tube radios better, we don’t need to worry about transistors.

The electronics companies were oriented towards serving the wealthiest customers they had, people who wanted the most audio quality and the nicest veneer on their cabinets. The Japanese did something different. They noticed that there was a market that the American electronic companies weren’t serving, that they could serve with their small, crappy low-power radios, and that market was teenagers. And the reason there was a market for that was that the teenagers wanted some way to listen to rock and roll, where they wouldn’t be getting crap from their parents about it. So they needed portable radios. So the Japanese thought, gee, we could sell cheap radios to teenagers, and they started doing that. They used the revenue from selling transistor radios to teenagers to develop better manufacturing processes, better designs so that the radios got better.

Well, the tube electronics were also getting better. But, there’s a certain level of quality in the market which is as much as radio users wanted to pay for. Most radio users didn’t actually want elaborate burled oak cabinets. Most radio consumers actually can’t notice a difference between a 10 decibel noise threshold versus a 4 decibel noise threshold.

So what happened was both technologies were improving, and at the point where the transistor radios improved so they met the needs of both consumers, the bottom fell out of the tube radio market. Really, really, fast. It took about 18 months, actually, if you look at the history. Because the transistor radios were good enough, and the tube radios were good enough but they were also a lot more expensive: boom, they were gone.

This has happened over and over again. The Japanese did it again in the 1970s when they started selling subcompact cars in the United States. There were repeated instances of it in computer industry, and the pattern is always the same. You have some garage outfit that comes up with a cheaper way to do a job that an existing technology is already doing. And they’ll find some niche market that’s so small that the big players aren’t interested in it. They use the revenues to fund the development to a point where gradually, without the big members noticing, the disruptive technology gets better until it finally reaches that magic point at which it meets most consumers’ needs, and — quaaam! — the bottom drops out of the market.

Then they talk about Prius as an example of disruptive technology. Earlier hybrid cars have been, to quote Eric Raymond, “sort of weird techie demonstration projects. They sort of showed the technology was there, but they weren’t something you actually wanted to drive. But the Prius is a nice car.” So, Raymond concludes, disruptive technology doesn’t even have to be cheap and crappy (since Prius isn’t cheaper than regular cars). Sometimes it’s just doing what all the others are not.

Eric Raymond. Yeah. The key is noticing that there is a niche market that the big players aren’t exploiting, and using the revenues from that to improve the technology to a point where it’s gonna attack the main market. And, as Christensen himself recognizes, open source is executing a classic disruptive attack on the software industry. Niche markets attack […], the whole pattern’s there.

John Quarterman. Speaking about disruptive technologies, what makes the internet itself a disruptive technology? (Pauses) There are many answers to that. There’s one that I’ll probably go on about, but at least…

A guy from audience. Texas school administrators think computers are disruptive presence in school. Students can’t concentrate on the classes. So they have computers in class, but not the internet.

John Quarterman evidently had something else in mind.

John Quarterman. It’s disturbing the traditional news media. Newspapers are laying people off left and right. There are some political ramifications that are pretty worrying. Many news media is bought by big conglomerates who don’t care what news we get.

But the things is, reporters do research and you don’t. You can say on the internet you’ll find information anyway, you don’t have to have a staff of reporters like the New York Times to do research. Sure, the research reporters do is imperfect, but there’s a difference between having and not having any. The New York Times or The Economist have people they pay a regular salary that they can send out to do research. The average blogger doesn’t have a staff of reporters to do that. On the other hand, there’s far more bloggers than reporters. And you can pick bloggers you care about for your all information needs.

Eric Raymond. And one thing bloggers are superb at is after-the-fact checking. If there is an inconsistency, or bias leading to distortion of a news story, they will find it.

Somebody in the audience mentions the Dan Rather scandal.

John Quarterman. So the big difference is, you take traditional media like radio, television, it broadcasts. You are at the mercy of the big broadcasters. [He means that it is a one-sided relationship: broadcasters have a monopoly on distributing the news. You can’t share your news and opinions with many people just as easily as they do.]

Then there’s things like telephone, that doesn’t scale very well. It only lets you you to talk to other individuals one-on-one (or, at best, do conferences).

And then there is internet, where everyone can talk to everyone else at the same time. You choose what groups you want to participate in, and where you are going to get your information, not some broadcaster chooses what you’re gonna send to you. What’s more, you can produce your own content. Everybody can be consumer and a producer. They can choose which groups they want to listen to, which groups they want to send to, which is completely different ball game.

So the short version is: the reason internet is disruptive is that it is group-forming. You can form all sorts of groups to exchange information.

Eric Raymond. I thought you were focusing mainly on the flipside, which is that it’s decreasing the power of media gatekeepers.

They illustrate the point about forming groups with an example of LiveJournal communities.

Eric Raymond asks the audience: how many of you have LiveJournals?

Quite a few people raise their hands.

John Quarterman. You know what we are talking about?

A girl in the audience. Yes.

Larval Mode panel audience at Linucon 2005
CIMG0816 The audience at the Larval Mode panel

John Quarterman. Tell us about it.

The girl, who is a high school student, talks about various LiveJournal communities that formed around students’ various interests in her high school alone. “It’s ridiculous how many groups there are”, she concludes.

John Quarterman. And that never would have happened without the internet.

Eric Raymond. In the presence of sufficiently low communication costs people will form 2n communities. (He giggles.)

Jay Maynard from the audience. 2n is a lower bound. It’s a number of unique subsets to the set.

Eric Raymond. Ha ha! I know what you mean. You can have different communities formed by the same group of friends talking about different subjects.

Eric Raymond. So we are down to our last 15 minutes. Questions?

Jay Maynard, from the audience. While we are talking about disruptive technologies, I’d like to a layer of abstraction. How about Google as a disruptive technology?

Eric Raymond. Ho ho ho! Yeah yeah yeah! Speaking as somebody who does a lot of writing that’s fairly fact-intensive, Google is the best friend I ever had.

John Quarterman. I just wrote a whole book without ever going to the library. [I wrote it by looking things up on Google and ordering books from Amazon.]

Eric Raymond. Yeah, the high school students always take this for granted.

One effect of Google may be to attempt to kill small libraries and make big ones more useful. The comparative utility of a small library with a small selection is going down as a result of Google.

John Quarterman. I agree. I’ve noticed that one effect of the net is that libraries now offer internet access.

A guy in the audience. It might fill a niche for people who don’t have internet access.

Eric Raymond. OK, so that class of people is approaching zero, [is going to be] zero shortly, so that’s not an interesting projection. We have reached a point where people have browser access on their CELLPHONES, for Chrissakes!

A guy from the audience. Books don’t deteriorate like URLs do.

Eric Raymond. True.

The guy from the audience. What’s the lifespan of a book versus the lifespan of a URL?

John Quarterman. [I had a problem with a book I was writing, because some of the URLs I originally looked up were gone six months later when I was wrapping it up.] But I balanced it against the huge array of information that would be hard to compile even if I lived in the library.

The guy. I’m not fighting for books, I just want your autograph…

He hands ESR a copy of “Art of UNIX programming”.

A guy from the audience. That book, do you think it should be, like, a basis for a class in 4-year colleges?

Eric Raymond. Of course, I do! Actually, what I think this book is good for, is I think it would make an excellent adjunct text for a course in software engineering.

  The tape recording of the panel was unintelligible in many places, so none of this text is a word-for-word transcription of what was said. I tried to stick as close to the recording as possible, but in most places I had to paraphrase it. Even the places that are presented as direct speech are paraphrased. Thus, all the grammatical, syntax and style errors are mine, and not the speakers. Places where I wasn’t sure I understood a particular word or phrase correctly I marked with a question mark in brackets: [?]