Following a longstanding HotOS tradition, the participants got together after dinner for a session of uninhibited discussions and expressions of opinion. Since it was a pleasant moonlit evening, people gathered on the lawn outside the HotOS hospitality suite, where an ample supply of libations was close at hand to liven the proceedings. Jeff Chase moderated the discussion.
Dave Patterson began by observing that he had come to HotOS to learn what was hot in operating systems, but was disappointed to see the community still fixated on performance even though it is no longer a problem. Ignoring Peter Chen's observation that there were papers on other issues like power management at the workshop, he observed that academic communities tend to get too stable in many fields, and just keep publishing and praising. From his perspective, the OS community seems in danger of settling into this comfortable middle age.
Not willing to let these remarks go unchallenged, Jeff Chase suggested that performance is the only thing we know how to measure and that is why the OS community focuses on it. He reminded Patterson of his own quote, ``For better or worse, benchmarks shape a field.'' Patterson admitted to having made the statement, and said that in this case it was for the worse -- at which the audience let out a collective groan. Jon Howell pointed out that performance benchmarks make a system that reboots quickly appear as good as a reliable system; Stefan Savage ended this avenue of discussion by saying it was too embarrassing for the community to have to admit to frequent reboots.
Continuing his role as rabble-rouser, Patterson observed that research labs are focusing on engineering these days, and rely on academics to do research. Unfortunately, the latter are often focused on near-term development, sometimes on topics that are over 20 years old. He urged the OS community to shift its focus to problems that are crying out for attention such as maintainability, usability, scalability and availability. Just because these are hard problems, they shouldn't be ignored. He pointed to fields like psychology, where researchers work on the human brain -- that's hard, but they do it anyway. The OS community should stop complaining, end the focus on easy stuff like performance, and get on with doing the hard but important research.
By now, the crowd was provoked. Robert Haas pointed out that performance continues to be important because new features are only good if they are feasible. Stefan Savage added that we are not the OS community, but the performance community. As evidence that the community addressed issues other than performance, Dickon Reed mentioned that the Nemesis operating system addressed QoS. But Patterson dismissed QoS as merely an extension of performance. Responding to this cavalier dismissal, Savage pointed out that reliability could also be reduced to a performance metric -- it is merely the time before you have hit the reset button. Armando Fox and Jeff Mogul, in different ways, responded to Patterson's claim that academics were doing engineering rather than science. Fox's position was that as long as academics are working on problems that matter to real-world people, who cares what label you attach to the work. Mogul's position was that we are engineers, and claiming that we are scientists is bogus.
Peter Chen tried to steer the debate along a less controversial path by suggesting that the metric for relevance should be the number of people who care about what one is doing. By that metric, he observed that performance does indeed rank low. Savage refused to be persuaded by this argument: he observed that the main complaint about the Web was that its services were too slow. Mitchell Tsai also disagreed, but for a different reason: he found that users were concerned about ease of setup, time for reboot, time for installation, and so on. These are performance metrics that determine how much of a user's time the system wastes, and people care very much about performance in this broader sense. Mogul voiced agreement, suggesting that a good metric would be to have a thousand grandmothers install Windows98 and see how many of them succeed by the next day. Patterson, to much cheering from the crowd, suggested that a better metric would be to use a thousand Microsoft executives rather than grandmothers.
A more fundamental rebuttal of Chen's viewpoint was offered by Robert Haas. He refused to accept the premise that a sign of good research is how many people on one's block care about the results. He reminded the audience that we were discussing research, not development. Chen countered this by saying that good research had to matter eventually, even if not immediately. Patterson asked what research done 20 years ago mattered to people today; can we explain it to them and will they care. Haas persisted that utility should not be required of research -- one should be free to try out wild new ideas. Patterson retorted that the OS community is not full of wild ideas -- just performance. Jeff Chase asked the audience what we were doing 10-15 years ago, to which there were various responses: file systems, performance, kernel organization, GUIs. Chase then pointed out the paradox that we all think we are researching OS's, yet 90++ percent of us are running Windows98, 99% or more of us have never seen its source, and most of us don't work for Microsoft!
Mitchell Tsai then posed this rhetorical question: if the OS community is so good, why didn't it predict the Web? Haas responded that we are not God, and Karin Petersen followed by observing that the Web did not arise out of the blue. Patterson asked what the OS community contributed to the Web. Tsai suggested TCP, but Patterson dismissed that saying that it was hard to think of Vint Cerf as an OS guy; Vint thought he was doing networking, and TCP is more about availability than performance. Tom Kroeger suggested that security and reliability were two areas through which the OS community could be relevant to the Web.
Patterson then turned to Satya and observed, amidst much laughter from the crowd, that he had been around a long time and asked what he saw new in the OS community. Satya returned the compliment by noting that although his hair was a lot grayer now, he hadn't lost as much of it as Patterson had over the last 20 years. On a more serious note, he pointed out that the relatively new field of mobile computing is at the forefront of OS research. The hostility of a typical mobile computing environment generates many new problems besides performance. Patterson asked if he was referring to the networking aspect, but Satya replied that it was more fundamental. Early attempts at coping with the challenges of networking have yielded ideas such as distillation of data, and degraded computation for applications like speech recognition. These are very different concepts from what one is used to on a desktop. Armando Fox suggested that many mobile computing folks would agree that good infrastructure support is the key, resulting in the same kinds of robustness and scalability problems that Patterson had alluded to earlier. Satya disagreed with such a narrow characterization of mobile computing. In his view, mobile computing demands a different way of looking at computation -- an acceptable answer now is preferable to a precise answer later. This shift in viewpoint is a fertile source of interesting problems.
Rich Draves then turned the discussion to the issue of software reliability. He pointed out that software engineering issues like reliability and maintainability are ignored in the toy systems that researchers build. As a result such systems do not scale. He wondered how we could avoid this syndrome. Patterson offered a ray of hope by observing that the OS community has some of the smartest people, and that not all communities have these people. But he agreed with the seriousness of the problem, quoting John Ousterhout as saying that software was a big catastrophe, and that no one could have predicted things could possibly be this bad. Draves concurred, saying that software seems to be the Achilles heel of civilization. Patterson agreed with this characterization and added that it was perhaps we are so smart. Chase suggested the alternative possibility that perhaps this was really a hard problem. Patterson was skeptical of this, saying that after 30-40 years and all those PhDs, one would expect better. He suggested that perhaps they only picked the low-hanging fruit, and that's the problem. This suggests the need for more improvements in areas that are harder but offer bigger rewards.
Dickon Reed offered the opinion that reliability isn't really a problem: Linux works and hardly crashes even though applications may crash. Jeff Chase added that his NT box doesn't crash either, if no applications are run on it. Satya offered the suggestion that perhaps software is in such a mess because it seems so easy. Patterson followed this line of reasoning by observing that hardware culture is to test exhaustively. The ``compile'' step is too expensive, so repeated and thorough testing is the practice. Perhaps software would work better if it cost as much to develop as hardware. Chase characterized this point of view as saying that the compiler people had done too good a job, and Rich Draves groaned in dismay at this answer. Patterson challenged Chase to prove him wrong by identifying the thing that should be improved to make software quality routine.
Stimulated by this challenge, a chorus of voices offered suggestions. Stefan Savage suggested automated checks for memory accesses. Satya suggested restricting programmers to one compile a day so that they would think harder about the code they write. Richard Draves offered the idea of continuous sanity checks of key data structures, as used in some telephone switching software. Armando Fox suggested better support for handling out-of-band events. Patterson himself suggested better programming languages. But most of these suggestions were soon shot down. Savage reminded Patterson that at a recent OSDI keynote speech he had claimed programming language research isn't real because it is not quantifiable. Srini Seshan disagreed with Satya, saying that limiting compiles isn't the answer; instead, what is needed is the equivalent of simulation in software, which is debugging.
Sensing an opportune moment to convey a key message, Dave Patterson urged the audience to become more actively involved in ensuring support for computer science research in Washington, D.C. He pointed out that both DARPA and NSF are in critical need of program managers. Further, there is a once in 20-year opportunity to substantially increase research funding, but this requires much more active participation by the research community. Senators and Congressmen need to hear from their constituents that support for information technology is important. He urged the audience to be active in this arena, and to call or write to their government representatives.
As the discussion languished, Jeff Chase observed that the proceedings weren't outrageous enough. David Ingram stepped in to offer the outrageous opinion that all software should be required to have open source code, and to publish all interfaces. But the crowd failed to be roused by this baiting. Stefan Savage then tried a new tack. He reminded everyone that at HotOS-4 in 1993, Brian Bershad had bet Jeff Mogul a bottle of Cabernet Sauvignon that in four years time Sun would no longer be in the OS business. At HotOS-6 in 1997, he had graciously admitted defeat. But Bershad and Mogul had made a second bet at that workshop, that by Dec 31 2001, DEC would no longer be in business. Savage accused Mogul of trying to weasel out of the bet by claiming that DEC still existed because it was a wholly-owned subsidiary of Compaq. He appealed to the audience to judge whether this was fair. Mogul, however, insisted that the exact wording of the bet be examined. Fortunately, someone had a copy of the scribe record of the wording of the bet, and Dave Patterson was charged with verifying it. There were numerous jeers, boos and hisses as Mogul defended his position that DEC was still alive, mentioning for example that he still received DEC's contributions to his pension plan. Savage, on the other hand, dismissed these arguments as merely appealing to the letter of law, and urged the audience to vote their consciences.
In the midst of these appeals for votes came the shocking news that top-rated Duke had lost the NCAA basketball championship to UConn. After a brief break for drink refills, the audience gathered on the lawn again to hear closing arguments on the Bershad-Mogul bet and to vote. A show of hands supported Mogul by a modest margin, and Savage congratulated him on his victory. But Mogul graciously suggested that the person who deserved the prize was Savage himself for putting up such a good fight on behalf of his advisor, who hadn't even shown up!
The rest of the session was anti-climactic. There were numerous attempts to rouse the energy of the crowd, but to no avail. In the waning moments of the evening, a number of wild ideas were tossed out, but it was clear that the evening was over. Jeff Chase declared the Outrageous Opinions session officially closed, and people headed back to their rooms.