Chat as a meeting tool isn't a universally good idea, of course. Every successful use of social software has environmental factors working in its favor.
First and foremost, the attendees were tech-savvy people who travel with WiFi-capable laptops and think about novel uses of social software, so they were inclined to want to use something like ARSC, even if only as an experiment. There was no resistance to trying something so odd and unproven, as there might be in less-techie groups.
The group was also self-motivated. Because their attendance was optional, they largely stayed on-topic. One can easily imagine that in a meeting where attendance is passive and forced ("The boss wants everyone in the conference room at 5:45") the contents of the chat would be much more tangential (to say nothing of libelous). Since most parliamentary rules, whether formal or informal, begin with the premise that only one person can speak at once, and then arrange elaborate rules for determining who can speak when, the presence of an alternate channel could severely disrupt highly-structured meetings, such as client conferences or legal negotiations. Whether this would be a bug or a feature depends on your point of view.
The goals of the meeting were in synch with the experience the chat room offered. We were not trying to forge a working group, get to consensus, or even converge on a small set of ideas. Indeed, the goals of the meeting were explicitly divergent, trying to uncover and share as much new material as possible in a short period. The chat room aided this goal admirably.
The scale of the meeting also worked in our favor. The group was large enough that sitting around a table with a laptop open wasn't rude or disruptive, but small enough that everyone could use a single chat room. At one point during the Saturday session, we broke into small groups to brainstorm around specific problems, and though there was no explicit request to do so, every single member of the group shut their laptop screens for two hours. Groups of six are small enough that all the members can feel engaged with the group, and the chat would have been much less useful and much more rude in that setting.
On the other hand, whenever things got really active on the chat channel (we averaged about four posts a minute, but it sometimes spiked to 10 or so), people complained about the lack of threading, suggesting that 30 was at or near an upper limit for participation.
There were also some more technical or formal aspects of the meeting that worked in our favor.
The plasma screen showing the Display view was surprisingly important. We had not announced the WiFi network or chat channel in advance, and we had no idea how many people would bring WiFi-capable laptops. (As it turned out, most did.) The plasma screen was there to share the chat room's contents with the disconnected members. However, the screen also added an aspect of social control -- because anything said in the chat room was displayed openly, it helped keep the conversation on-topic. Curiously, this seemed to be true even though most of the room was reading the contents of the chat on their laptop screens. The plasma screen created a public feeling without actually exposing the contents to a "public" different from the attendees.
During a brief post-mortem, several users reported using the plasma screen for chunking, so that they could mainly pay attention to the speaker, but flash their eyes to the screen occasionally to see what was in the chat room, taking advantage of the fact that most people read much faster than most speakers talk. (Viz. the horror of the speaker who puts up a PowerPoint page, and then reads each point.)
There were two bits of organizational structure that also helped shape the meeting. The first was our adoption of Jerry Michalski's marvelous "Red Card/Green Card" system, where participants were given a set of colored cards about 20 cm square in three colors, red, green, and gray. The cards were used to make explicit but non-verbal commentary on what was being said at the time. A green card indicates strong assent, red strong dissent, and gray confusion.
In an earlier experiment with ARSC, Greg added virtual cards; users could click on red or green icons and have those added to the chat. This proved unsatisfying, and for this meeting we went back to the use of physical cards. The use of the cards to indicate non-verbal and emotive reactions seemed to provide a nice balance with the verbal and less emotive written material. At one point, we spent half an hour in conversation with the only rule being "No talking." The entire room was chatting for 30 minutes, and even in that situation, people would physically wave green cards whenever anyone posted anything particularly worthy in the chat room.
While the no-talking experiment was interesting, it was not particularly useful. One of the participants whose work was being discussed in the chat (he had just finished talking when we entered the no-talking period) reported missing the actual verbal feedback from colleagues. The chat comments made about his ideas, while cogent, lacked the emotional resonance that makes face-to-face meetings work. By enforcing the no-talking rule, we had re-created some of the disadvantages of virtual meetings in a real room.
The other bit of organizational structure was borrowed from Elliott Maxwell and the Aspen Institute, where participants wanting to speak would turn their name cards vertically, thus putting comments in a queue. This was frustrating for many of the participants, who had to wait several minutes to react to something. This also severely altered interrupt logic. (At several points, people gave up their turn to speak, saying "the moment has passed.") Despite the frustration it caused, this system kept us uncovering new material as opposed to going down rat holes, and it made the chat room an attractive alternative for saying something immediate.
Though we found ARSC to be a useful addition to the meeting, there was an unusually good fit between the tool and the environment. For every favorable bit of context listed above, there is a situation where an in-room chat channel would be inappropriate. Meetings where the attention to the speaker needs to be more total would be problematic, as would situations where a majority of the audience is either not connected or uncomfortable with chat.
Even in this group, not everyone had a laptop, and for those people, the chat contents were simply a second channel of information that they could observe but not affect. Absolute ubiquity of the necessary hardware is some way off for even tech-savvy groups, and several years away, at least, for the average group. Any meeting wanting to implement a system like this will have to take steps to make the chat optional, or to provide the necessary hardware where it is lacking.
It may also be that increasing phone/PDA fusion will actually reduce the number of laptops present at meetings. Using social tools at events where phones are the personal device of choice will require significant additional thought around the issues of small screens, thumb keyboards, and other ergonomic issues.
In-room chat is unlikely to be useful for small groups (fewer than a dozen, at a guess), and its usefulness for groups larger than 30 may also be marginal (though in that case, ways of providing multiple chat rooms may be helpful).
Given that the most profound effects of the chat were in changing interrupt logic, many of the downsides came from the loss of interruption as a tool. As annoying as interruptions may be, they help keep people honest, by placing a premium on brevity, and on not spinning castles in the air. Without interruption, the speaker loses an important feedback mechanism, and may tend towards long-winded and unsupported claims. At the very least, the use of in-room chat puts a premium on strong moderation of some sort, to make up for the structural loss of interruption.
Perhaps most importantly, it will almost certainly be unhelpful for groups that need to function as a team. Because the two-track structure encourages a maximum number of new and tangential items being placed together, it would probably be actively destructive for groups where consensus was a goal. As Steven Johnson has noted about the event, the chat room moved most of the humor from real world interjections to network ones, which preserved the humor but suppressed the laughter. (Most of then time when people write "lol," they aren't.) Though this helps on the "interrupt logic" front, it also detracts from building group cohesion.