oreilly.comSafari Books Online.Conferences.
Articles Radar Books  

The Parable of Umbrellas and Taxicabs

by Clay Shirky

Many companies in the P2P space are trying to figure out how to deploy P2P resources most effectively in a dynamic system. This problem is particularly acute for the companies selling distributed computation, such as Popular Power and United Devices, or the companies trying to build a general P2P framework, such as ROKU and Globus.

The first successful implementations of distributed computation like SETI@Home and distributed.net relied on non-financial incentives: The participants donated their cycles because they felt good about the project, or because they enjoyed "collaborative competition," such as distributed.net's team rankings for its encryption-cracking contests.

Ego gratification is a powerful tool, but it is a finicky one. Someone happy to donate their time to chess problems may not want to donate their cycles to designing bioengineered food. The problem that companies who rely on distributed resources face is how to get people to give time to commercial projects.

The general solution to this problem seems to be "if you give people an incentive to do something, they will respond, so find the right incentive." If ego gratification is an effective way to get people to donate resources, the trick is in finding other kinds of incentives to replace ego gratification for commercial services.

The obvious all-purpose incentive is money (or at least some sort of currency), and several systems are being designed with the idea that paying users is the best way convince them to provide resources.

Related Articles:

Why Micropayments Don't Work

Slash dot user comments on this article

Peers not Pareto

P2P futures like deregulated electric?

In Praise of Freeloaders

Mojo Nation Responds

What is P2P, and What's Not

More from the P2P DevCenter Previous Features

This solution may not work well, however, for things like cycles and disk space, because there are two kinds of resources that can be deployed in a P2P system, and they have very different characteristics -- a difference that can best be illustrated by The Parable of the Umbrellas and Taxis.

Umbrellas and taxis

Anyone who's spent any time in New York City knows that when it begins to rain, two things happen immediately: It becomes easier to buy an umbrella and it becomes harder to hail a cab. As soon as the first few drops fall, people appear on the street selling cheap umbrellas, while a lucky few pedestrians occupy all the available cabs.

Why does an increase in demand produce opposite effects on supply -- more available umbrellas and fewer available taxis? The answer is the nature of the resources themselves. Umbrellas are small and inexpensive to store, so it's easy to take them out when it's raining and put them back when the rain stops. Additional umbrellas can be deployed in response to demand.

Taxis, on the other hand, are large and expensive to store. In addition, taxis have all sorts of up-front costs: registration for a yellow cab or car service, license for the driver, local regulations, the cost of an automobile. These up-front costs can be high or low, but whatever they are, they set some maximum number of cabs available in the city on any given day. And if it starts raining, too bad: Additional taxis cannot be deployed in response to peak demand. Every city has a total number of cabs which represents a compromise between the number of potential riders in sun vs. rain, or 4 a.m. vs. 4 p.m., or April vs. August.

Not all P2P resources are created equal

Some of the resources in a P2P system are umbrellas. Some are taxis.

Umbrella resources are what make things like file-sharing systems so powerful. If you decide at 3:17 a.m. that you must listen to Golden Earring's "Radar Love" or you will surely die, it's Napster to the rescue: Your demand produces extra supply.

If, however, you decide that you must have a faster chip, you're out of luck. For that, you have to make a trip to the store. You could use someone else's cycles, of course, but you can't increase the total number of cycles in your system: Your demand does not produce extra supply.

This has ramifications for getting users to provide additional resources to a P2P system. If United Devices wants to use more cycles than you have, you can't instantly produce more chip. Ditto bandwidth and disk space. You have what you have, and you can't deploy any more than that in the short term.

Thresholds and gaps

Since the increased demand during a rainstorm doesn't create more taxis on the spot, the way to have more taxis when it rains is to have more taxis overall -- to change the overall capacity of the system. One could make it cheaper to buy a taxi, raise the rates drivers could charge, or relax the legal restrictions on the business, and any of these things would increase capacity.

What doesn't increase taxi capacity is momentarily increased demand, at least not in well-off economies. It's easy to see how more cab service could become available -- drivers of ordinary cars could simply offer their services to damp pedestrians in times of increased demand. Why don't they do this?

The answer is that the pedestrians aren't willing to pay enough to make it worth the drivers' time. There are many sorts of obstacles here -- from the time it would take to haggle, to the laws regulating such a thing -- but all these obstacles add up to a higher cost than the pedestrian is willing to pay. A passenger would pay more to get in a cab when it's raining than when it's dry, but not enough more to change the behavior of private drivers.

One particular effect here deserves mention: Unlike cab drivers, whose vehicles are deployed for use by others, drivers of private cars are presumably driving those cars for some reason other than picking up random passengers. Part of the threshold that keeps them from picking up riders for a fee is that they are unwilling to give up their own use of their cars for the period the passenger wants to use it.

Variable use equals variable value

With that in mind, consider the case of PC users in a distributed computing system like Popular Power. Assume the average computer chassis costs $1,000. Assume also that the average user replaces their computer every two and a half years. This means $1,000 buys roughly 20,000 hours of device use prior to replacement.

Now imagine you owned such a machine and were using it to play Quake, but Popular Power wanted to use it to design flu vaccines. To compensate you for an hour of your computing time, Popular Power should be willing to offer you a nickel, which is to say 1/20,000th of $1,000, the cost of your device for that hour.

Would you be willing to give up an hour of playing Quake (or working on a spreadsheet, or chatting with your friends) for a nickel? No. And yet, once the cost crosses the nickel threshold, Popular Power is spending enough, pro-rata, to buy their own box.

The secret to projects like Popular Power, in other words, is to use the gap between the permanence of your computer and the variability of your use of it. Of the 20,000 hours of so you will own your computer, between 1,000 and 5,000 of those hours are likely to be highly valuable to you -- too valuable for Popular Power to be able to get you to give them up for any amount of money they are willing to pay.

Successful P2P programs recognize the implicit difference between the times you want to use your PC and the times you don't. Napster gives users total control of when they want to be logged in, and allows the owner of any PC to unilaterally terminate any uploads that annoy them. Popular Power runs in the background, only actively using cycles when the user is not.

Incentives that match value

Cycles, disk space, and bandwidth are like taxis: They are resources that get provisioned up front and are used variably from then on. The way to get more such resources within a P2P system is to change the up-front costs -- not the ongoing costs -- since it is the up-front costs that determine the ceiling on available resources.

There are several sorts of up-front incentives that could raise this ceiling:

  • PeoplePC could take $100 off the price of your computer in return for your spare cycles.
  • AudioGalaxy could pay for the delta between 384- and 768-Kbps DSL in return for running the AudioGalaxy client in the background.
  • United Devices could pay your electricity bill in return for leaving your machine on 24/7.

Those kinds of incentives match the way resources are currently deployed, and those incentives will have far more effect on the resources available to P2P systems than simply telling you that there's momentary demand for more resources than you have.

Futures markets and big winners

Because these resources need to be provisioned in large chunks when the machines are bought, and because users don't want to spend their time and effort putting spot prices on unused cycles, the markets that form around P2P resources are not likely to be real-time micromarkets but futures macromarkets. By providing up-front incentives, or ongoing incentives that don't need to be re-priced (a donation to a charity, offsetting the cost of electricity or bandwidth), companies that have access to distributed computing resources are likely to be able to create and maintain vast pools of untapped computing power.

As long as end users aren't required to give up their use of the PC during the 1,000 to 5,000 hours they need it, they will prefer seeing the remaining 15,000 hours used in a simple way they approve of, rather than spending the time working out the bidding between companies paying pennies an hour at best. Evenness of reward and lowered mental transaction costs are a big incentive to adopt a "set it and forget it" attitude.

Indeed, the price fluctuations and market are likely to be at the other end of the scale, on a futures market for vast amounts of bandwidth. If a company wants 100,000 hours of computing time on the day they close their quarterly books, they may be willing to pay more for that than simply getting any 100,000 hours spread over three months.

This suggests a futures market dealing in massive quantities of computing power, a market participated in by a small group of companies that can guarantee delivery of certain minimums on certain dates, or within certain time periods. The price of a bushel of wheat is a commodity, a price not set or affected by individual wheat producers (without operating a cartel), so the price fluctuation is set by the largest consumers. No one goes onto the futures market to buy guaranteed future delivery of a single barrel of oil or bushel of wheat -- the price is set by buying and selling in bulk.

Far from being an atomized many-to-many market of buyers, aggregators, and sellers, distributed computing will likely be a "butterfly" market with many providers of spare cycles, many consumers of cycles, and a very few aggregators, all of whom are pursuing network effects and massive economies of scale.

The likeliest winners are the companies or consortia that have the most open framework and the most installed clients, because once one or a few leaders emerge, they will be in a better position to create such a futures market than hundreds of also-ran competitors. (Of particular note here is Microsoft, who has access to more desktops than everyone else put together. A real P2P framework, run by Microsoft, could become the market leader in selling aggregated computing power.)

Many people in the P2P world are talking about general P2P frameworks for sharing any and all computing resources, and this language makes it seem like all resources are equally fungible. In fact, the vendors of umbrellas are operating under conditions very different from the operators of taxis. The companies aggregating and reselling the resources that are allocated up front and in big chunks, will likely face brutal competition between an ever-smaller group of ever-larger players. The economics of this part of the business so favor economies of scale that within 12 months, even as the rest of the P2P infrastructure is developing, the winners in the world of distributed computation will be anointed.

Clay Shirky writes about the Internet and teaches at NYU's Interactive Telecommunications Program. He publishes a mailing list on Networks, Economics, and Culture at shirky.com/nec.html.

Discuss this article in the O'Reilly Network General Forum.

Return to the P2P DevCenter.


P2P Weblogs

Richard Koman Richard Koman's Weblog
Supreme Court Decides Unanimously Against Grokster
Updating as we go. Supremes have ruled 9-0 in favor of the studios in MGM v Grokster. But does the decision have wider import? Is it a death knell for tech? It's starting to look like the answer is no. (Jun 27, 2005)

> More from O'Reilly Developer Weblogs

More Weblogs
FolderShare remote computer search: better privacy than Google Desktop? [Sid Steward]

Data Condoms: Solutions for Private, Remote Search Indexes [Sid Steward]

Behold! Google the darknet/p2p search engine! [Sid Steward]

Open Source & The Fallacy Of Composition [Spencer Critchley]