Decentralized Systems Aren't

Below the fold is the text of a talk I gave to Berkeley's Information Systems Seminar exploring the history of attempts to build decentralized systems and why so many of them end up centralized.

NFTs and Web Archiving

One of the earliest observations of the behavior of the Web at scale was "link rot". There were a lot of 404s, broken links. Research showed that the half-life of Web pages was alarmingly short. Even in 1996 this problem was obvious enough for Brewster Kahle to found the Internet Archive to address it. From the Wikipedia entry for Link Rot:

A 2003 study found that on the Web, about one link out of every 200 broke each week,^[1] suggesting a half-life of 138 weeks. This rate was largely confirmed by a 2016–2017 study of links in Yahoo! Directory (which had stopped updating in 2014 after 21 years of development) that found the half-life of the directory's links to be two years.^[2]

One might have thought that academic journals were a relatively stable part of the Web, but research showed that their references decayed too, just somewhat less rapidly. A 2013 study found a half-life of 9.3 years. See my 2015 post The Evanescent Web.

I expect you have noticed the latest outbreak of blockchain-enabled insanity, Non-Fungible Tokens (NFTs). Someone "paying $69M for a JPEG" or $560K for a New York Times column attracted a lot of attention. Follow me below the fold for the connection between NFTs, "link rot" and Web archiving.

Principles For The Decentralized Web

A week ago yesterday the Internet Archive launched both a portal for the Decentralized Web (DWeb) at https://getdweb.net/, designed by a team led by Iryna Nezhynska of Jolocom, and a set of principles for the Decentralized Web, developed with much community input by a team led by Mai Ishikawa Sutton and John Ryan.

Nezhynska led a tour of the new website and the thinking behind its design, including its accessibility features. It looks very polished; how well it functions as a hub for the DWeb community only time will tell.

Brewster Kahle introduced the meeting by stressing that, as I have written many times, if the DWeb is successful it will be attacked by those who have profited massively from the centralized Web. The community needs to prepare for technical, financial and PR attacks.

Below the fold I look at how the principles might defend against some of these attacks.

Moxie Marlinspike On Decentralization

The Ecosystem Is Moving: Challenges For Distributed And Decentralized Technology is a talk by Moxie Marlinspike that anyone interested in the movement to re-decentralize the Internet should watch and think about. Marlinspike concludes "I'm not entirely optimistic about the future of decentralized systems, but I'd also love to be proven wrong".

I spent nearly two decades building and operating in production the LOCKSS system, a small-ish system that was intended, but never quite managed, to be completely decentralized. I agree with Marlinspike's conclusion, and have been writing with this attitude at least 2014's Economies Of Scale In Peer-to-Peer Networks. It is always comforting to find someone coming to the same conclusion via a completely different route, as with scalability expert Todd Hoff in 2018 and now Moxie Marlinspike based on his experience building the Signal encrypted messaging system. Below the fold I contrast his reasons for skepticism with mine.

Proof-of-Stake In Practice

At the most abstract level, the work of Eric Budish, Raphael Auer, Joshua Gans and Neil Gandal is obvious. A blockchain is secure only if the value to be gained by an attack is less than the cost of mounting it. These papers all assume that actors are "economically rational", driven by the immediate monetary bottom line, but this isn't always true in the real world. As I wrote when commenting on Gans and Gandal:

As we see with Bitcoin's Lightning Network, true members of the cryptocurrency cult are not concerned that the foregone interest on capital they devote to making the system work is vastly greater than the fees they receive for doing so. The reason is that, as David Gerard writes, they believe that "number go up". In other words, they are convinced that the finite supply of their favorite coin guarantees that its value will in the future "go to the moon", providing capital gains that vastly outweigh the foregone interest.

Follow me below the fold for a discussion of a recent attack on a Proof-of-Stake blockchain that wasn't motivated by the immediate monetary bottom line.

The Data Isn't Yours (updated)

Most discussions of Internet privacy, for example Jaron Lanier Fixes the Internet, systematically elide the distinction between "my data" and "data about me". In doing so they systematically exaggerate the value of "my data".

The typical interaction that generates data about an Internet user involves two parties, a client and a server. Both parties know what happened (a link was clicked, a purchase was made, ...). This isn't "my data", it is data shared between the client ("me") and the server. The difference is that the server can aggregate the data from many interactions and, by doing so, create something sufficiently valuable that others will pay for it. The client ("my data") cannot.

Below the fold, an update.

Personal Pods and Fatcat

Sir Tim Berners-Lee's Solid project envisages a decentralized Web in which people control their own data stored in personal "pods":

The basic idea of Solid is that each person would own a Web domain, the "host" part of a set of URLs that they control. These URLs would be served by a "pod", a Web server controlled by the user that implemented a whole set of Web API standards, including authentication and authorization. Browser-side apps would interact with these pods, allowing the user to:

• Export a machine-readable profile describing the pod and its capabilities.
• Write content for the pod.
• Control others access to the content of the pod.

Pods would have inboxes to receive notifications from other pods. So that, for example, if Alice writes a document and Bob writes a comment in his pod that links to it in Alice's pod, a notification appears in the inbox of Alice's pod announcing that event. Alice can then link from the document in her pod to Bob's comment in his pod. In this way, users are in control of their content which, if access is allowed, can be used by Web apps elsewhere.

In his Paul Evan Peters Award Lecture, my friend Herbert Van de Sompel applied this concept to scholarly communication, envisaging a world in which access, for both humans and programs, to all the artifacts of research would be greatly enhanced.

In Herbert's vision, institutions would host their researchers "research pods", which would be part of their personal domain but would have extensions specific to scholarly communication, such as automatic archiving upon publication.

Follow me below the fold for an update to my take on the practical possibilities of Herbert's vision.

Irina Bolychevsky on Solid

Although I'm an enthusiast for the idea of a decentralized Web, I've been consistently skeptical that the products proposed to implement it have viable businesses. Two months ago, in How solid is Tim’s plan to redecentralize the web?, Irina Bolychevsky (@redecentralize founder and self-described "product person") made related points. Below the fold, some commentary.

What Does The Decentralized Web Need?

In, among others, It Isn't About The Technology, Decentralized Web Summit2018: Quick Takes and Special Report on Decentralizing the Internet I've been skeptical at considerable length about the prospect of a decentralized Web. I would really like the decentralized Web to succeed, so I admit I'm biased, just pessimistic.

I was asked to summarize what would be needed for success apart from working technology (which we pretty much have)? My answer was four things:

• A sustainable business model
• Anti-trust enforcement
• The killer app
• A way to remove content

Below the fold, I try to explain of each of them at more readable length.

Decentralized Web Summit 2018: Quick Takes

Last week I attended the main two days of the 2018 Decentralized Web Summit put on by the Internet Archive at the San Francisco Mint. I had many good conversations with interesting people, but it didn't change the overall view I've written about in the past. There were a lot of parallel sessions, so I only got a partial view, and the acoustics of the Mint are TERRIBLE for someone my age, so I may have missed parts even of the sessions I was in. Below the fold, some initial reactions.

Amazon's Margins Again

AMZN operating margins

I've been pointing out that economies of scale allow for the astonishing margins Amazon enjoys on S3, and the rest of AWS, for six years. Now, This is the Amazon everyone should have feared — and it has nothing to do with its retail business by Jason Del Rey and Rani Molla documents AWS' margins in this table.

Amazon’s $52.9 billion of revenue in the second quarter of the year came in a tad below what Wall Street analysts expected — and that doesn’t matter whatsoever.

That’s because the massive online retailer once again posted its largest quarterly profit in history — $2.5 billion for the quarter — on the back of two businesses that were afterthoughts just a few years ago: Amazon Web Services, its cloud computing unit, as well as its fast-growing advertising business.

Below the fold, I discuss one of the implications of these amazing margins.

DINO and IINO

One of the things that I, as an observer of the blockchain scene, find fascinating is how the various heists illuminate the deficiencies of actual, as opposed to the Platonic ideal, blockchain-based systems.

I've been writing for more than 4 years that, at scale, blockchains are DINO (Decentralized In Name Only) because irresistible economies of scale drive centralization. Now, a heist illuminates that, in practice, "smart contracts" such as those on the Ethereum blockchain (which is DINO) are also IINO (Immutable In Name Only). Follow me below the fold for the explanation.

Special Report on Decentralizing the Internet (Updated)

The Economist's June 30th issue features a special report from Ludwig Siegele entitled How to fix what has gone wrong with the internet consisting of the following articles:

• How to fix what has gone wrong with the internet
• The story of the internet is all about layers
• Blockchain technology may offer a way to re-decentralise the internet
• China has the world’s most centralised internet system
• How regulators can prevent excessive concentration online
• There is no single solution to making the internet more decentralised

I really like the way The Economist occasionally allows its writers to address a topic at length. Siegele provides a good overview of what has gone wrong and the competing views of how to fix it. Below the fold, my overall critique, and commentary on some of the articles.

The "Grand Challenges" of Curation and Preservation

I'm preparing for a meeting next week at the MIT Library on the "Grand Challenges" of digital curation and preservation. MIT, and in particular their library and press, have a commendable tradition of openness, so I've decided to post my input rather than submit it privately. My version of the challenges is below the fold.

Magical Thinking At The New York Times

Steven Johnson's Beyond The Bitcoin Bubble in the New York Times Magazine is a 9000-word explanation of how the blockchain can decentralize the Internet that appeared 5 days after my It Isn't About The Technology. Which is a good thing, because otherwise my post would have had to be much longer to address his tome. Follow me below the fold for the part I would have had to add to it.

Herbert Van de Sompel's Paul Evan Peters Award Lecture

In It Isn't About The Technology, I wrote about my friend Herbert Van de Sompel's richly-deserved Paul Evan Peters award lecture entitled Scholarly Communication: Deconstruct and Decentralize?, but only in the context of the push to "decentralize the Web". I believe Herbert's goal for this lecture was to spark discussion. In that spirit, below the fold, I have some questions about Herbert's vision of a future decentralized system for scholarly communications built on existing Web protocols. They aren't about the technology but about how it would actually operate.

Web Advertising and the Shark, revisited (and updated)

There's a lot to add to Has Web Advertising Jumped The Shark? (which is a violation of  Betteridge's Law). Follow me below the fold for some of it.

Not Really Decentralized After All

Here are two more examples of the phenomenon that I've been writing about ever since Economies of Scale in Peer-to-Peer Networks more than three years ago, centralized systems built on decentralized infrastructure in ways that nullify the advantages of decentralization:

• Khadim Shubber's The bitcoin drugs trade is highly centralised is based on Bitcoin Laundering by Tom Robinson and Yaya Fanusie. Shubber writes:
  

  just three mixers and gambling sites account for 97 per cent of the volume in their categories and 50 per cent of the volume overall.
  
  In short, the illicit bitcoin ecosystem is centered around a small number of services that could be subject to scrutiny, regulation and co-option by law enforcement.

• Nellie Bowles' Everyone Is Getting Hilariously Rich and You’re Not points out that:
  

  The goal may be decentralization, but the money is extremely concentrated. Coinbase has more than 13 million accounts that own cryptocurrencies. Data suggests that about 94 percent of the Bitcoin wealth is held by men, and some estimate that 95 percent of the wealth is held by 4 percent of the owners.

  Where is the money flowing to the whales coming from? New Survey Reveals Staggering Number Of People Are Buying BitCoin On Their Credit Cards. Poor people's credit cards, that is. Bowles' whole article is very nostalgic for those of us who were in Silicon Valley for the dot-com boom. But this time it's different!

It Isn't About The Technology

A year and a half ago I attended Brewster Kahle's Decentralized Web Summit and wrote:

I am working on a post about my reactions to the first two days (I couldn't attend the third) but it requires a good deal of thought, so it'll take a while.

As I recall, I came away from the Summit frustrated. I posted the TL;DR version of the reason half a year ago in Why Is The Web "Centralized"? :

What is the centralization that decentralized Web advocates are reacting against? Clearly, it is the domination of the Web by the FANG (Facebook, Amazon, Netflix, Google) and a few other large companies such as the cable oligopoly.

These companies came to dominate the Web for economic not technological reasons.

Yet the decentralized Web advocates persist in believing that the answer is new technologies, which suffer from the same economic problems as the existing decentralized technologies underlying the "centralized" Web we have. A decentralized technology infrastructure is necessary for a decentralized Web but it isn't sufficient. Absent an understanding of how the rest of the solution is going to work, designing the infrastructure is an academic exercise.

It is finally time for the long-delayed long-form post. I should first reiterate that I'm greatly in favor of the idea of a decentralized Web based on decentralized storage. It would be a much better world if it happened. I'm happy to dream along with my friend Herbert Van de Sompel's richly-deserved Paul Evan Peters award lecture entitled Scholarly Communication: Deconstruct and Decentralize?. He describes a potential future decentralized system of scholarly communication built on existing Web protocols. But even he prefaces the dream with a caveat that the future he describes "will most likely never exist".

I agree with Herbert about the desirability of his vision, but I also agree that it is unlikely. Below the fold I summarize Herbert's vision, then go through a long explanation of why I think he's right about the low likelihood of its coming into existence.

Why Decentralize?

In Blockchain: Hype or Hope? (paywalled until June '18) Radia Perlman asks what exactly you get in return for the decentralization provided by the enormous resource cost of blockchain technologies? Her answer is:

a ledger agreed upon by consensus of thousands of anonymous entities, none of which can be held responsible or be shut down by some malevolent government ... [but] most applications would not require or even want this property.

Two important essays published last February by pioneers in the field provide different answers to Perlman's question:

• Vitalik Buterin's answer in The Meaning of Decentralization is that what you get depends on what exactly you mean by "decentralization".
• Nick Szabo's answer in Money, blockchains, and social scalability is "social scalability"

Below the fold I try to apply our experience with the decentralized LOCKSS technology to ask whether their arguments hold up. I'm working on a follow-up post based on Chelsea Barabas, Neha Narula and Ethan Zuckerman's Defending Internet Freedom through Decentralization from last August, which asks the question specifically about the decentralized Web and thus the idea of decentralized storage.