Decentralized Consensus: The Technical Challenge of Bitcoin

Distributed Consensus So we've talked in a generic manner about centralization and decentralization. Let's now talk, at a bit more technical level, about Bitcoin and decentralization. And a key word that's gonna come up again and again here is consensus, specifically distributed consensus. So what am I talking about here? At a technical level, the key challenge that you have to solve to build a distributed e-cash system is called distributed consensus. And this is a class of protocols that's been studied for decades in the computer science literature. So, but intuitively, you can think of it as our goal being to decentralize ScroogeCoin, which is the hypothetical currency that we saw in the first lecture. So as I said, there's decades of research in computer science on these consensus protocols. And the traditional motivating application for this is reliability in distributed systems. What do I mean by that? Imagine you're in charge of the backend for a company like Google or Facebook. These companies typically have thousands, or even millions of servers, which form a massive distributive database that records all of the actions that happen on the system, like users' comments and likes and posts, and so on. So when a new comment, let's say, comes in, the way it'll be recorded is that there might be 10 or 15 different nodes, in that massive backend, that might contain copies of this action. The server must now ensure that either every copy of that database contains that comment or none of them do. If for some reason, because some of these nodes might be faulty, the action gets recorded in none of the databases, it's okay. You can go back to the user and say, there was a problem saving your post, would you please try again? On the other hand, if some of the copies of the database saved it and others didn't, then you'd be in a lot of trouble, because you'd have an inconsistent database. So this is the key problem that motivated the traditional research on distributed consensus. And you can sort of see the similarities to Bitcoin here. But we're gonna talk in a bit more detail about the similarities and differences. So that was the traditional motivating application. But we can also imagine that if we achieved a distributed consensus protocol, and we were able to use that to build a massive, global scale distributed key value store that maps arbitrary keys or names to arbitrary values, then that will enable a lot of applications. For example, a distributed domain name system, which is simply a mapping between human understandable domain names to IP addresses. Or a public key directory, which is a mapping between user email addresses, let's say, to their public keys. Or even things like stock trades. Because this distributed database, instead of keeping track of who's paid whom how much money, would keep track of who's transferred what units of which stock to whom. And the cool thing about this is that now that Bitcoin has solved the distributed consensus problem, in a certain sense that we'll try to understand in this lecture. We can also go ahead and try to think about solutions to all of these other related problems. And in fact, there are many Altcoins. And Altcoins, we'll have several more lectures about Altcoins. But very briefly, Altcoins are systems built on Bitcoin-like principles to achieve, perhaps, slightly different goals. Sometimes currency systems, sometimes not currency systems, such as one of these applications. And so, given that we can solve distributed consensus now. And given that we can build a global distributed key value store. It enables a lot of these other cool applications. Now, let's proceed to a technical definition now. In reality, distributed consensus has a very straightforward technical definition. Suppose that there are n nodes or processes, where n is a fixed number. And each of these nodes has a value that may be input. And then a consensus protocol happens. And the two requirements on this consensus protocol are that the protocol should terminate, and all correct nodes should decide on some value, the consensus value, all right? And I say correct nodes, because some of the nodes might be faulty, or even outright malicious. The second stipulation is that the value they decide upon cannot be chosen at random. But it must be a value that at least one of these reliable nodes suggested as input. So, it really is that easy. Let's attempt to interpret this in the perspective of Bitcoin, though. So, to understand how distributed consensus could work in Bitcoin, let's start with a reminder that Bitcoin is a peer-to-peer system, all right? So, what I mean when I say Bitcoin is a peer to peer system is that when Alice wants to pay Bob, what she does is she's going to broadcast the transaction to all of the Bitcoin nodes that comprise the peer-to-peer network. And you can see here the structure of the transaction. This is similar to GoofyCoin, that we saw in the first lecture. And what a transaction is going to have is it's going to have Alice's signature. Which the other nodes need in order to know that it really, in fact, came from Alice. It's going to have Bob's public key, which also accesses his address at which he wants to receive bitcoins. Moreover, it includes a hash. What is hash, exactly? Remember the idea of hash pointers from the first lesson. So this hash is a means for Alice to connect this transaction or this currency to the coin she previously received from someone else, correct? These thus are the elements that make up the data structure that we refer to as a transaction. And she will broadcast that information to all Bitcoin P2P nodes. And notice something funny here. Bob's computer is nowhere in this picture. Now Bob, if he wants to be notified that this transaction did in fact happen and that he got paid, he might wanna run a Bitcoin node that's one of these peer-to-peer nodes, in order to listen in on the network and be sure that he's received that transaction. But his listening is not, in fact, necessary for him to receive the funds. The bitcoins will be his whether or not he's running a node on the network. So, given this peer-to-peer system, what is it exactly that the nodes might want to reach consensus on? Well, given that a variety of users are broadcasting these transactions to the network. What everybody wants to reach consensus on is exactly which transactions were broadcasted, and the order in which these transactions happened. So what does that mean, specifically? How consensus could work in Bitcoin is that at any given time, all the nodes in the peer-to-peer network would have a sequence of blocks of transactions that they've reached consensus on. So recall that in ScroogeCoin, for optimization purposes, for efficiency, we put transactions into blocks. And we link these blocks together on a block chain. So we're utilizing a similar principle here. We could do consensus on transactions one by one, that would be okay. It would just be inefficient. So instead, we do consensus on a block by block basis. So at any given point, all these nodes in the peer-to-peer network would have the sequence of blocks that they have agreed upon already. And each node would then have a set of outstanding transactions that it has heard about. So recall that for these transactions, consensus has not yet happened. And so, almost by definition, each node might have a slightly different version of the outstanding transactions thats it's heard about. The peer-to-peer network is not perfect, so some node may have heard about a transaction, but not other nodes. So given that we the setup, what could happen is that you have the sequence of blocks that everybody has agreed upon. A block is just a series of transactions. Why Bitcoin's Consensus Protocol Is a Technical Challenge The consensus protocol for Bitcoin is a peer-to-peer, decentralized network. The protocol's goal is to make sure that everyone on the network agrees to use the same version of the blockchain. But creating a consensus protocol that can function in a decentralized setting like Bitcoin is a difficult technological challenge. This essay will examine the difficulties the Bitcoin consensus mechanism encounters and the reasons why it is a challenging technical issue. The Basics of Bitcoin's Consensus Protocol To understand why Bitcoin's consensus protocol is a technical challenge, it's essential to understand how it works. The consensus protocol allows nodes in the network to agree on a single version of the blockchain. Nodes in the network can propose blocks of transactions, which are then verified by other nodes in the network. Once a block of transactions is verified, it becomes part of the blockchain, and the consensus protocol moves to the next block. The consensus protocol is crucial to Bitcoin's success because it ensures that all transactions are valid, and no one can spend the same bitcoins twice. However, designing a consensus protocol that can work in a decentralized environment like Bitcoin is a challenging technical problem. Challenges Faced by the Bitcoin Consensus Protocol There are several reasons why designing a consensus protocol that can work in a decentralized environment like Bitcoin is a challenging technical problem. The following are some of the most significant challenges faced by the Bitcoin consensus protocol: Node Failure: Nodes in the network might fail, and some nodes might be malicious. In either case, the consensus protocol must be designed to handle these situations. Imperfect Network: The Bitcoin network is highly imperfect. Not all nodes are connected to each other, and there can be faults in the network due to poor internet connectivity. High Latency: There is a lot of latency in the Bitcoin network because all transactions happen over the internet. There is no notion of global time, so not all nodes can agree to a common ordering of events based on observing timestamps. Pessimistic Literature: Many impossibility results have been proved in the literature on distributed consensus, which has made the design of consensus protocols more challenging. The Byzantine Generals Problem: The Byzantine Generals Problem is a well-known impossibility result that proves that consensus is impossible, even with a single faulty process. The Fischer-Lynch-Paterson Impossibility Result: Under certain conditions, consensus is impossible, even with a single faulty process. Despite these challenges, there are a few well-known protocols, and Paxos is probably one of the better known. What Paxos does is it makes certain compromises. It never produces an inconsistent result, but it accepts the tradeoff that, under certain conditions, the protocol can get stuck and fail to make any progress. Why Bitcoin's Consensus Protocol Is a Hard Technical Problem Because of the difficulties mentioned above, creating a consensus mechanism that can function in a decentralized setting like Bitcoin is a difficult technological problem. The network is quite unreliable, there is a lot of latency, and nodes could malfunction or be malicious. In addition, there is no concept of global time, which makes it difficult to create algorithms that can manage event ordering. The literature on distributed consensus is also somewhat pessimistic, with many impossibility results having been proved. These results have made the design of consensus protocols even more challenging. While some well-known protocols, such as Paxos, have been designed to handle these challenges, they still make certain compromises and may not be perfect in all situations. Conclusion It's a challenging technological challenge to create a consensus mechanism that can function in a decentralized setting like Bitcoin. Inadequate networks, high latency, no concept of global time, and node failure must all be taken into account when developing the Bitcoin consensus process.