Earlier this week, we released MultiChain 2.2 with integrated support for non-fungible tokens (NFTs). Over the past year, we’ve been talking to lots of customers about how they’d like to see NFTs supported on our platform. And with the release of MultiChain 2.2, we’re delivering exactly what they need.

What’s an NFT?

But first some background. If you’ve been watching the blockchain world for long enough, you’ll recognize the pattern. Every year or two, there’s a big new idea about how blockchains will transform some part of the economy. The media and money pile on and the price of everything skyrockets.

In 2013, it was all about bitcoin as a currency. In 2015, Ethereum and decentralized autonomous organizations (DAOs). In 2016, private blockchains for banks. In 2018, an explosion of new public blockchains. In 2020, decentralized finance (DeFi). Now, in 2021, it’s the turn of non-fungible tokens (NFTs).

So what is an NFT? It’s a single unit of value, issued onto a blockchain, which can be transferred, sold and exchanged by that blockchain’s users. Unlike cryptocurrencies or regular on-chain assets, only one unit of each NFT is available, so it can only be owned by one person or organization. By contrast, dollars and bitcoins are “fungible”, meaning that many units of these assets are in circulation, and each unit is perfectly interchangeable with any other.

But why would anyone pay for an NFT? Because it’s associated, through a piece of data on the blockchain, with an item of value. In theory, an NFT could represent the ownership of a house or car, so whoever owns the NFT on the blockchain has legal ownership of the related physical asset. In reality, most NFTs issued and bought today represent a piece of digital art, or some other type of digital collectible.

Digital but Unique

But surely digital things can be infinitely reproduced because, well, they’re digital? What is the value of owning the NFT for a big JPEG or viral video if anyone can easily duplicate the content? This question, while valid, could equally be asked about physical art. Right now, you can buy a full-sized “museum quality” replica of the Mona Lisa for a few hundred dollars online. And yet, the original is worth closer to a billion.

Whether it makes rational sense or not, we humans see psychological and cultural value in owning the original of something. The first copy that ever existed. The version produced by the artist herself. The same one that everyone else wants. In this respect, digital art, represented by a NFT, is not all that different from a classical painting. Sure, anyone can fake it. But everyone knows where the real one is.

Bragging rights aside, there can also be concrete benefits to owning an NFT. Holding the token can grant you the right to display the digital art in public or on a website. It can allow you to lend out a digital collectible or create derivative works. And all of this is enforced by old-school copyright law, with the artist maintaining the right to sue someone using their work illegally.

At the moment, the big NFT news is all on public blockchains. For the most part, that means Ethereum, with some activity on newer chains such as Solana, FLOW and Origin.

But what about organizations that can’t work with a free-for-all public blockchain? What about NFT applications that need predictable transaction cost and speed, in a blockchain with some degree of governance and control? Or what if the NFTs represent physical goods in a supply chain, rather than abstract financial assets?

NFTs in MultiChain 2.2

This is where MultiChain comes in. MultiChain has supported issuing, transacting and exchanging assets over a blockchain all the way back to version 1.0 in 2015. In a world just starting to move past bitcoin, this was the first blockchain feature we implemented. Any number of named assets can be created on a MultiChain blockchain, and there are easy API commands for checking asset balances, sending assets in transactions, and creating safe atomic swaps between multiple assets and parties.

In MultiChain 2.2, NFTs are implemented as a simple extension to this approach. When a new asset is created, it can optionally be declared as non-fungible. For non-fungible assets, each issuance of units creates a new token with a unique name. Each token is separately identified and tracked across transactions, and can be exchanged with other tokens or assets in atomic operations. Each asset offers full control over who can issue tokens, as well as optional restrictions on who can send and receive them. And there are several ways to permanently associate data with a new token being issued – a JSON object, up to 64 MB of on-chain data, or up to 1 GB of off-chain data.

Version 2.2 adds several other features to MultiChain assets, useful for both fungible or non-fungible assets. Depending on the options passed during creation, assets can now be closed and/or reopened for issuance, giving the asset administrator(s) more control. A limit can be set for the number of new units per issuance (for NFTs this should generally be 1). And a separate limit is available for the total amount issued over an asset’s lifetime. All of this could previously be implemented using smart filters, but now they’re available out-of-the-box with no programming required.

What else is new in MultiChain 2.2?

Aside from NFT support, MultiChain 2.2 introduces two other major enhancements. First, it supports MultiChain Explorer 2, a completely rewritten web-based blockchain browser. Explorer 2 is far faster and more scalable than the previous version, because it leverages MultiChain to track the blockchain’s state instead of its own local database. It supports multiple blockchains simultaneously and no longer needs to run on the same computer as a node. Explorer 2 is open source under the BSD license and available now on Github.

Second, read operations for MultiChain streams are now multi-threaded. Many of our users rely on streams for large-scale data applications, publishing thousands of pieces of data per second at peak times. While MultiChain can handle this load comfortably, nodes can take around a second to process the resulting large blocks. In previous versions of MultiChain, API commands to query streams would have to wait while the node was processing a block. Now, in MultiChain 2.2, all stream read operations are processed in parallel and receive immediate responses.

How to get started

To get started with NFTs on MultiChain, simply download MultiChain 2.2 and follow sections 1, 2, 3 and 7 of the Getting Started guide. For more information, check out the updated API documentation for details of how to issue non-fungible assets and their associated tokens with the issue and issuetoken commands.

MultiChain continues to be available in two editions – Community (open source) and Enterprise (commercial). The Enterprise edition adds a number of features relating to confidentiality, scalability and compliance, such as read-restricted streams, end-to-end encryption, external database integration and purging of off-chain data. A time-limited demo version of MultiChain Enterprise is available for download.

Finally, we’re interested in discussing ideas for setting up a low-cost, high-volume network for transacting NFTs, built on MultiChain. One idea raised is to create a blockchain which is open to the public but uses permission-based mining to avoid the energy demands of proof-of-work. If this sounds of interest, please get in touch.

Please post any comments on LinkedIn.

Coin Sciences Ltd is proud to announce the release of MultiChain 2.2, with integrated support for non-fungible tokens (NFTs). MultiChain now offers a simple and rapid path for organizations looking to deploy NFTs in a permissioned blockchain environment.

MultiChain’s support for NFTs extends its existing multi-asset functionality, which has been available since version 1.0. Now, in MultiChain 2.2, assets can be made non-fungible, so that each of their units is separately named and tracked. Any number of these unique tokens can be issued, transacted and atomically swapped with other assets or NFTs in the blockchain. To enable digital art and collectibles, issued tokens can be directly associated with metadata, such as a JSON object, image or video up to 1GB in size. As a permissioned blockchain platform, MultiChain also offers customized control over who can create, send and receive NFTs.

Unlike most enterprise blockchain platforms, MultiChain’s NFT functionality is provided out-of-the-box without requiring custom programming. This makes it easy for organizations to build NFT applications without needing to write and audit smart contracts. For those users who do need more control, MultiChain Smart Filters allow custom rules to be defined for tokens, transactions and data, using the popular JavaScript language.

Version 2.2 of MultiChain also adds support for Explorer 2, a completely rewritten web-based blockchain browser which is faster and more scalable than the previous version. Explorer 2 is open source under the BSD license and available for download now. In addition, MultiChain 2.2 drastically improves the performance of large-scale data applications, in which thousands of pieces of information are published and queried per second.

MultiChain 2.2 is available for Linux and Windows at: https://www.multichain.com/download-install/. It is available in two editions – Community (open source) and Enterprise (commercial). The Enterprise edition adds a number of features relating to confidentiality, scalability and compliance, such as read-restricted streams, end-to-end encryption, external database integration and purging of off-chain data.

“We were already building our trade asset management solution on MultiChain,” said Kieran Kelly, CTO of ubloquity. “With the addition of native NFT support, we’re excited to take this to the next level, allowing individual assets to be verified and tracked across the supply chain in an easy, sustainable and scalable way.”

“With MultiChain 2.2, we’re delighted to add extensive support for non-fungible tokens, empowering organizations to build NFT applications while retaining some governance and control,” said Dr Gideon Greenspan, CEO and Founder of Coin Sciences Ltd. “This capability was added in response to a large number of requests from our users, partners and customers. Our goal with MultiChain remains the same: to provide the most powerful, stable and easy-to-use platform for building permissioned blockchain solutions.”

Today, we’re delighted to release MultiChain 2.1, with two important new features for MultiChain developers.

A year and a half ago, MultiChain 2.0 introduced Smart Filters, which enable custom logic to be embedded in a blockchain for validating transactions and data. Smart Filters are conceptually similar to the “smart contracts” provided by other blockchain platforms, but have a different design to fit MultiChain’s faster transaction model.

Smart Filters come in two varieties – transaction filters and stream filters. A transaction filter validates on-chain transactions in their entirety, by examining their inputs, outputs and metadata. If a transaction doesn’t pass the filter, it is rejected by every node in the network. A stream filter validates individual items written to a MultiChain stream, looking at their key(s), publisher(s) and on-chain or off-chain data, in JSON, text or binary format. If an item doesn’t pass the filter, it is marked as invalid and its data is hidden by every node subscribed to the stream.

Both types of Smart Filter are written in JavaScript and run within a deterministic version of Google’s V8, the super-fast JavaScript engine which powers Chrome, Node.js and many other platforms. Simple filters are easy to code and understand – for example, here’s a stream filter which validates that items have at least two keys:

function filterstreamitem()
{
    var item=getfilterstreamitem(); // callback function
    if (item.keys.length<2)
        return "At least two keys required";
}

Overall, we've had great feedback on Smart Filters, but we've also heard repeatedly about two ways we could make them even better. First, many users want filters that can read information which is on the blockchain but not within the transaction or stream item being validated. Use cases include a changing list of permitted countries, exchange rates provided by an external "oracle" or a switch to toggle certain rules.

Second, some developers want to use a set of JavaScript functions in multiple filters, without duplicating code. They also want to be able to update these functions, to fix a bug or cover some new situations, without disabling their existing filters and creating new ones in their place. For example, shared code could contain application-specific logic, a third-party library for validating JSONs or parsing the contents of a PDF.

MultiChain 2.1 introduces two new types of on-chain entity, variables and libraries, to answer these needs.

Variables

Let's start with MultiChain variables. These work quite like those in regular programming languages, but with a blockchain twist. Any number of named variables can be created on the blockchain. Each variable has a dynamic set of addresses which is permitted to update its value, and this set is managed by one or more variable administrators (by default, only the variable's creator). Variables are created or updated in a blockchain transaction, which can be sent using a simple high-level API command. Alternatively, lower-level APIs can be used to build complex transactions which atomically set one or more variables, write stream items, transfer assets, change permissions, and so on.

The variable value itself can contain any JSON structure, including numbers, strings, booleans, nested objects and arrays, and is stored on-chain in the efficient UBJSON serialization format. Of course, Smart Filters can query a variable's current value using a simple callback function. But because this is a blockchain, the full history of the variable's values and writers is also available, and can be retrieved in part or in full using another callback function. To make development easier, these callbacks are also available through the application-facing API.

Libraries

Let's move on to libraries, which are variables' bigger sibling. As with variables, any number of named libraries can be created on the blockchain. But libraries have a richer model for updating, with three available modes – immutable, instant, and subject-to-approval.

Once an immutable library is created, its code can never be changed. A library with instant updates can be changed like a variable, with a set of addresses that can individually replace its code. But in a library with subject-to-approval updates, an update is only applied after being approved by a certain proportion of the blockchain's global administrators. This last mode provides a great trade-off between security and flexibility.

The code for a library is written in regular JavaScript, and defines one or more functions for Smart Filters to use. As with variables, libraries are created or updated in a special transaction, sent easily using the API. When a Smart Filter is created, its required libraries are then provided in an optional parameter. A library always runs in the context of the filter which requires it, so it can use Smart Filter callbacks where appropriate. MultiChain provides extensive functionality for testing (and rolling back) libraries and their updates locally, before making a change on the blockchain.

Wrapping it up

As with all features relating to the blockchain's rules, variables and libraries are available in both the Community and Enterprise editions of MultiChain 2.1. Our developer documentation provides a detailed description of the new JSON-RPC APIs and Smart Filter callbacks available. To use the new features on a blockchain already running, first migrate the nodes to MultiChain 2.1, then upgrade the chain's protocol to version 20012.

So what's next? Over the coming months, we'll be focused on some detailed and intense under-the-hood work, to improve the performance and concurrency of MultiChain while it's under significant load. This will increase the responsiveness of applications which need to query a node while it continues to process hundreds or thousands of new transactions per second.

In the meantime, all of us on the MultiChain team wish our users and customers health and sanity for the year ahead. We know these are difficult times for everyone, and we look forward to returning to the normal rhythm of meetings and conferences as soon as it's safe.

Please post any comments on LinkedIn.

With the first public release of MultiChain, way back in 2015, we saw interest in blockchain applications from a surprising direction. While we had originally designed MultiChain to enable the issuance, transfer and custody of digital assets, an increasing number of users were interested in using it for data-oriented applications.

In these use cases, the blockchain’s purpose is to enable the storage and retrieval of general purpose information, which need not be financial in nature. The motivation for using a blockchain rather than a regular database is to avoid relying on a trusted intermediary to host and maintain that database. For commercial, regulatory or political reasons, the database’s users want this to be a distributed rather than a centralized responsibility.

The Evolution of Streams

In response to this feedback, in 2016 we introduced MultiChain streams, which provide a simple abstraction for the storage, indexing and retrieval of general data on a blockchain. A chain can contain any number of streams, each of which can be restricted for writing by certain addresses. Each stream item is tagged by the address of its publisher as well as an optional key for future retrieval. Each node can independently decide whether to subscribe to each stream, indexing its items in real-time for rapid retrieval by key, publisher, time, block, or position. Streams were an instant hit with MultiChain’s users and strongly differentiated it from other enterprise blockchain platforms.

In 2017, streams were extended to support native JSON and Unicode text, multiple keys per item and multiple items per transaction. This last change allows over 10,000 individual data items to be published per second on high-end hardware. Then in 2018, we added seamless support for off-chain data, in which only a hash of some data is published on-chain, and the data itself is delivered off-chain to nodes who want it. And later that year we released MultiChain 2.0 Community with Smart Filters, allowing custom JavaScript code to perform arbitrary validation of stream items.

During 2019 our focus turned to MultiChain 2.0 Enterprise, the commercial version of MultiChain for larger customers. The first Enterprise Demo leveraged off-chain data in streams to allow read permissioning, encrypted data delivery, and the selective retrieval and purging of individual items. As always, the underlying complexity is hidden behind a simple set of APIs relating to permissions and stream items. With streams, our goal has consistently been to help developers focus on their application’s data, and not worry about the blockchain running behind the scenes.

The Database Dilemma

As MultiChain streams have continued to evolve, we’ve been faced with a constant dilemma. For reading and analyzing the data in a stream, should MultiChain go down the path of becoming a fully-fledged database? Should it be offering JSON field indexing, optimized querying and advanced reporting? If so, which database paradigm should it use – relational (like MySQL or SQL Server), NoSQL (MongoDB or Cassandra), search (Elastic or Solr), time-series (InfluxDB) or in-memory (SAP HANA)? After all, there are blockchain use cases suited to each of those approaches.

One option we considered is using an external database as MultiChain’s primary data store, instead of the current combination of embedded LevelDB and binary files. This strategy was adopted by Chain Core (discontinued), Postchain (not yet public) and is available as an option in Hyperledger Fabric. But ultimately we decided against this approach, because of the risks of depending on an external process. You don’t really want your blockchain node to freeze because it lost its database connection, or because someone is running a complex query on its data store.

Another factor to consider is technology and integration agnosticism. In a blockchain network spanning multiple organizations, each participant will have their own preferences regarding database technology. They will already have applications, tools and workflows built on the platforms that suit their needs. So in choosing any particular database, or even in offering a few options, we’d end up making some users unhappy. Just as each blockchain participant can run their node on a wide variety of Linux flavors, they should be able to integrate with their database of choice.

Introducing MultiChain Feeds

Today we’re delighted to release our approach to database integration – MultiChain Feeds. A feed is a real-time on-disk binary log of the events relating to one or more blockchain streams, for reading by external processes. We are also offering the open source MultiChain Feed Adapter which can read a feed and automatically replicate its content to a Postgres, MySQL or MongoDB database (or several at once). The adapter is written in Python and has a liberal license, so it can be easily modified to support additional databases or to add data filtering and transformation. (We’ve also documented the feed file format for those who want to write a parser in another language.)

A node need not subscribe to a stream in order to replicate its events to a feed. This allows MultiChain’s built-in stream indexing to be completely bypassed, to save time and disk space. Feeds also reflect the retrieval and purging of off-chain data, and can report on the arrival of new blocks on the chain. In order to save on disk space, you can control exactly which events are written to a feed, and which fields are recorded for each of those events. In addition, feed files are rotated daily and there’s a simple purge command to remove files after processing.

Why are MultiChain feeds written to disk, rather than streamed between processes or over the network? Because we want them to serve as an ultra-reliable replication log that is resilient to database downtime, system crashes, power loss and the like. By using disk files, we can guarantee durability, and allow the target database to be updated asynchronously. If for some reason this database becomes overloaded or disconnected, MultiChain can continue operating without interruption, and the database will catch up once things return to normal.

Getting Started with Feeds

Feeds are integrated into the latest demo/beta of MultiChain Enterprise, which is available for download now. Get started by reading the documentation for the MultiChain Feed Adapter, or reviewing the feed-related APIs. We’d love to hear your feedback on this feature and how we can expand it in future.

With the release of feeds, version 2.0 of MultiChain Enterprise is now feature complete – see the Download and Install page for a full comparison between the Community and Enterprise editions. Over the next couple of months we’ll be completing its testing and optimization, and expect it to be ready for production around the end of Q1. In the meantime, for information about MultiChain Enterprise licensing or pricing, please don’t hesitate to get in touch.

Please post any comments on LinkedIn.

A few months ago, we published the details of ten enterprise blockchain networks which are built on MultiChain and running in live production. One of these solutions, made for the pharmaceutical industry, is of particular interest – not only because of its scale, but also because it was built by SAP, the third largest enterprise software vendor in the world.

SAP’s core business runs on centralized systems, so why are they exploring decentralization and blockchains? You’ll find the answers below, in two parts. First, a detailed case study of the pharmaceutical project, written together with SAP, which explains the problem and why a blockchain solution was chosen. And second, an interview with Raimund Gross, the Innovation Manager who played a large role in driving SAP’s work with blockchains.

We hope you enjoy the read.

Please post any comments on LinkedIn.

Today we’re excited to release our demo version of MultiChain 2.0 Enterprise for all MultiChain users to try. The demo can be downloaded now and freely deployed for application development and testing. It unlocks all the features in MultiChain 2.0 Enterprise and can be used for the first 3 months of each blockchain’s life.

MultiChain Enterprise is commercial software that provides all the functionality of MultiChain Community, plus high-end blockchain features relating to confidentiality, scalability and compliance. It is fully compatible with all Community API commands and parameters, and you can mix Community and Enterprise nodes on a single network.

What’s in MultiChain 2.0 Enterprise?

Below is a list of new features that are available in MultiChain 2.0 Enterprise:

• Stream Read Restrictions. Enterprise nodes can use read-restricted streams, in order to publish and retrieve data that is only visible to nodes with the appropriate permissions. This is valuable in networks in which some information must be hidden from some participants.
• Encrypted Data Delivery. Enterprise nodes can send and receive read-restricted data to each other using automatic encryption. Even if two enterprise nodes are not directly connected, this prevents intermediate nodes or routers from reading that data in transit.
• Selective Stream Indexing. When subscribing to a stream, an Enterprise node can control exactly how that stream is indexed. The indexes for keys and publishers can be individually controlled, as can all indexes with local ordering. This improves performance and reduces disk usage for subscribing nodes.
• Selective Data Retrieval. Enterprise nodes subscribed to a stream can control which specific off-chain items are retrieved from the network, instead of retrieving every item automatically. The items to retrieve can be specified by key and/or publisher, transaction, block or timestamp. This saves bandwidth and disk space.
• Off-Chain Data Purging. Enterprise nodes can selectively delete off-chain data retrieved over the network from local storage. As with selective retrieval, the items to purge can be specified by key, publisher, transaction, block or timestamp. This enables compliance with right-to-be-forgotten regulations such as GDPR or CCPA.

The technical documentation for these features can be found on the regular API commands and runtime parameters pages. Commands and parameters which are only available in MultiChain Enterprise are highlighted in yellow.

What’s next?

Over the coming years, we will be developing both the Community and Enterprise versions of MultiChain in parallel. Some high-end features will be available in MultiChain Enterprise only.

Our next big area for development in MultiChain Enterprise is the database bridge, which will allow a regular database to act as a real-time replica of the global blockchain state or the content of a stream. This feature will enable large scale data analysis to be performed more efficiently than by querying a node directly. It will be implemented in a database-independent way, allowing replication to any specific database via a small adapter. If the database bridge feature would be useful in a project you’re working on, please contact us and we’ll be happy to receive your input.

In the meantime, download MultiChain 2.0 Enterprise Demo and try out these great new features today!

Please post any comments on LinkedIn.

This is a write-up of a talk given at the Consensus 2019 conference. A video of the talk is also available.

In the four years since the first alpha version of MultiChain, hundreds (if not thousands) of proof-of-concept and pilot projects have been built by our partners on the platform. While many of the early ones were pointless blockchains, over time we have seen a consistent rise in the proportion of projects using the technology appropriately. Now we rarely hear about a blockchain-based application which lacks a good answer to the question: “Why not just use a regular database?” What a relief!

Proof-of-concepts and pilots are all well and good, but to my mind, the most important signal comes from solid enterprise blockchain projects that make it to live production. To be clear, this means networks containing multiple blockchain nodes belonging to multiple parties, where more than one of these parties is involved in generating real transactions and participating in the blockchain’s consensus algorithm. Without these characteristics, the blockchain is providing little or no value compared to a centralized database.

This article is a survey of ten of the most interesting permissioned blockchain applications built on MultiChain that are in production today. Each application will be described briefly, along with an explanation of why it made sense to use a blockchain and some numbers to give a sense of scale. Note that confidentiality agreements prevent us from revealing some of these projects’ details, but we’re telling you as much as we can. After reviewing the ten projects, I’ll finish with a list of five important lessons that I believe we can learn.

Ready? Then let’s begin…

Blockchain #1: SAP for Pharmaceuticals

Some medicines bought by large customers such as hospitals don’t end up being used, and are returned to wholesalers unopened for resale elsewhere. However this process brings a significant risk of counterfeiting, where the so-called “returns” have been faked somewhere along the way. To help combat this problem, every box of drugs can be shipped with a barcode label that identifies its contents and origin, with the barcode being recorded in a database for future verification. But who should be responsible for managing this critical database of drug shipment barcodes? In Europe, a centralized EU-level body was set up for this purpose, but there is no corresponding governmental entity in the USA.

To solve this dilemma, SAP built a blockchain-based solution on top of MultiChain, where multiple drug manufacturers and wholesalers have their own node, granting them direct access for reading and writing the chain. Each barcode is recorded as an item in a MultiChain data stream, allowing it to be looked up directly by scanning a printed label. The system is already running live and has been successfully tested to scale to 1.5 billion recorded barcodes and 30 million verifications per year.

Blockchain #2: TruBudget

When donor countries finance public projects in developing countries, it’s vital to keep track of important events in each project’s lifecycle, including tenders, contracts and disbursements. Both the donors and recipients want to maintain these records in a database for easy searching, but who should be in charge of that database? Neither side in the relationship is politically comfortable with ceding full control to the other, so this has often led to both parties maintaining their own records, and trying to keep them in sync. The picture is complicated further when there are multiple donor countries partnering together.

TruBudget is an open source application which uses a MultiChain blockchain to solve this dilemma. Each of the important stakeholders maintains its own node, writing important events to streams while sharing an identical picture of the project’s progress through their own front end. The system was commissioned by Germany’s Federal Ministry for Economic Cooperation and Development and developed by Accenture and KfW, Germany’s third largest bank. Two blockchains are now running in production for projects in Brazil and Burkina Faso respectively, with each expected to record up to 300 projects and 5,000 events per project.

Blockchain #3: Connected Health

In order to improve patient care and reduce bureaucracy, an Indian state government is implementing an electronic medical record system to enable information sharing between hospitals and other healthcare facilities in the state. When designing the system, two particular concerns arose. First, how can the records be secured against loss or tampering? Second, how do we ensure that the information is available locally in each city, in the event of a temporary loss of Internet connectivity?

These requirements were solved together by building the system on a blockchain rather than a centralized database. MultiChain streams are being used to store the medical records – currently with text only but with richer data such as images to be integrated later on. Participating cities will have their own nodes running locally, which take part in the consensus process. The system was built by RapidQube and is already in early production, with around 2 million records stored for over 50,000 people.

Blockchain #4: Collateralizing livestock

In many developing countries, farmers find it difficult to access affordable loans, even if they own valuable assets such as cattle that could serve as collateral. In order for a farmer’s cow to be used in this way, it must be identified and tagged, immunized against diseases and insured against potential mishaps. In addition, each cow can only be collateralized once. All this requires extensive data coordination between a country’s animal healthcare system, insurance companies and financial institutions, each of which has different incentives and governance structures.

FarmTrek is a blockchain-based solution developed by InfoCorp which enables this coordination to take place without being controlled by a central party. Each major stakeholder runs one or mode MultiChain nodes which work together to store and secure the data written to streams. Each cow is physically tagged with a tamper-proof NFC (near field communications) device, which connects to an Android mobile application used by the farmer to sign transactions and publish them to the blockchain. The project is now in live production in Myanmar and expected to scale to 100,000 farmers within two years, with an additional pilot in the works in Rwanda.

Blockchain #5: Tagcash KYC

As in many countries, when somebody opens a new bank account in the Philippines, the bank must perform rigorous KYC (know your customer) checks to verify the customer’s identity and residence. This costs time and money, meaning that banks and other financial service providers would benefit by sharing KYC information through a single database. Once built, this database can also form the basis of a credit scoring system, by adding information about customer loans and repayments (or failures thereof). Unfortunately, the Philippines has no centralized KYC and credit scoring mechanism, so this integration has been difficult to achieve.

In order to address this problem, Tagcash has created a blockchain-based KYC and credit scoring solution, using a network of nodes belonging to banks and smaller fintech companies. Some nodes have write privileges while others are permitted to read only. The information is stored within MultiChain streams, using a hash of each person’s name and birth date as a unique key for identifying their data. With the initial roll-out, around 100 records are being written per day, and this is expected to grow to 10,000/day over time.

Blockchain #6: Bureau Veritas Origin

With increasing awareness of food supply chain scandals, interest has grown in giving consumers greater transparency into how their food is sourced, processed, transported and stored. The goal is to create a comprehensive record of the steps involved in preparing an item for sale, and to enable consumers to access this information directly. To increase transparency and prevent tampering or corruption, it is preferable not to centralize control of this database at any individual company or location.

Bureau Veritas, a global company focused on testing and certification, has partnered with Atos Worldline to develop Origin, a blockchain-based food traceability solution. Nodes are run by multiple companies within the food supply chain, with data written to streams in a proprietary binary format. The finished products are labelled with QR codes, which consumers can scan in order to browse a web-based summary. With the initial roll-out, up to 100 records are being written per day.

(To avoid a common fallacy, it should be emphasized that the sources of data still need to be trusted when using a blockchain. The chain only improves the security of that data once it is stored.)

Blockchain #7: ILSBlockchain

An insurance linked security (ILS) is a bond which enables an insurance policy to be covered collectively by a group of investors. For example, the owners of a ship could pay a premium to the holders of an ILS, but if catastrophe strikes and the ship sinks, those holders lose some or all of their original investment. As with any financial asset, digitizing ILS ownership allows sales and transfers to take place efficiently. This is traditionally achieved using a custodian such as Euroclear, but the cost can be prohibitive for smaller insurance policies in the $10-20 million value range.

This problem was solved by Solidum Partners who issue and track ILS bonds on a MultiChain blockchain, removing the need for a highly regulated centralized custodian. Each bond is issued as a MultiChain asset, with participants transferring and exchanging these assets on a peer-to-peer basis. Nodes are run by the bond trustee, investors and reinsurers, with the consensus generated by a small group of senior participants. So far, four bonds have been issued on the blockchain, for over $50 million in total value.

Blockchain #8: Air Quality Chain

When it comes to collecting environmental data, three particular challenges need to be addressed. First, each type of data is generated in a different location, due to the need for specialized equipment. Second, the data must be stored safely and reliably for the very long term, to enable trends and changes to be analyzed. And third, different types of data may need to be cross-referenced in real time, to create a full picture of anomalies at the moment they occur.

These requirements can be addressed together by using blockchain. The Air Quality Chain project, implemented by Baumann, aggregates data on levels of ozone, radiation and air quality in Austria, using a network of nodes which collect data from multiple sources. Raw data is written directly to MultiChain streams, and so automatically replicated to all of the nodes in the network, which collectively ensure that it cannot be lost or modified. The system is running in production and collecting 2.7 million records annually, containing around 4 GB of raw data.

Blockchain #9: Deepshore Archive

Metro Group, the world’s fourth largest retailer, is required to archive all point-of-sale data for internal and external auditing purposes. Whereas Metro used to rely on a single vendor for this purpose, they recently migrated to a more flexible model, where the data can be redundantly stored on a number of different cloud providers. This gives them much greater freedom and the ongoing ability to negotiate over pricing.

However, this fragmentation presents a challenge in ensuring that all of the data is stored correctly and cannot be changed. To solve this problem, Metro have deployed a blockchain-based system, built by Deepshore, where a hash and some other metadata for each data set is stored in MultiChain streams for verification purposes. Multiple nodes are running in different subsidiaries and locations within the Metro Group, so even though this is an “internal blockchain”, control is effectively decentralized within a vast organization. The system is already running live and notarizing approximately 9 million datasets per day.

Blockchain #10: Fantastec SWAP

Growing up in the 1980s in the UK, collecting football stickers was hugely popular. We spent our pocket money on packets of random stickers, containing the faces of players, team photos and badges, and swapped obsessively with each other in at attempt to complete each year’s album. Fantastec has now developed a digital equivalent, where users download the SWAP app and purchase limited edition “cards”, complete with player videos and interactive statistics. Naturally, this application needs some database to keep track of card ownership, but it wasn’t clear where this database should be hosted. On the one hand, each participating football club should maintain its own database, to guarantee the authenticity and rarity of its issued cards. On the other hand, much of the product’s value derives from the ability to swap cards that were issued by different clubs.

This dilemma was solved by building the system on a blockchain, where each club has its own node that issues its digital collectibles as MultiChain assets, all of which are tracked together on a chain that is managed by consensus. The system, which makes extensive use of MultiChain’s built-in atomic exchange functionality, was built by Fantastec with assistance from partners such as PricewaterhouseCoopers. SWAP was recently launched with three big-name partners: Real Madrid, Arsenal and Borussia Dortmund. After less than 3 months it has grown to 15,000 users with over 250,000 collectibles issued.

Lessons learned

Now that we’ve reviewed ten of the most interesting MultiChain-based networks in production, what can we learn from this group as a whole? What differentiates these projects from the hundreds and thousands of proofs-of-concept and pilots that never made it to the next stage?

Lesson #1: Focus on new applications

While there has been much talk about blockchains as an upgrade for existing systems, for now at least, we’re primarily seeing them deployed in new applications. I can think of two related reasons why this might be.

First, blockchains are still a new technology, and are perceived as more risky than centralized databases. This uncertainty can be tolerated when building new applications, which inevitably comes with some risk of failure. However, it makes blockchains less attractive for replacing something that is already known to work.

Second, any running centralized application must already have a trusted intermediary, who has presumably proven their reliability over time. While moving to a decentralized architecture might save money in bypassing this intermediary, this has to be weighed against the cost and risk of rebuilding the system from the ground up.

Lesson #2: Find a strong motive

Every application implemented on a blockchain must answer a crucial question: Why use a blockchain instead of a centralized database or file server? Blockchains will always be slower, less scalable and more complex than centralized systems, as a result of their fundamental design.

So if you have a suitable trusted intermediary who can host an application centrally, you should use it! The only reason to use a blockchain is if there is a strong motive to avoid this kind of centralization. In practice we see four main types of motive appear:

1. Commercial concerns. The participants in a network do not want to grant too much power to a competitor or some other central body, who could charge a lot for the service.
2. Regulatory requirements. Some regulation prevents the deployment of a centralized system, or would render it too expensive in terms of compliance.
3. Political risks. There is no place where the database could be hosted that would be politically acceptable to all of its users.
4. Secure replication. Multiple copies of the data need to be stored for redundancy, so using a blockchain provides the additional benefit of proven synchronization and tamper resistance.

Lesson #3: Think about data in general

Early discussions about enterprise blockchains were triggered by the rise of cryptocurrencies, in which the blockchain allows users to directly hold and transfer a virtual asset while preventing double spends. While some of the production networks we described (#7, #10) are using MultiChain in this way, the majority are doing something fundamentally different – building a decentralized architecture for storing and securing data.

Any database or file system, whether it is holding structured or unstructured data, could be implemented on a blockchain. Each piece of data can be stored in full on the chain, or notarized as a short on-chain hash (fingerprint) which serves to verify the data which is delivered off-chain. Unlike in asset use cases, there is no notion of ownership changing over time. The blockchain’s sole purpose is to enable some information to be stored and secured by a group, without relying on a central party.

In data-driven applications, “smart contracts” are the wrong transactional model, since they require every piece of data to be represented as a message sent to a contract, rather than being validated and then directly embedded (or hashed) in the chain. The central issue is the scale and speed with which information can be stored, indexed and retrieved.

Lesson #4: Look beyond “transformation”

For too long, the enterprise blockchain narrative has focused on buzzwords like “revolution” and “transformation”. But in reality, if we look at those blockchain projects that actually make it to production, only a few are doing things that would be impossible to achieve using more traditional technologies such as centralized databases, replication and point-to-point messaging. So what exactly is being transformed?

In most cases, a blockchain is being used simply because it’s the most appropriate and convenient tool for the job. It enables a new application to be easily built on top of a unified data store, while avoiding some concern about that store being centrally controlled. The blockchain provides additional robustness and tamper resistance, whose value outweighs the complexity and cost of running multiple nodes. While this all might seem rather unromantic, since when has enterprise IT been anything else?

But there is an additional, more subtle, part of the story. In rare cases we see projects being built on a blockchain, where there is no immediate justification for that choice. It turns out that the application’s users are happy for it to start out centralized, but want to keep their options open for the future. Using a blockchain (even with one node!) rather than a database allows the intermediary to be swapped or removed just by adding or removing nodes and changing some permissions. All this can happen with zero downtime and without touching the application’s code.

Lesson #5: Be very patient

With all of the noise surrounding blockchains, it’s easy to forget just how new this industry is. MultiChain, along with most other enterprise blockchain platforms, only reached a version 1.0 release in mid-to-late 2017 (it’s now at version 2.0.2). Since it’s quite common for enterprise IT projects, whether based on blockchains or not, to take two years from initiation to go live, it’s no surprise that the number of real blockchain networks in production is still rather small.

Indeed, two particular phenomena demonstrate just how early things are. First, we often find our partners performing the most basic tests on MultiChain just to convince themselves that it actually works! Second, we see some participants in production blockchain networks lacking the confidence to take responsibility for their own node, instead relying on some third party to host it on their behalf.

So as with any other new enterprise technology, people working in the blockchain space should hunker down for the very long term. I expect it will take another ten years before blockchains are commonly considered as an alternative for information system architectures, and another ten after that before they reach their full potential. By then, bandwidth, storage and cryptography will be so cheap and fast that it may seem quaint (if not ridiculous) for shared applications to store their data in only one place.

Please post any comments on LinkedIn.

Today we’re thrilled to announce the production release of MultiChain 2.0, after 18 months of development and 3 months in beta testing. Version 2.0 of MultiChain is now available to download for Linux and Windows, and can also be compiled for Linux, Windows or Mac OS X.

MultiChain 2.0 adds smart filters for custom transaction or data validation rules, and a ton of enhancements for streams, including support for text and JSON data, multiple keys, off-chain data and richer querying. Other new features include blockchain parameter upgrading, per-asset and custom permissions, inline metadata and a binary cache for dealing with large pieces of data. Click for more details on all the new functionality.

For production systems based on MultiChain 2.0 we offer both Commercial Licenses (non-GPL) and Service Level Agreements (SLAs). For more information about these products and guidelines for when they are appropriate, see the pricing page or contact us.

Consensus 2019

We’re delighted to be speaking and exhibiting at Consensus 2019, the biggest annual blockchain conference taking place in New York from May 13-15. If you would like to attend but don’t yet have a ticket, you can register with the promo code MultiChain300 to receive a $300 discount. And if you’re using or evaluating MultiChain for a project and would like to meet us there in person, be sure to reach out and we’ll set aside some time.

Running MultiChain in production?

Our talk at Consensus will cover a number of the most innovative blockchain networks in production on MultiChain, and will also be written up as a post on our blog. So if you’ve built or are running a live application on MultiChain, and would like it publicized to thousands of blockchain enthusiasts and developers, please get in touch so we can find out more. To be clear, we’re specifically interested in projects that have moved beyond the proof-of-concept and pilot stages.

What’s next for MultiChain?

We’re already hard at work developing the Enterprise edition of MultiChain 2.0, which will include several high-end features required for enterprise applications, while maintaining full compatibility with the open source Community version 2.0. These features relate to confidentiality, scalability and regulatory requirements. More details will be released publicly in due course, but come to our booth at Consensus 2019 for a sneak preview!

Finally I’d like to take this opportunity to thank our development team, led by Dr Michael Rozantsev, for their ongoing dedication and engineering excellence. Almost five years ago to the day, we founded this company on the hypothesis that blockchains, as a technology, have useful applications beyond cryptocurrencies. It’s remarkable to see how far our product, and the entire blockchain industry, have come.

Please post any comments on LinkedIn.

Today we’re delighted to release the first beta version of MultiChain 2.0, the next generation of the MultiChain blockchain platform, after 16 months in development. MultiChain 2.0 (download) includes three major new areas of functionality to help developers rapidly build powerful blockchain applications:

• Smart Filters. These allow custom rules to be coded for validating transactions or data. Smart Filters are written in JavaScript and run within a deterministic version of the high-performance V8 engine that powers Google Chrome. Click for more on Smart Filters or a comparison with Fabric, Ethereum and Corda.
• Off-chain data. Any item published in a MultiChain stream can optionally be stored off-chain, in order to save bandwidth and storage space. Off-chain data (up to 1 GB per item) is automatically hashed into the blockchain, with the data itself delivered rapidly over the peer-to-peer network. Click for more about off-chain data.
• Richer data streams. JSON and Unicode text are now supported natively and stored efficiently on- or off-chain. Multiple JSON items can be merged together, allowing a stream to serve as a database with a full audit history. Stream items can have multiple keys, and be queried by multiple keys and/or publishers together. Finally, to increase data throughput, a single transaction can publish multiple items to one or more streams.

In addition, MultiChain 2.0 provides several other smaller new features:

• Blockchain upgrading. Many blockchain parameters can be changed over time, subject to administrator consensus. These include the block time interval, maximum block size, and many transaction size limits.
• Per-asset permissions. Assets can optionally be issued with their own send and receive permissions, which can be controlled for each address by that asset’s issuer and/or its assigned administrators.
• Binary cache. Large pieces of binary data (up to 1 GB) can be added to MultiChain over multiple API calls, or uploaded directly via the file system.
• Inline metadata. Transaction outputs containing assets and/or native currency can now contain metadata in JSON, text or binary format. Smart Filters can easily read and respond to this metadata.
• Custom permissions. Six new permissions (three “high” and three “low”) can be assigned to addresses by two levels of administrator. These are useful for defining roles enforced by Smart Filters.

We’re also delighted to welcome over 40 new companies to the MultiChain partner program, bringing the total number to 86. New members include SAP who have built a deep integration with MultiChain in the SAP Cloud Platform.

MultiChain 2.0 beta 1 can be downloaded here. It is backwards compatible with version 1.0 with a few exceptions – see the API compatibility note. MultiChain 1.0 nodes and networks can be upgraded to version 2.0 in the usual way (be sure to back up first). We’ll also continue to maintain and fix any bugs in MultiChain 1.0 through 2019 at least.

Below is the full official press release about the 2.0 beta release.

MultiChain Releases Beta Version 2.0 with Over Forty New Partners

December 19, 2018 – Coin Sciences Ltd is delighted to announce the first beta release of MultiChain 2.0, along with the addition of 43 new members of the MultiChain Partner Program, bringing the total number to 86.

MultiChain 2.0 beta 1 has been released after sixteen months of intensive development including seven alpha versions, and is available for Linux and Windows at: https://www.multichain.com/download-install/. Enhancements over MultiChain 1.0 include richer data publishing with support for JSON and Unicode text, blockchain parameter upgrading, seamless integration of off-chain data storage and delivery, and Smart Filters, MultiChain’s approach to the smart contract paradigm.

MultiChain Smart Filters allow application developers to embed custom rules for transaction and data validation within the blockchain, using the popular JavaScript programming language. Filters are run within a deterministic version of V8, the highly optimized runtime engine used by Google Chrome and Node.js. For more information on MultiChain Smart Filters and how they compare to smart contracts in Hyperledger Fabric, Ethereum and R3 Corda, see: https://www.multichain.com/blog/2018/12/smart-contract-showdown/

The new members of the MultiChain Partner Program include SAP, who have integrated MultiChain into the SAP Cloud Platform and are deploying it for client projects. HCL Technologies, the multinational consulting company, also recently joined, along with 41 other blockchain and software companies. Members of the partner program have access to the MultiChain engineering team, can use MultiChain branding in their marketing materials, and are promoted on the MultiChain website. A full list of MultiChain’s partners can be found at: https://www.multichain.com/platform-partners/

“At SAP we are extending business solutions with MultiChain blockchain functionality via our SAP Cloud Platform offering.” said Torsten Zube, SAP’s Head of Blockchain. Furthermore, “We strategically decided that MultiChain should be part of our offering due to its proven, easy and mature distributed ledger technology addressing enterprise needs. The upcoming MultiChain 2.0 release will provide more functionality such as Smart Filters and off-chain data that we see as particularly relevant for enterprise scenarios going forward.”

“Version 2.0 represents a huge upgrade for MultiChain, integrating several major features commonly requested by our developer community,” said Dr Gideon Greenspan, CEO and Founder of Coin Sciences Ltd. “With version 1.0 in stable production since August 2017, our goal with MultiChain 2.0 remains the same: to provide a powerful, stable and easy-to-use platform for blockchain application developers. We look forward to continued cooperation with our partners to bring MultiChain-driven applications to enterprises, governments and beyond.”

Please post any comments on LinkedIn.

In most discussions about blockchains, it doesn’t take long for the notion of “smart contracts” to come up. In the popular imagination, smart contracts automate the execution of interparty interactions, without requiring a trusted intermediary. By expressing legal relationships in code rather than words, they promise to enable transactions to take place directly and without error, whether deliberate or not.

From a technical viewpoint, a smart contract is something more specific: computer code that lives on a blockchain and defines the rules for that chain’s transactions. This description sounds simple enough, but behind it lies a great deal of variation in how these rules are expressed, executed and validated. When choosing a blockchain platform for a new application, the question “Does this platform support smart contracts?” isn’t the right one to ask. Instead, we need to be asking: “What type of smart contracts does this platform support?”

In this article, my goal is to examine some of the major differences between smart contract approaches and the trade-offs they represent. I’ll do this by looking at four popular enterprise blockchain platforms which support some form of customized on-chain code. First, IBM’s Hyperledger Fabric, which calls its contracts “chaincode”. Second, our MultiChain platform, which introduces smart filters in version 2.0. Third, Ethereum (and its permissioned Quorum and Burrow spin-offs), which popularized the “smart contract” name. And finally, R3 Corda, which references “contracts” in its transactions. Despite all of the different terminology, ultimately all of these refer to the same thing – application-specific code that defines the rules of a chain.

Before going any further, I should warn the reader that much of the following content is technical in nature, and assumes some familiarity with general programming and database concepts. For good or bad, this cannot be avoided – without getting into the details it’s impossible to make an informed decision about whether to use a blockchain for a particular project, and (if so) the right type of blockchain to use.

Blockchain basics

Let’s begin with some context. Imagine an application that is shared by multiple organizations, which is based on an underlying database. In a traditional centralized architecture, this database is hosted and administered by a single party which all of the participants trust, even if they do not trust each other. Transactions which modify the database are initiated only by applications on this central party’s systems, often in response to messages received from the participants. The database simply does what it’s told because the application is implicitly trusted to only send it transactions that make sense.

Blockchains provide an alternative way of managing a shared database, without a trusted intermediary. In a blockchain, each participant runs a “node” that holds a copy of the database and independently processes the transactions which modify it. Participants are identified using public keys or “addresses”, each of which has a corresponding private key known only to the identity owner. While transactions can be created by any node, they are “digitally signed” by their initiator’s private key in order to prove their origin.

Nodes connect to each other in a peer-to-peer fashion, rapidly propagating transactions and the “blocks” in which they are timestamped and confirmed across the network. The blockchain itself is literally a chain of these blocks, which forms an ordered log of every historical transaction. A “consensus algorithm” is used to ensure that all nodes reach agreement on the content of the blockchain, without requiring centralized control. (Note that some of this description does not apply to Corda, in which each node has only a partial copy of the database and there is no global blockchain. We’ll talk more about that later on.)

In principle, any shared database application can be architected by using a blockchain at its core. But doing so creates a number of technical challenges which do not exist in a centralized scenario:

• Transaction rules. If any participant can directly change the database, how do we ensure that they follow the application’s rules? What stops one user from corrupting the database’s contents in a self-serving way?
• Determinism. Once these rules are defined, they will be applied multiple times by multiple nodes when processing transactions for their own copy of the database. How do we ensure that every node obtains exactly the same result?
• Conflict prevention. With no central coordination, how do we deal with two transactions that each follow the application’s rules, but nonetheless conflict with each other? Conflicts can stem from a deliberate attempt to game the system, or be the innocent result of bad luck and timing.

So where do smart contracts, smart filters and chaincode come in? Their core purpose is to work with a blockchain’s underlying infrastructure in order to solve these challenges. Smart contracts are the decentralized equivalent of application code – instead of running in one central place, they run on multiple nodes in the blockchain, creating or validating the transactions which modify that database’s contents.

Let’s begin with transaction rules, the first of these challenges, and see how they are expressed in Fabric, MultiChain, Ethereum and Corda respectively.

Transaction rules

Transaction rules perform a specific function in blockchain-powered databases – restricting the transformations that can be performed on that database’s state. This is necessary because a blockchain’s transactions can be initiated by any of its participants, and these participants do not trust each other sufficiently to allow them to modify the database at will.

Let’s see two examples of why transaction rules are needed. First, imagine a blockchain designed to aggregate and timestamp PDF documents that are published by its participants. In this case, nobody should have the right to remove or change documents, since doing so would undermine the entire purpose of the system – document persistence. Second, consider a blockchain representing a shared financial ledger, which keeps track of the balances of its users. We cannot allow a participant to arbitrarily inflate their own balance, or take others’ money away.

Inputs and outputs

Our blockchain platforms rely on two broad approaches for expressing transaction rules. The first, which I call the “input–output model”, is used in MultiChain and Corda. Here, transactions explicitly list the database rows or “states” which they delete and create, forming a set of “inputs” and “outputs” respectively. Modifying a row is expressed as the equivalent operation of deleting that row and creating a new one in its place.

Since database rows are only deleted in inputs and only created in outputs, every input must “spend” a previous transaction’s output. The current state of the database is defined as the set of “unspent transaction outputs” or “UTXOs”, i.e. outputs from previous transactions which have not yet been used. Transactions may also contain additional information, called “metadata”, “commands” or “attachments”, which don’t become part of the database but help to define their meaning or purpose.

Given these three sets of inputs, outputs and metadata, the validity of a transaction in MultiChain or Corda is defined by some code which can perform arbitrary computations on those sets. This code can validate the transaction, or else return an error with a corresponding explanation. You can think of the input–output model as an automated “inspector” holding a checklist which ensures that transactions follow each and every rule. If the transaction fails any one of those checks, it will automatically be rejected by all of the nodes in the network.

It should be noted that, despite sharing the input–output model, MultiChain and Corda implement it very differently. In MultiChain, outputs can contain assets and/or data in JSON, text or binary format. The rules are defined in “transaction filters” or “stream filters”, which can be set to check all transactions, or only those involving particular assets or groupings of data. By contrast, a Corda output “state” is represented by an object in the Java or Kotlin programming language, with defined data fields. Corda’s rules are defined in “contracts” which are attached to specific states, and a state’s contract is only applied to transactions which contain that state in its inputs or outputs. This relates to Corda’s unusual visibility model, in which transactions can only be seen by their counterparties or those whose subsequent transactions they affect.

Contracts and messages

The second approach, which I call the “contract–message model”, is used in Hyperledger Fabric and Ethereum. Here, multiple “smart contracts” or “chaincodes” can be created on the blockchain, and each has its own database and associated code. A contract’s database can only be modified by its code, rather than directly by blockchain transactions. This design pattern is similar to the “encapsulation” of code and data in object-oriented programming.

With this model, a blockchain transaction begins as a message sent to a contract, with some optional parameters or data. The contract’s code is executed in reaction to the message and parameters, and is free to read and write its own database as part of that reaction. Contracts can also send messages to other contracts, but cannot access each other’s databases directly. In the language of relational databases, contracts act as enforced “stored procedures”, where all access to the database goes via some predefined code.

Both Fabric and Quorum, a variation on Ethereum, complicate this picture by allowing a network to define multiple “channels” or “private states”. The aim is to mitigate the problem of blockchain confidentiality by creating separate environments, each of which is only visible to a particular sub-group of participants. While this sounds promising in theory, in reality the contracts and data in each channel or private state are isolated from those in the others. As a result, in terms of smart contracts, these environments are equivalent to separate blockchains.

Example rules

Let’s see how to implement the transaction rules for a single-asset financial ledger with these two models. Each row in our ledger’s database has two columns, containing the owner’s address and the quantity of the asset owned. In the input–output model, transactions must satisfy two conditions:

1. The total quantity of assets in a transaction’s outputs has to match the total in its inputs. This prevents users from creating or deleting money arbitrarily.
2. Every transaction has to be signed by the owner of each of its inputs. This stops users from spending each other’s money without permission.

Taken together, these two conditions are all that is needed to create a simple but viable financial system.

In the contract–message model, the asset’s contract supports a “send payment” message, which takes three parameters: the sender’s address, recipient’s address, and quantity to be sent. In response, the contract executes the following four steps:

1. Verify that the transaction was signed by the sender.
2. Check that the sender has sufficient funds.
3. Deduct the requested quantity from the sender’s row.
4. Add that quantity to the recipient’s row.

If either of the checks in the first two steps fails, the contract will abort and no payment will be made.

So both the input–output and contract–message models are effective ways to define transaction rules and keep a shared database safe. Indeed, on a theoretical level, each of these models can be used to simulate the other. In practice however, the most appropriate model will depend on the application being built. Does each transaction affect few or many pieces of information? Do we need to be able to guarantee transaction independence? Does each piece of data have a clear owner or is there some global state to be shared?

It is beyond our scope here to explore how the answers should influence a choice between these two models. But as a general guideline, when developing a new blockchain application, it’s worth trying to express its transaction rules in both forms, and seeing which fits more naturally. The difference will express itself in terms of: (a) ease of programming, (b) storage requirements and throughput, and (c) speed of conflict detection. We’ll talk more about this last issue later on.

Built-in rules

When it comes to transaction rules, there is one way in which MultiChain specifically differs from Fabric, Ethereum and Corda. Unlike these other platforms, MultiChain has several built-in abstractions that provide some basic building blocks for blockchain-driven applications, without requiring developers to write their own code. These abstractions cover three areas that are commonly needed: (a) dynamic permissions, (b) transferrable assets, and (c) data storage.

For example, MultiChain manages permissions for connecting to the network, sending and receiving transactions, creating assets or streams, or controlling the permissions of other users. Multiple fungible assets can be issued, transferred, retired or exchanged safely and atomically. Any number of “streams” can be created on a chain, for publishing, indexing and retrieving on-chain or off-chain data in JSON, text or binary formats. All of the transaction rules for these abstractions are available out-of-the-box.

When developing an application on MultiChain, it’s possible to ignore this built-in functionality, and express transaction rules using smart filters only. However, smart filters are designed to work together with its built-in abstractions, by enabling their default behavior to be restricted in customized ways. For example, the permission for certain activities might be controlled by specific administrators, rather than the default behavior where any administrator will do. The transfer of certain assets can be limited by time or require additional approval above a certain amount. The data in a particular stream can be validated to ensure that it consists only of JSON structures with required fields and values.

In all of these cases, smart filters create additional requirements for transactions to be validated, but do not remove the simple rules that are built in. This can help address one of the key challenges in blockchain applications: the fact that a bug in some on-chain code can lead to disastrous consequences. We’ve seen endless examples of this problem in the public Ethereum blockchain, most famously in the Demise of The DAO and the Parity multisignature bugs. Broader surveys have found a large number of common vulnerabilities in Ethereum smart contracts that enable attackers to steal or freeze other peoples’ funds.

Of course, MultiChain smart filters may contain bugs too, but their consequences are more limited in scope. For example, the built-in asset rules prevent one user from spending another’s money, or accidentally making their own money disappear, no matter what other logic a smart filter contains. If a bug is found in a smart filter, it can be deactivated and replaced with a corrected version, while the ledger’s basic integrity is protected. Philosophically, MultiChain is closer to traditional database architectures, where the database platform provides a number of built-in abstractions, such as columns, tables, indexes and constraints. More powerful features such as triggers and stored procedures can optionally be coded up by application developers, in cases where they are actually needed.

Determinism

Let’s move on to the next part of our showdown. No matter which approach we choose, the custom transaction rules of a blockchain application are expressed as computer code written by application developers. And unlike centralized applications, this code is going to be executed more than one time and in more than one place for each transaction. This is because multiple blockchain nodes belonging to different participants have to each verify and/or execute that transaction for themselves.

This repeated and redundant code execution introduces a new requirement that is rarely found in centralized applications: determinism. In the context of computation, determinism means that a piece of code will always give the same answer for the same parameters, no matter where and when it is run. This is absolutely crucial for code that interacts with a blockchain because, without determinism, the consensus between the nodes on that chain can catastrophically break down.

Let’s see how this looks in practice, first in the input–output model. If two nodes have a different opinion about whether a transaction is valid, then one will accept a block containing that transaction and the other will not. Since every block explicitly links back to a previous block, this will create a permanent “fork” in the network, with one or more nodes not accepting the majority opinion about the entire blockchain’s contents from that point on. The nodes in the minority will be cut off from the database’s evolving state, and will no longer be able to effectively use the application.

Now let’s see what happens if consensus breaks down in the contract–message model. If two nodes have a different opinion about how a contract should respond to a particular message, this can lead to a difference in their databases’ contents. This in turn can affect the contract’s response to future messages, including messages it sends to other contracts. The end result is an increasing divergence between different nodes’ view of the database’s state. (The “state root” field in Ethereum blocks ensures that any difference in contracts’ responses leads immediately to a fully catastrophic blockchain fork, rather than risking staying hidden for a period of time.)

Sources of non-determinism

So non-determinism in blockchain code is clearly a problem. But if the basic building blocks of computation, such as arithmetic, are deterministic, what do we have to worry about? Well, it turns out, quite a few things:

• Most obviously, random number generators, since by definition these are designed to produce a different result every time.
• Checking the current time, since nodes won’t be processing transactions at exactly the same time, and in any event their clocks may be out of sync. (It’s still possible to implement time-dependent rules by making reference to timestamps within the blockchain itself.)
• Querying external resources such as the Internet, disk files, or other programs running on a computer. These resources cannot be guaranteed to always give the same response, and may become unavailable.
• Running multiple pieces of code in parallel “threads”, since this leads to a “race condition” where the order in which these processes finish cannot be predicted.
• Performing any floating point calculations which can give even minutely different answers on different computer processor architectures.

Our four blockchain platforms employ several different approaches to avoiding these pitfalls.

Deterministic execution

Let’s start with Ethereum, since its approach is the most “pure”. Ethereum contracts are expressed in a special-purpose format called “Ethereum bytecode”, which is executed by the Ethereum Virtual Machine (“EVM”). Programmers do not write bytecode directly, but rather generate or “compile” it from a JavaScript-like programming language called Solidity. (Other languages used to be available but have since been deprecated.) Determinism is guaranteed by the fact that Solidity and Ethereum bytecode cannot encode any non-deterministic operations – it’s that simple.

MultiChain filters and Corda contracts choose a different approach, by adapting existing programming languages and runtime environments. MultiChain uses JavaScript running in Google’s V8 engine, which also forms the core of the Chrome browser and the Node.js platform, but with sources of non-determinism disabled. Similarly, Corda uses Java or Kotlin, both of which are compiled to “Java bytecode” which executes within a Java Virtual Machine (“JVM”). For now, Corda uses Oracle’s standard non-deterministic JVM, but work is under way to integrate a deterministic version. In the meantime, Corda contract developers must take care not to allow non-determinism in their code.

How does Ethereum’s purism compare with the evolutionary approach taken by MultiChain and Corda? The main advantage for Ethereum is risk minimization – a built-for-purpose virtual machine is less likely to contain an inadvertent source of non-determinism. While any such oversight could be fixed by a software update, it would be disruptive to any chain that was unlucky enough to encounter it. Ethereum’s problem, however, is that Solidity and the EVM constitute a tiny and nascent ecosystem in the wider context of programming languages and runtime environments. By comparison, JavaScript and Java are the top two languages on Github, run on billions of digital devices, and have runtimes that have been optimized over decades. Presumably this is why the public Ethereum blockchain is considering a transition to eWASM, a deterministic fork of the emerging WebAssembly standard.

Determinism by endorsement

When it comes to determinism, Hyperledger Fabric adopts a completely different approach. In Fabric, when a “client” node wants to send a message to some chaincode, it first sends that message to some “endorser” nodes. Each of these nodes executes the chaincode independently, forming an opinion of the message’s effect on that chaincode’s database. These opinions are sent back to the client together with a digital signature which constitutes a formal “endorsement”. If the client receives enough endorsements of the intended outcome, it creates a transaction containing those endorsements, and broadcasts it for inclusion in the chain.

In order to guarantee determinism, each piece of chaincode has an “endorsement policy” which defines exactly what level of approval is required in order to render its transactions valid. For example, one chaincode’s policy might state that endorsements are required from at least half of the blockchain’s nodes. Another might require an endorsement from any one of three trusted parties. Either way, every node can independently check if the necessary endorsements were received.

To clarify the difference, determinism in most blockchain platforms is based on the question: “What is the result of running this code on this data?” – and we need to be absolutely sure that every node will answer this question identically. By contrast, determinism in Fabric is based on a different question: “Do enough endorsers agree on the result of running this code on this data?” Answering that is a rather simple matter of counting, and there’s no room for non-determinism to creep in.

Fabric’s determinism-by-endorsement has a number of interesting consequences. First, chaincode can be written in many different programming languages, since these don’t need to be adapted for determinism (Go, Java and JavaScript are currently supported). Second, chaincode can be hidden from some of a blockchain’s participants, since it only needs to be executed by clients and endorsers (the database itself is globally visible). Finally and most notably, Fabric chaincode can do things that are forbidden in other blockchain environments, such as checking the weather using an online web API. In the worst case, where every endorser gets a different answer from this API, the client will fail to obtain enough endorsements for any particular outcome, and no transaction will take place. (It should be noted that Fabric team members still recommend using deterministic logic inside chaincode, in order to avoid surprises.)

What price does Fabric pay for this flexibility? If the purpose of a blockchain is to remove intermediaries from a shared database-driven application, then Fabric’s reliance on endorsers takes a big step away from that goal. For the participants in the chain, it is no longer enough to follow the chaincode’s rules – they also need certain other nodes to agree that they have done so. Even worse, a malicious subset of endorsers could approve database changes that do not follow chaincode at all. This gives endorsers much more power than the validators in regular blockchains, who can censor transactions but cannot violate the blockchain’s rules. Blockchain application developers must decide whether this trade-off makes sense in their particular case.

Conflict prevention

So far, we’ve discussed how different blockchain platforms express transaction rules in code, and how they deterministically ensure that every node applies those rules identically. Now it’s time to talk about a third aspect of our showdown: How does each platform deal with the possibility that two transactions, which are valid in and of themselves, conflict with each other? In the simplest example, imagine that Alice has $10 in a financial ledger and broadcasts two transactions – one sending $8 to Bob, and the other sending $7 to Charlie. Clearly, only one of these transactions can be allowed to succeed.

Two models

We can begin by grouping MultiChain’s and Corda’s approach to this problem together. As described earlier, both of these use an input–output model for representing transactions and their rules, in which each transaction input spends a previous transaction output. This leads to a simple principle for preventing conflicts: Every output can only be spent once. MultiChain filters and Corda contracts can rely on their respective platforms to enforce this restriction absolutely. Since Alice’s $10 is represented by a previous transaction output, this single-spend rule automatically stops her sending it to both Bob and Charlie.

Despite this similarity, it’s important to point out a key difference in how MultiChain and Corda prevent conflicts. In MultiChain, every node sees every transaction and so can independently verify that each output is only spent once. Any transaction which performs a double spend against a previously confirmed transaction will be instantly and automatically rejected. By contrast, in Corda there is no global blockchain, so “notaries” are required to prevent these double spends. Every Corda output state is assigned to a notary, who has to sign any transaction spending that output, confirming it has not been spent before. A blockchain’s participants must trust notaries to follow this rule honestly, and malicious notaries can cause havoc at will. As with endorsements in Fabric, this “single-spend as a service” design has advantages in terms of confidentiality but reintroduces intermediaries, going against the blockchain grain. (It’s important to clarify that Corda notaries can be run by groups of participants using a consensus algorithm, so the integrity of the ledger can still be protected against individual bad actors).

Let’s move on to Ethereum. To recall, Ethereum uses contracts and messages rather than inputs and outputs. As a result, transaction conflicts such as Alice’s two payments are not immediately visible to the blockchain engine. Instead, they are detected and blocked by the contract which processes the transactions, after their order is confirmed on the chain. When processing each of Alice’s payments, the contract verifies whether her balance is sufficient. If the transaction paying $8 to Bob comes first, it will be processed as usual, leaving Alice with $2 in her account. As a result, when the contract processes the second transaction paying $7 to Charlie, it sees that Alice lacks the necessary funds and the transaction aborts.

Outputs vs contracts

So far we’ve seen two different techniques for preventing conflicting transactions – single-spend outputs in MultiChain and Corda, and contract-based verification in Ethereum. So which is better?

In order to help answer this question, let’s consider an example “1-of-2 multisignature” account which holds $100 on behalf of Gavin and Helen, and allows either of them to spend that money independently. Gavin instructs his application to pay $80 to Donna, and a few seconds later, Helen wants to send $40 to Edward. Since there are insufficient funds for both payments, these transactions would inevitably conflict. In the event that both transactions are broadcast, the outcome will be determined by whichever makes it first into the chain. Note that unlike Alice’s example, this conflict is accidental, since no one is trying to break the application’s rules – they simply had unlucky timing.

In considering the likelihood of this conflict occurring, the key question is this: After Gavin sends out his transaction, how long will it take Helen’s node to know that her payment might fail? The shorter this period is, the more likely Helen is to be stopped from attempting that payment, saving her and her application from a subsequent surprise.

With the input–output model, any conflict between transactions is directly visible to the blockchain platform, since the two transactions will be explicitly attempting to spend the same previous output. In MultiChain, this happens as soon as Gavin’s transaction has propagated to Helen’s node, usually in a second or less. In Corda, the output’s notary will refuse the request to sign Helen’s transaction, since it has already signed Gavin’s, so Helen will instantly know that her payment will fail. (Although if the Corda notary is itself distributed, she may have to wait a few seconds for a reply.) Either way, there is no need to wait for a transaction to be confirmed and ordered in the blockchain.

What about Ethereum’s model? In this case, there is no immediate way for the blockchain platform to know that a conflict will occur. While Helen’s node may see Gavin’s transaction on the network, it cannot know how this will affect Helen’s own transaction, since from its perspective these are simply two messages being sent to the same contract. Perhaps ten seconds later, once the final ordering of the conflicting transactions is confirmed on the blockchain, Helen’s node will recalculate the actual instead of the expected outcome, and her application will update its display accordingly. In the meantime, both Gavin and Helen will be left in the dark.

But we shouldn’t conclude from this that the input–output model always works best. Consider a variation on our example scenario, where both Gavin and Helen request smaller $40 payments from the original balance of $100, at exactly the same time. In the input–output model these transactions would conflict, since they are both spending the same database row containing that $100, and only one of the payments would succeed. But in Ethereum, both transactions would be successfully processed, irrespective of their final order, since the account contains sufficient funds for both. In this case, Ethereum more faithfully fulfills Gavin’s and Helen’s intentions.

Read-write sets

Finally, let’s talk about Fabric, whose endorsement-based approach is a hybrid of these two techniques. As explained earlier, when a Fabric “client” node wants to send a message to a contract, it first asks some endorsing nodes to execute that message on its behalf. The endorsing nodes do so in a similar way to Ethereum – running the contract against their local database – but this process is observed rather than immediately applied. Each endorser records the set of rows that would be read and written, noting also the exact version of those rows at that point in time. This “read-write set” of versioned rows is explicitly referenced in the endorsement, and included in the transaction which the client broadcasts.

Conflicts between Fabric transactions are resolved once their order is finalized in the chain. Every node processes each transaction independently, checking endorsement policies and applying the database changes specified. However, if a transaction reads or writes a database row version that has already been modified by a previous transaction, then that second transaction is ignored. To go back to Alice’s conflicting payments to Bob and Charlie, both of these transactions will read and modify the same row version, containing the $10 with which Alice started. So the second transaction will be safely and automatically aborted.

Fabric’s approach to conflict resolution works just fine, but in terms of performance and flexibility it combines the worst of the previous two models. Because endorsements convert transactions into specific read-write sets, Gavin and Helen’s simultaneous but compatible $40 payments would lead to a conflict that Ethereum avoids. However, Fabric does not gain the speed advantage of the input–output model, since endorsers execute contracts against the most recent version of the database confirmed by the blockchain, ignoring unconfirmed transactions. So if Helen initiates her payment a few seconds after Gavin, but before Gavin’s has been confirmed on the blockchain, Fabric will create conflicting transactions that a pure input–output model avoids.

A complex choice

In this piece, we reviewed many of the different ways in which Corda, Ethereum, Fabric and MultiChain address the key challenges of “smart contracts”, or application code that is embedded in a blockchain. And each platform has different answers to our three core questions: How are transaction rules represented? How is code executed deterministically? And how do we prevent conflicts?

So who is the winner of our smart contract showdown? It should be obvious by now that there is no simple answer. Each platform represents a complex multi-way trade-off between flexibility, simplicity, performance, disintermediation, safety and confidentiality. So the choice of platform for a particular application has to begin with a detailed understanding of that application’s trust model, the types of transactions it involves, and their likely patterns of conflict. If you find someone pushing a specific smart contract solution before they know the answers to these questions, I suggest politely but firmly insisting that they adopt a “smarter” approach.

Please post any comments on LinkedIn.

By now it’s clear that many blockchain use cases have nothing to do with financial transactions. Instead, the chain’s purpose is to enable the decentralized aggregation, ordering, timestamping and archiving of any type of information, including structured data, correspondence or documentation. The blockchain’s core value is enabling its participants to provably and permanently agree on exactly what data was entered, when and by whom, without relying on a trusted intermediary. For example, SAP’s recently launched blockchain platform, which supports MultiChain and Hyperledger Fabric, targets a broad range of supply chain and other non-financial applications.

The simplest way to use a blockchain for recording data is to embed each piece of data directly inside a transaction. Every blockchain transaction is digitally signed by one or more parties, replicated to every node, ordered and timestamped by the chain’s consensus algorithm, and stored permanently in a tamper-proof way. Any data within the transaction will therefore be stored identically but independently by every node, along with a proof of who wrote it and when. The chain’s users are able to retrieve this information at any future time.

For example, MultiChain 1.0 allowed one or more named “streams” to be created on a blockchain and then used for storing and retrieving raw data. Each stream has its own set of write permissions, and each node can freely choose which streams to subscribe to. If a node is subscribed to a stream, it indexes that stream’s content in real-time, allowing items to be retrieved quickly based on their ordering, timestamp, block number or publisher address, as well as via a “key” (or label) by which items can be tagged. MultiChain 2.0 (since alpha 1) extended streams to support Unicode text or JSON data, as well as multiple keys per item and multiple items per transaction. It also added summarization functions such as “JSON merge” which combine items with the same key or publisher in a useful way.

Confidentiality and scalability

While storing data directly on a blockchain works well, it suffers from two key shortcomings – confidentiality and scalability. To begin with confidentiality, the content of every stream item is visible to every node on the chain, and this is not necessarily a desirable outcome. In many cases a piece of data should only be visible to a certain subset of nodes, even if other nodes are needed to help with its ordering, timestamping and notarization.

Confidentiality is a relatively easy problem to solve, by encrypting information before it is embedded in a transaction. The decryption key for each piece of data is only shared with those participants who are meant to see it. Key delivery can be performed on-chain using asymmetric cryptography (as described here) or via some off-chain mechanism, as is preferred. Any node lacking the key to decrypt an item will see nothing more than binary gibberish.

Scalability, on the other hand, is a more significant challenge. Let’s say that any decent blockchain platform should support a network throughput of 500 transactions per second. If the purpose of the chain is information storage, then the size of each transaction will depend primarily on how much data it contains. Each transaction will also need (at least) 100 bytes of overhead to store the sender’s address, digital signature and a few other bits and pieces.

If we take an easy case, where each item is a small JSON structure of 100 bytes, the overall data throughput would be 100 kilobytes per second, calculated from 500 × (100+100). This translates to under 1 megabit/second of bandwidth, which is comfortably within the capacity of any modern Internet connection. Data would accumulate at a rate of around 3 terabytes per year, which is no small amount. But with 12 terabyte hard drives now widely available, and RAID controllers which combine multiple physical drives into a single logical one, we could easily store 10-20 years of data on every node without too much hassle or expense.

However, things look very different if we’re storing larger pieces of information, such as scanned documentation. A reasonable quality JPEG scan of an A4 sheet of paper might be 500 kilobytes in size. Multiply this by 500 transactions per second, and we’re looking at a throughput of 250 megabytes per second. This translates to 2 gigabits/second of bandwidth, which is faster than most local networks, let alone connections to the Internet. At Amazon Web Services’ cheapest published price of $0.05 per gigabyte, it means an annual bandwidth bill of $400,000 per node. And where will each node store the 8000 terabytes of new data generated annually?

It’s clear that, for blockchain applications storing many large pieces of data, straightforward on-chain storage is not a practical choice. To add insult to injury, if data is encrypted to solve the problem of confidentiality, nodes are being asked to store a huge amount of information that they cannot even read. This is not an attractive proposition for the network’s participants.

The hashing solution

So how do we solve the problem of data scalability? How can we take advantage of the blockchain’s decentralized notarization of data, without replicating that data to every node on the chain?

The answer is with a clever piece of technology called a “hash”. A hash is a long number (think 256 bits, or around 80 decimal digits) which uniquely identifies a piece of data. The hash is calculated from the data using a one-way function which has an important cryptographic property: Given any piece of data, it is easy and fast to calculate its hash. But given a particular hash, it is computationally infeasible to find a piece of data that would generate that hash. And when we say “computationally infeasible”, we mean more calculations than there are atoms in the known universe.

Hashes play a crucial role in all blockchains, by uniquely identifying transactions and blocks. They also underlie the computational challenge in proof-of-work systems like bitcoin. Many different hash functions have been developed, with gobbledygook names like BLAKE2, MD5 and RIPEMD160. But in order for any hash function to be trusted, it must endure extensive academic review and testing. These tests come in the form of attempted attacks, such as “preimage” (finding an input with the given hash), “second preimage” (finding a second input with the same hash as the given input) and “collision” (finding any two different inputs with the same hash). Surviving this gauntlet is far from easy, with a long and tragic history of broken hash functions proving the famous maxim: “Don’t roll your own crypto.”

To go back to our original problem, we can solve data scalability in blockchains by embedding the hashes of large pieces of data within transactions, instead of the data itself. Each hash acts as a “commitment” to its input data, with the data itself being stored outside of the blockchain or “off-chain”. For example, using the popular SHA256 hash function, a 500 kilobyte JPEG image can be represented by a 32-byte number, a reduction of over 15,000×. Even at a rate of 500 images per second, this puts us comfortably back in the territory of feasible bandwidth and storage requirements, in terms of the data stored on the chain itself.

Of course, any blockchain participant that needs an off-chain image cannot reproduce it from its hash. But if the image can be retrieved in some other way, then the on-chain hash serves to confirm who created it and when. Just like regular on-chain data, the hash is embedded inside a digitally signed transaction, which was included in the chain by consensus. If an image file falls out of the sky, and the hash for that image matches a hash in the blockchain, then the origin and timestamp of that image is confirmed. So the blockchain is providing exactly the same value in terms of notarization as if the image was embedded in the chain directly.

A question of delivery

So far, so good. By embedding hashes in a blockchain instead of the original data, we have an easy solution to the problem of scalability. Nonetheless, one crucial question remains:

How do we deliver the original off-chain content to those nodes which need it, if not through the chain itself?

This question has several possible answers, and we know of MultiChain users applying them all. One basic approach is to set up a centralized repository at some trusted party, where all off-chain data is uploaded then subsequently retrieved. This system could naturally use “content addressing”, meaning that the hash of each piece of data serves directly as its identifier for retrieval. However, while this setup might work for a proof-of-concept, it doesn’t make sense for production, because the whole point of a blockchain is to remove trusted intermediaries. Even if on-chain hashes prevent the intermediary from falsifying data, it could still delete data or fail to deliver it to some participants, due to a technical failure or the actions of a rogue employee.

A more promising possibility is point-to-point communication, in which the node that requires some off-chain data requests it directly from the node that published it. This avoids relying on a trusted intermediary, but suffers from three alternative shortcomings:

• It requires a map of blockchain addresses to IP addresses, to enable the consumer of some data to communicate directly with its publisher. Blockchains can generally avoid this type of static network configuration, which can be a problem in terms of failover and privacy.
• If the original publisher node has left the network, or is temporarily out of service, then the data cannot be retrieved by anyone else.
• If a large number of nodes are interested in some data, then the publisher will be overwhelmed by requests. This can create severe network congestion, slow the publisher’s system down, and lead to long delays for those trying to retrieve that data.

In order to avoid these problems, we’d ideally use some kind of decentralized delivery mechanism. Nodes should be able to retrieve the data they need without relying on any individual system – be it a centralized repository or the data’s original publisher. If multiple parties have a piece of data, they should share the burden of delivering it to anyone else who wants it. Nobody needs to trust an individual data source, because on-chain hashes can prove that data hasn’t been tampered with. If a malicious node delivers me the wrong data for a hash, I can simply discard that data and try asking someone else.

For those who have experience with peer-to-peer file sharing protocols such as Napster, Gnutella or BitTorrent, this will all sound very familiar. Indeed, many of the basic principles are the same, but there are two key differences. First, assuming we’re using our blockchain in an enterprise context, the system runs within a closed group of participants, rather than the Internet as a whole. Second, the blockchain adds a decentralized ordering, timestamping and notarization backbone, enabling all users to maintain a provably consistent and tamper-resistant view of exactly what happened, when and by whom.

How might a blockchain application developer achieve this decentralized delivery of off-chain content? One common choice is to take an existing peer-to-peer file sharing platform, such as the amusingly-named InterPlanetary File System (IPFS), and use it together with the blockchain. Each participant runs both a blockchain node and an IPFS node, with some middleware coordinating between the two. When publishing off-chain data, this middleware stores the original data in IPFS, then creates a blockchain transaction containing that data’s hash. To retrieve some off-chain data, the middleware extracts the hash from the blockchain, then uses this hash to fetch the content from IPFS. The local IPFS node automatically verifies the retrieved content against the hash to ensure it hasn’t been changed.

While this solution is possible, it’s all rather clumsy and inconvenient. First, every participant has to install, maintain and update three separate pieces of software (blockchain node, IPFS node and middleware), each of which stores its data in a separate place. Second, there will be two separate peer-to-peer networks, each with its own configuration, network ports, identity system and permissioning (although it should be noted that IPFS doesn’t yet support closed networks). Finally, tightly coupling IPFS and the blockchain together would make the middleware increasingly complex. For example, if we want the off-chain data referenced by some blockchain transactions to be instantly retrieved (with automatic retries), the middleware would need to be constantly up and running, maintaining its own complex state. Wouldn’t it be nice if the blockchain node did all of this for us?

Off-chain data in MultiChain 2.0

Today we’re delighted to release the third preview version (alpha 3) of MultiChain 2.0, with a fully integrated and seamless solution for off-chain data. Every piece of information published to a stream can be on-chain or off-chain as desired, and MultiChain takes care of everything else.

No really, we mean everything. As a developer building on MultiChain, you won’t have to worry about hashes, local storage, content discovery, decentralized delivery or data verification. Here’s what happens behind the scenes:

1. The publishing MultiChain node writes the new data in its local storage, slicing large items into chunks for easy digestion and delivery.
2. The transaction for publishing off-chain stream items is automatically built, containing the chunk hash(es) and size(s) in bytes.
3. This transaction is signed and broadcast to the network, propagating between nodes and entering the blockchain in the usual way.
4. When a node subscribed to a stream sees a reference to some off-chain data, it adds the chunk hashes for that data to its retrieval queue. (When subscribing to an old stream, a node also queues any previously published off-chain items for retrieval.)
5. As a background process, if there are chunks in a node’s retrieval queue, queries are sent out to the network to locate those chunks, as identified by their hashes.
6. These chunk queries are propagated to other nodes in the network in a peer-to-peer fashion (limited to two hops for now – see technical details below).
7. Any node which has the data for a chunk can respond, and this response is relayed to the subscriber back along the same path as the query.
8. If no node answers the chunk query, the chunk is returned back to the queue for later retrying.
9. Otherwise, the subscriber chooses the most promising source for a chunk (based on hops and response time), and sends it a request for that chunk’s data, again along the same peer-to-peer path as the previous response.
10. The source node delivers the data requested, using the same path again.
11. The subscriber verifies the data’s size and hash against the original request.
12. If everything checks out, the subscriber writes the data to its local storage, making it immediately available for retrieval via the stream APIs.
13. If the requested content did not arrive, or didn’t match the desired hash or size, the chunk is returned back to the queue for future retrieval from a different source.

Most importantly, all of this happens extremely quickly. In networks with low latency, small pieces of off-chain data will arrive at subscribers within a split second of the transaction that references them. And for high load applications, our testing shows that MultiChain 2.0 alpha 3 can sustain a rate of over 1000 off-chain items or 25 MB of off-chain data retrieved per second, on a mid-range server (Core i7) with a decent Internet connection. Everything works fine with off-chain items up to 1 GB in size, far beyond the 64 MB limit for on-chain data. Of course, we hope to improve these numbers further as we spend time optimizing MultiChain 2.0 during its beta phase.

When using off-chain rather than on-chain data in streams, MultiChain application developers have to do exactly two things:

• When publishing data, pass an “offchain” flag to the appropriate APIs.
• When using the stream querying APIs, consider the possibility that some off-chain data might not yet be available, as reported by the “available” flag. While this situation will be rare under normal circumstances, it’s important for application developers to handle it appropriately.

Of course, to prevent every node from retrieving every off-chain item, items should be grouped together into streams in an appropriate way, with each node subscribing to those streams of interest.

On-chain and off-chain items can be used within the same stream, and the various stream querying and summarization functions relate to both types of data identically. This allows publishers to make the appropriate choice for every item in a stream, without affecting the rest of an application. For example, a stream of JSON items about people’s activities might use off-chain data for personally identifying information, and on-chain data for the rest. Subscribers can use MultiChain’s JSON merging to combine both types of information into a single JSON for reading.

If you want to give off-chain stream items a try, just follow MultiChain’s regular Getting Started tutorial, and be sure not to skip section 5.

So what’s next?

With seamless support for off-chain data, MultiChain 2.0 will offer a big step forwards for blockchain applications focused on large scale data timestamping and notarization. In the longer term, we’re already thinking about a ton of possible future enhancements to this feature for the Community and/or Enterprise editions of MultiChain:

• Implementing stream read permissions using a combination of off-chain items, salted hashes, signed chunk queries and encrypted delivery.
• Allowing off-chain data to be explicitly “forgotten”, both voluntarily by individual nodes, or by all nodes in response to an on-chain message.
• Selective stream subscriptions, in which nodes only retrieve the data for off-chain items with particular publishers or keys.
• Using merkle trees to enable a single on-chain hash to represent an unlimited number of off-chain items, giving another huge jump in terms of scalability.
• Pluggable storage engines, allowing off-chain data to be kept in databases or external file systems rather than local disk.
• Nodes learning over time where each type of off-chain data is usually available in a network, and focusing their chunk queries appropriately.

We’d love to hear your feedback on the list above as well as off-chain items in general. With MultiChain 2.0 still officially in alpha, there’s plenty of time to enhance this feature before its final release.

In the meantime, we’ve already started work on “Smart Filters”, the last major feature planned for MultiChain 2.0 Community. A Smart Filter is a piece of code embedded in the blockchain which implements custom rules for validating data or transactions. Smart Filters have some similarities with “smart contracts”, and can do many of the same things, but have key differences in terms of safety and performance. We look forward to telling you more in due course.

Please post any comments on LinkedIn.

Technical details

While off-chain stream items in MultiChain 2.0 are simple to use, they contain many design decisions and additional features that may be of interest. The list below will mainly be relevant for developers building blockchain applications, and can be skipped by less technical types:

• Per-stream policies. When a MultiChain stream is created, it can optionally be restricted to allow only on-chain or off-chain data. There are several possible reasons for doing this, rather than allowing each publisher to decide for themselves. For example, on-chain items offer an ironclad availability guarantee, whereas old off-chain items may become irretrievable if their publisher and other subscribers drop off the network. On the flip side, on-chain items cannot be “forgotten” without modifying the blockchain, while off-chain items are more flexible. This can be important in terms of data privacy rules, such as Europe’s new GDPR regulations.
• On-chain metadata. For off-chain items, the on-chain transaction still contains the item’s publisher(s), key(s), format (JSON, text or binary) and total size. All this takes up very little space, and helps application developers determine whether the unavailability of an off-chain item is of concern for a particular stream query.
• Two-hop limit. When relaying chunk queries across the peer-to-peer network, there is a trade-off between reachability and performance. While it would be nice for every query to be propagated along every single path, this can clog the network with unnecessary “chatter”. So for now chunk queries are limited to two hops, meaning that a node can retrieve off-chain data from any peer of its peers. In the smaller networks of under 1000 nodes that tend to characterize enterprise blockchains, we believe this will work just fine, but it’s easy for us to adjust this constraint (or offer it as a parameter) if we turn out to be wrong.
• Local storage. Each MultiChain node stores off-chain data within the “chunks” directory of its regular blockchain directory, using an efficient binary format and LevelDB index. A separate subdirectory is used for the items in each of the subscribed streams, as well as those published by the node itself. Within each of these subdirectories, duplicate chunks (with the same hash) are only stored once. When a node unsubscribes from a stream, it can choose whether or not to purge the off-chain data retrieved for that stream.
• Binary cache. When publishing large pieces of binary data, whether on-chain or off-chain, it may not be practical for application developers to send that data to MultiChain’s API in a single JSON-RPC request. So MultiChain 2.0 implements a binary cache, which enables large pieces of data to be built up over multiple API calls, and then published in a brief final step. Each item in the binary cache is stored as a simple file in the “cache” subdirectory of the blockchain directory, allowing gigabytes of data to also be pushed directly via the file system.
• Monitoring APIs. MultiChain 2.0 alpha 3 adds two new APIs for monitoring the asynchronous retrieval of off-chain data. The first API describes the current state of the queue, showing how many chunks (and how much data) are waiting or being queried or retrieved. The second API provides aggregate statistics for all chunk queries and requests sent since the node started up, including counts of different types of failure.
• Flush on publish. When publishing an off-chain item, MultiChain ensures that its local copy of the data is fully written (or “flushed”) to the physical disk drive before the transaction referencing that data is broadcast to the network. Otherwise, if the node was unlucky enough to lose power immediately after broadcasting the transaction, the off-chain data could be permanently lost. This isn’t an issue for MultiChain itself, since the delays between a chunk’s retrieval attempts grow automatically over time. But it could cause problems at the application level, where everyone knows of the existence of some data but nobody is able to find it.
• Publishing performance. By flushing off-chain data to disk in this way, MultiChain can incur a performance penalty, since physical disks are slow. For example, a mid-range 7200 rpm hard drive can only perform around 100 random data writes per second, limiting in turn the rate at which an individual node can publish off-chain items. There are three possible workarounds for this problem. First and most simply, nodes can use a solid state device (SSD) drive instead of a regular hard drive, which supports 10,000s of random write operations per second. Second, multiple off-chain items can be published in a single transaction using the “createrawsendfrom” API. In this case, MultiChain writes all the off-chain data referenced by a transaction in a single disk operation. Finally, MultiChain can be configured not to flush off-chain data to disk before broadcasting the transaction which references it. Use this option with care.
• Native currency integration. For use cases which require it, MultiChain has always offered the option of using a native currency on a blockchain to prevent transaction spam and/or incentivize block validators (“miners”). In these cases, transactions must offer miners a minimum fee which is proportional to their size in bytes, in order to be relayed and confirmed on the chain. This mechanism has been extended to allow off-chain spam to be prevented, by requiring a minimum additional fee per kilobyte of off-chain data referenced in a transaction.
• Archive nodes. If a node wishes to subscribe to every stream, and therefore retrieve and store every off-chain item published, it can be configured to do so using the “autosubscribe” runtime parameter. Any such node will act as a backup for the entire network, guaranteeing that off-chain items will not be lost or unavailable, no matter which other nodes disappear. One can imagine third party companies offering this as a commercial service.

Full details of all the relevant API calls and parameters can be found on the MultiChain 2.0 preview page.

As time goes on, the blockchain world has been separating into two distinct parts. On one hand, public blockchains with their associated cryptocurrencies have enjoyed a remarkable recent comeback, minting many a multi-millionaire. On the other hand, use of permissioned or enterprise blockchains has been growing quietly but steadily, seeing their first live deployments across multiple industries during 2017.

One interesting question to consider is the appropriate level of similarity between these two types of chain. Both implement a shared database using peer-to-peer networking, public–private key cryptography, transaction rules and consensus mechanisms that can survive malicious actors. That’s a great deal of common ground. Nonetheless, public and private blockchains have different requirements in terms of confidentiality, scalability and governance. Perhaps these differences point to the need for radically divergent designs.

The Corda platform, developed by the R3 banking consortium, adopts a clear stance on this question. While some aspects were inspired by public blockchains, Corda was designed from scratch based on the needs of R3’s members. Indeed, although R3 still uses the word “blockchain” extensively to help market their product, Corda has no chain of blocks at all. More than any other “distributed ledger” platform I’m aware of, Corda departs radically from the architecture of conventional blockchains.

My goal in this piece is to explain these differences and discuss their implications, for good and bad. Actually, good and bad is the wrong way to put it, because the more interesting question is “Good and bad for what?” This article is far from short. But by the end of it, I hope that readers will gain some understanding of the differences in Corda and their consequent trade-offs. Corda is important because its design decisions bring many of the dilemmas of enterprise blockchains into sharp relief.

One last thing before we dive in. As the CEO of the company behind MultiChain, a popular enterprise blockchain platform, why am I writing in such depth about a supposedly competing product? The standard reason would be to argue for MultiChain’s superiority, but that’s not my motivation here. In fact, I do not see Corda and MultiChain as competitors, because they are fundamentally different in terms of design, architecture and audience. Corda and MultiChain compete in the same way as cruise liners and jet skis – while both transport people by sea, there are almost no real-world situations in which both could be used.

On a more personal note, I’ve learned a great deal from Corda’s technical leadership over the past few years, whether through meetings, correspondence or their public writings, much of which occurred before they joined R3. Some of my interest in Corda stems from the respect I have for this team, and for this reason alone, Corda is worth studying for anyone seeking an understanding of the distributed ledger field.

Introducing blockchains

In order to understand Corda, it’s helpful to start with conventional blockchains. The purpose of a blockchain is to enable a database or ledger to be directly and safely shared by non-trusting parties. This contrasts with centralized databases, which are stored and controlled by a single organization. A blockchain has multiple “nodes”, each of which stores a copy of the database and can belong to a different organization. Nodes connect to each other in a dense peer-to-peer fashion, using a “gossip protocol” in which each node is constantly telling its peers everything it learns. As a result, any node can rapidly broadcast a message to the entire network via many alternative paths.

A database, whether centralized or blockchain-powered, begins in an empty state, and is updated via “transactions”. A transaction is defined as a set of database changes which are “atomic”, meaning that they succeed or fail as a whole. Imagine a database representing a financial ledger, with one row per account. A transaction in which Alice pays $10 to Bob has three steps: (1) verify that Alice’s account contains at least $10, (2) subtract $10 from Alice’s account, and (3) add $10 to Bob’s account. As a basic requirement, any database platform must ensure that no transaction interferes with another. This “isolation” is achieved by locking the rows for both Alice and Bob while the payment is under way. Any other transaction involving these rows must wait until this one is finished.

In a blockchain, every node independently processes every transaction on its own copy of the database. Transactions are created anywhere on the network and automatically propagated to all other nodes. Since the organizations running nodes may have different (or even conflicting) interests, they cannot trust each other to transact fairly. Blockchains therefore need rules which define whether or not a particular transaction is valid. In a shared financial ledger, these rules prevent users from spending each other’s money, or conjuring funds from thin air.

Along with the rules that determine transaction validity, blockchains must also define how transactions will be ordered, since in many cases this ordering is critical. If Alice has $15 and tries to send $10 to both Bob and Charlie in two separate transactions, only one of these payments can succeed. While we might like to say that the first transaction takes precedence, a peer-to-peer network has no objective definition of “first”, since messages can arrive at different nodes in different orders.

Transaction rules

In a general sense, the information in any database is separated into records or “rows”, and a transaction can do three different things: delete rows, create rows, and/or modify rows. These can be reduced further to two, since modifying a row is equivalent to deleting that row and creating a new one in its place. To go back to Alice’s payment to Bob, her row containing $15 is deleted, and two new rows are created – one containing $10 for Bob and the other with $5 in “change” for Alice.

Following bitcoin’s and Corda’s terminology, we denote the rows deleted by a transaction as its “inputs”, and those created as its “outputs”. Any row deleted by a transaction must have been created by a previous transaction. Therefore each transaction input consumes (or “spends”) a previous transaction’s output. The up-to-date content of the database is defined by the set of “unspent transaction outputs” or “UTXOs”.

In a blockchain, a transaction is valid if it fulfills the following three conditions:

1. Correctness. The transaction must represent a legitimate transformation from inputs to outputs. For example, in a financial ledger, the total quantity of funds in the inputs must match the total in the outputs, to prevent money from magically appearing or disappearing. The only exceptions are special “issuance” or “retirement” transactions, in which funds are explicitly added or removed.
2. Authorization. The transaction must be authorized by the owner of every output consumed by its inputs. In a financial ledger, this prevents participants from spending each other’s money without permission. Transaction authorization is managed using asymmetric (or public–private key) cryptography. Every row has an owner, identified by a public key, whose corresponding private key is kept secret. In order to be authorized, a transaction must be digitally signed by the owner of each of its inputs. (Note that rows can also have more complex “multisignature” owners, for example where any two out of three parties can authorize their use.)
3. Uniqueness. If a transaction consumes a particular output, then no other transaction can consume that output again. This is how we prevent Alice from making conflicting payments to both Bob and Charlie. While the transactions for both of these payments could be correct and authorized, the uniqueness rule ensures that only one will be processed by the database.

In a conventional blockchain, every node checks every transaction in terms of these three rules. Later on, we’ll see how Corda divides up this responsibility differently.

Building blocks

A blockchain is literally a chain of blocks, in which every block links to the previous one via a “hash” that uniquely identifies its contents. Each block contains an ordered set of transactions which must not conflict with each other or with those in previous blocks, as well as a timestamp and some other information. Just like transactions, blocks propagate rapidly across the network and are independently verified by every node. Once a transaction appears in a block, it is “confirmed”, leading nodes to reject any conflicting transaction.

Who is responsible for creating these blocks, and how can we be sure that all nodes will agree on the authoritative chain? This question of “consensus algorithms” is a huge subject in itself, filled with wondrous acronyms such as PoW (Proof of Work), PBFT (Practical Byzantine Fault Tolerance) and DPoS (Delegated Proof of Stake). We won’t be getting into all that here. Suffice to say that permissioned blockchains for enterprises use some kind of voting scheme, where votes are granted to “validator nodes” who are collectively responsible. The scheme ensures that, so long as a good majority of validator nodes are functioning correctly and honestly, transactions will enter the chain in a (close to) fair order, timestamps will be (approximately) correct, and confirmed transactions cannot be subsequently reversed.

Before discussing some of the challenges of blockchains, I’d like to clarify three additional points. First, while I am using a financial ledger by example throughout this piece, the input–output model of transactions supports a much broader variety of use cases. Each row can contain a rich data object (think JSON) containing many different types of information – indeed, Corda uses the word “state” rather than “row” for this reason. Richer states change nothing fundamental about transaction rules: correctness is still defined in terms of inputs and outputs, authorization is still required for every input, and uniqueness ensures that each output can only be spent once.

Second, there are many blockchain use cases in which rows are only created in the database, and never deleted. These applications relate to general data storage, timestamping and notarization, rather than maintaining some kind of ledger which is in flux. In these data-only applications, transactions add data in their outputs but consume none in their inputs, allowing the rules for correctness, authorization and uniqueness to be simplified. Although data-only use cases are an increasing focus of our own development at MultiChain, I only mention them in passing here, since Corda was clearly not designed with them in mind.

Finally, it’s worth noting that some blockchain platforms do not use an input–output model. Ethereum presents an alternative paradigm, in which the chain controls a virtual computer with a global state that is managed by “contracts”, and transactions do not connect to each other explicitly. A discussion of Ethereum’s model in permissioned blockchains is beyond our scope here, but see this article for a detailed explanation and critique. One key advantage of the input–output paradigm is that most transactions can be processed in parallel and independently of each other. This property is crucial for Corda, as we’ll see later on.

Blockchain challenges

Let’s imagine that the world’s banks created a shared ledger to represent the ownership, transfer and exchange of a variety of financial assets. In theory, this could be implemented on a regular blockchain, as described above. Each row would contain three columns – an asset identifier such as GOOG or USD, the quantity owned, and the owner’s public key. Each transaction would transfer one or more assets from its inputs to its outputs, with special cases for issuance and retirement.

Every bank in the network would run one or more nodes which connect to the others, propagating and verifying transactions. Senior members would act as validators, with the collective responsibility of confirming, ordering and timestamping transactions. Any validator’s misbehavior would be visible to all the nodes in the network, leading to censure, banishment and/or legal proceedings. With all this in place, any financial asset could be moved across the world in seconds, with the rules of correctness, authorization and uniqueness guaranteeing the ledger’s integrity.

What’s wrong with this picture? Actually, there are three problems: scalability, confidentiality and interoperability. The issue of scalability is simple enough. Our proposed interbank blockchain would require every member to verify, process and store every transaction performed by every bank in the world. Even if this would be technically feasible for the largest financial institutions, the cost of computation and storage would create a significant barrier for many. Surely we’d prefer a system in which participants only see those transactions in which they are immediately involved.

But let’s put scalability aside, since it can ultimately be solved using expensive computers and clever engineering. A more fundamental issue is confidentiality. While it might sound utopian for every transaction to be visible everywhere, in the real world such radical transparency is a non-starter in terms of competition and regulation. If J.P. Morgan and HSBC exchange a pair of assets, they’re unlikely to want Citi and the Bank of China to see what they did. If the transaction was conducted on behalf of these banks’ customers, it could be illegal for them to expose it in this way.

One proposed solution to the problem of confidentiality is “channels”, as implemented in Hyperledger Fabric. Each channel has certain members, who are a subset of the nodes in the network as a whole. A channel’s transactions are visible only to its members, so that each channel effectively acts as a separate blockchain. While this does help with confidentiality, it also undermines the entire point of the exercise. Assets cannot be moved from one channel to another without the help of a trusted intermediary which is active on both. The difficulty of this approach was recently highlighted by SWIFT’s reconciliation proof-of-concept, which estimated that over 100,000 channels would be needed in production. That’s 100,000 islands between which assets cannot be directly moved.

In data-only use cases, where transactions do not consume data in inputs, the confidentiality problem can be sidestepped by encrypting or hashing the data in outputs, and delivering the decryption key or unhashed data outside of the chain. But for a transaction whose inputs consume other transactions’ outputs, every node has to see those inputs and outputs in order to validate the transaction. While advanced cryptographic techniques such as confidential assets and zero knowledge proofs have been developed to partially or completely solve this problem for financial ledgers, these impose a significant performance burden and/or cannot be generalized to any correctness rule.

Finally, let’s talk about interoperability. In an ideal world, every bank would immediately join our global blockchain on the day it was launched. In reality however, multiple blockchains would be adopted by different groups of banks, based on geography or pre-existing relationships. Over time, a member of one group might wish to start transacting with a member of another, by transferring an asset between chains. Just as with channels, this can only be achieved with the help of a trusted intermediary, defeating the blockchain’s purpose.

Corda aims to solve these interrelated problems of scalability, confidentiality and interoperability via a radical rethink of how distributed ledgers work.

Corda’s partial view

The fundamental difference in Corda is easy to explain: Each node only sees some, rather than all, of the transactions processed on the network. While a single logical and conceptual ledger is defined by all these transactions, no individual node sees that ledger in its entirety. To draw a comparison, at any point in time, every dollar bill in the world is in a particular place, but nobody knows where they all are.

So which transactions does a Corda node see? First of all, those in which it is directly involved, because it owns one of that transaction’s inputs or outputs. In a financial ledger, this includes every transaction in which a node is sending or receiving funds. Let’s say Alice creates a transaction which consumes her $15 in an input and has two outputs – one with $10 for me, and the other with $5 in “change” for her. After Alice sends me this transaction, I can check it for correctness and authorization, verifying that the inputs and outputs balance and that Alice has signed.

However, this transaction on its own is not enough. I also need to verify that Alice’s $15 input state really exists, and she didn’t just make it up. That means I need to see the transaction which created this state, and check it for correctness and authorization as well. If this previous transaction, which sent Alice $15, has a $10 input belonging to Denzel and another $5 input from Eric, then I must also verify the transactions which created those. And so on it goes, all the way back to the original “issuance” transaction in which the asset was created. The number of transactions I need to verify will depend on how many times the assets have changed hands and the extent of backwards branching.

Since Corda nodes don’t automatically see every transaction, how do they obtain the ones they need? The answer is from the sender of each new transaction. Before Alice creates a transaction consuming her $15, she must already have verified the transaction in which she received it. And since Alice must have applied the recursive technique above, she will have a copy of every transaction needed for this verification. Bob simply requests these transactions from Alice as part of their interaction. If Alice doesn’t respond appropriately, Bob concludes that Alice is trying to trick him, and rejects the incoming payment. In the case where Bob is sent a new transaction whose inputs have multiple owners, he can obtain the necessary proofs from each.

Introducing notaries

So far we’ve explained how Bob can verify the correctness and authorization of an incoming transaction, including recursively retracing its inputs’ origins. But there is one more rule we need to think about: uniqueness. Let’s say Alice is malicious. She can generate one transaction in which she pays $10 to Bob, and another in which she pays the same $10 to Charlie. She can send these transactions to Bob and Charlie respectively, along with a full proof of correctness and authorization of each. While both transactions conflict with each other by consuming the same state, there is no way for Bob and Charlie to know this.

Conventional blockchains solve this problem by every node seeing every transaction, making conflicts easy to detect and reject. So how does Corda, with its partial transaction visibility, address the same problem? The answer is with the help of a “notary”. A notary is a trusted party (or parties working together) which guarantees that a particular state is only consumed once. Each state has a specific notary, which must sign any transaction in which that state is consumed. Once a notary has done this, it must not sign another transaction for the same state. Notaries are the network’s guardians of transaction uniqueness.

While every state can have a different notary, all of the states consumed by a particular transaction must be assigned to the same one. This avoids issues relating to deadlocks and synchronization, which should be familiar for those with distributed database experience. Let’s say Alice and Bob agree to exchange Alice’s $10 for Bob’s £7. The transaction for this exchange must be signed by the notaries of both states, but which one goes first? If Alice’s notary signs but Bob’s fails for some reason, then Alice will be left with an incomplete transaction and can never use her $10 again. If Bob’s signs first then he is similarly exposed. While we might like notaries to simply work together, in practice this requires mutual trust and the use of a consensus protocol, complications which Corda’s designers chose to avoid.

If states with different notaries are required as inputs to a single transaction, their owners first execute special “notary change” transactions, which move a state from one notary to another, changing nothing else. So when parties are building a transaction with multiple inputs, they must first agree on the notary to be used, and then perform the notary changes necessary. While the developer in me felt a small twinge of pain when reading about this workaround, there’s no reason why it won’t work so long as notaries play along.

It should also be clarified that, while each notary is a single logical actor in terms of signing transactions, it need not be under the control of a single party. A group of organizations could run a notary collectively, using an appropriate consensus protocol in which a majority of the participants are needed to generate a valid signature. This would prevent any single malicious party from undermining uniqueness by signing transactions that conflict. In theory, we could even allow every node in the network to participant in this kind of shared notarization, although in that case we’d be more-or-less back to a conventional blockchain.

Taking score

Let’s recap the key differences between Corda and conventional blockchains. In Corda, there is no unified blockchain which contains all of the transactions confirmed. Nodes only see those transactions in which they are directly involved, or upon which they depend historically. Nodes are responsible for checking transaction correctness and authorization but rely on trusted notaries to verify uniqueness.

Of course, there is a lot more to Corda than this: the use of digital certificates to authenticate identity, “network maps” to help nodes find and trust each other, per-state “contracts” which define correctness from each state’s perspective, a deterministic version of the Java Virtual Machine which executes these contracts, “flows” which automate transaction negotiations, “time windows” which restrict transactions by time, “oracles” that attest to external facts and “CorDapps” which bundle many things together for easy distribution. While each of these features is interesting, equivalents for all can be found in other blockchain platforms. My goal in this article is to focus on that which makes Corda unique.

So does Corda live up to its promise? Does it solve the scalability, confidentiality and interoperability problems of blockchains? And in making its particular choices, how much of a price does Corda pay?

More scalable, sometimes

Let’s start with scalability. Here, Corda’s advantage appears clear, since nodes only see some of the transactions in a network. In a regular blockchain, the maximum throughput is constrained by the speed of the slowest node in processing transactions. By contrast, a Corda network could process a million transactions per second, while each node sees just a tiny fraction of that. Scalability extends to notaries as well, since the task of signing transactions for uniqueness can be spread between many different notaries, each of which is responsible for a small proportion of the network’s states.

Having said that, there is one situation in which Corda performs far worse than a blockchain. This occurs when a node receives a new transaction which depends on many other transactions it has not seen before. Imagine a highly liquid asset that was issued 10 years ago, and changes hands about every five minutes. The path from any new transaction back to this asset’s issuance will be over a million transactions long. When a node receives this asset for the first time, it must retrieve these million transactions from the sender and verify each one in turn. At a (fairly optimistic) rate of 1000 transactions per second, there would be a 17 minute delay before the recipient could send the asset on – clearly too long for something so liquid.

Why don’t blockchains suffer from this problem? Because nodes see and verify every transaction as it occurs, they are constantly updating the state of the ledger, and know exactly who owns every asset at the present time. Even if a node has never held a particular asset before, it can instantly verify the transaction in which it receives it, and then immediately send it on. To put it another way, blockchain nodes have to verify transactions that might not be relevant to them, but in so doing, they prepay the cost of checking any future transaction that might come in. While Corda nodes are less busy overall, they run the risk of needing to do a huge amount of work at a moment’s notice. There’s nothing scalable about that.

Somewhat more confidential

Let’s move on to confidentiality. In Corda, nodes only see some of a network’s transactions, which undeniably means better privacy than conventional blockchains. Nonetheless, Corda is far from solving the confidentiality problem, because nodes still see some transactions that are none of their business. To take a simple example, if Alice pays Bob $10, then Bob sends that $10 on to Charlie, Charlie’s node has to be shown the transaction between Alice and Bob, even though it doesn’t involve him. At the time that Alice paid Bob, she had no way of knowing who might see this transaction in future, and anyone might be sent it at any time.

To be fair, Corda’s developers are aware of this problem, and discuss it in chapter 15 of their Technical White Paper. The paper suggests simple strategies such as using multiple public keys per entity or reducing traceability by returning assets to issuers for reissuance (similar to cryptocurrency “coin mixers”). It also mentions more advanced future possibilities such as using Tor-like anonymization networks to hide participants’ IP addresses and leveraging zero knowledge proofs or Intel’s secure enclaves to validate transactions without revealing their contents. While all of these suggestions are valid, they can also be applied to regular blockchains using the input–output model, and indeed have been in cryptocurrencies such as Dash, Zcash and Verge. So Corda’s only unique advantage in terms of confidentiality remains its reduced transaction visibility – an incomplete solution at best.

All in the breeding

To better understand Corda’s scalability and confidentiality advantage, we should note how this depends on the density and overlap of the relationships between transactions. Imagine a “family tree” of the transactions performed in a network, in which each transaction’s parents are the previous ones on which it immediately depends. Specifically, when one transaction’s output is consumed by another’s input, we draw an arrow representing the relationship from parent to child. Transactions can have any number of parents and children, although in most cases we’d expect just a few.

Given this family tree, we define the ancestors of a transaction as its parents, grandparents, great-grandparents, and so on. Our tree’s “Adam and Eve” are the issuance transactions which created assets and have no parents of their own. As in regular family trees, two transactions cannot be ancestors of each other. In formal computer science terms, this is a directed acyclic graph or DAG, in which ancestry is defined as the transitive closure of the parent relation.

Recall that when a Corda node processes a transaction, it must download and verify all of that transaction’s ancestors, apart from those it has seen before. So if the family tree is deep, new incoming transactions may have a large number of ancestors that need to be verified, triggering Corda’s scalability problem. In addition, if the family tree contains a high degree of interbreeding, a new transaction’s ancestors might include many or most past transactions in the network. In this case, Corda will provide little advantage in terms of privacy.

By contrast, if the family tree of transactions is shallow, and contains many disconnected islands that do not interact with each other, Corda’s advantages come to the fore. Nodes will never need to verify a large number of transactions at once, and can be kept in the dark about the majority of transactions which are unrelated to their own. If used as a financial ledger, we might say that Corda is ideal for highly fragmented markets whose assets rarely change hands.

Interoperability for the win

Here is one area in which Corda truly shines. Imagine two separate Corda networks, with different sets of assets and participants. At some point a participant in one network wants to send an asset to someone in the other. Unlike conventional blockchains, there is no expectation that a node will have verified all past transactions, so the node receiving this new asset will experience nothing unusual. When the transaction comes in, it simply requests and verifies the relevant history, with no awareness that this is from a “separate network”. To stretch a cliché, we might say that there are no strangers in Corda – just friends who have not yet met.

In reality, things aren’t quite that simple. Any Corda node explicitly decides which notaries to trust, since a misbehaving notary can cause financial mayhem. In addition, nodes need a “certificate” granted by a “doorman” to connect to other nodes in a network, since we can’t allow random members of the public to start connecting to nodes and wasting their resources. So before a node on one network can start requesting and verifying transactions from another network, it will need to add to its list of trusted notaries and obtain the appropriate certificate. While this does involve some manual configuration and administration, it is the minimum that can be expected for a system of this nature. Overall, it’s fair to conclude that interoperability is Corda’s big win over conventional blockchains.

Reintermediation

It’s time to talk about disintermediation, the elephant in Corda’s room. In the context of blockchains, disintermediation means that every participant can verify every transaction for themselves, without depending on the good behavior of third parties. In my view, disintermediation is the core advantage of blockchains over centralized databases, in which all participants fully depend on that database’s owner. If the participants in a network have an intermediary they can rely on, and there is no business or regulatory case for disintermediation, then there’s no point in using a blockchain. Centralized databases are faster and more efficient, and avoid the issue of transaction confidentiality.

So do the participants in a Corda network achieve disintermediation? Well, yes, yes and yes but no. For transaction delivery, Corda ticks the box, since the nodes involved in a transaction talk directly to each other. In terms of correctness and authorization, it’s also in good shape, since each node is able to check these properties for itself. However, when it comes to verifying transaction uniqueness, Corda fails the disintermediation test. Nodes cannot confirm uniqueness for themselves, since they do not see every transaction in the network, and the task is outsourced to trusted notaries.

Corda participants are at the mercy of notaries in a number of ways. First, a notary may refuse to sign a transaction, even if its inputs consume outputs that have never been used before. In a financial ledger, this prevents someone from sending or exchanging their assets. Second, a notary could sign two conflicting transactions which consume the same output, leading two parties to believe they received the same thing. As both recipients of the duplicate asset send or exchange it in further transactions, the contagion spreads, and the integrity of the entire ledger could soon be undermined. Finally, a notary may refuse to sign a “notary change” transaction to transfer a state to a competitor, effectively holding the asset owner hostage. For a transaction involving states with different notaries, it’s far to say that Corda introduces more intermediation than a centralized database, because several third parties are in control.

To put this risk in perspective, it’s worth recalling that Corda notaries need not be controlled by a single organization. They can also consist of a group of nodes running a consensus algorithm that can tolerate bad actors. In this case, a notary will work fine so long as most of its member nodes are following the rules. On the surface, this sounds rather like a blockchain, which depends on a majority of validators behaving well. However in Corda the risks are significantly higher. The worst a cabal of blockchain validators can do is prevent some transactions from being confirmed. A malicious Corda notary can also sign conflicting transactions, sending the ledger into an inconsistent abyss.

A strange animal

Putting scalability, confidentiality, interoperability and disintermediation together, it’s hard to reach a simple verdict on the Corda alternative. Overall, from the perspective of this blockchain platform developer, it seems, well… compelling but strange. Designed to solve the key problems of scalability and confidentiality, Corda’s solutions are incomplete, and greatly depend on the shape of the transaction “family tree”. Yet in order to achieve these partial victories, Corda loses a core property of blockchains – the removal of transaction intermediaries. While Corda undoubtedly excels at interoperability, is that really enough?

If we wanted to be skeptical, we might say that Corda’s team was set an impossible task – to design a flavor of blockchain that would suit the banks funding R3. But the key benefit of blockchains over centralized databases is disintermediation, which comes at the price of reduced confidentiality. How could this trade-off make sense for financial institutions who make money by acting as intermediaries, and are highly sensitive about privacy? Viewed in this light, one might eulogize Corda as a heroic but ultimately unsatisfactory compromise between the desire of R3’s members to do something blockchainy, and the commercial and regulatory constraints under which they exist.

Custodian 2.0

But I prefer to adopt a more positive approach. Instead of focusing on the comparison with blockchains, we can view Corda as a major technical upgrade to the financial status quo. Simply replace the word “notary” with “custodian”, and it all falls into place rather neatly. (A custodian is a financial institution that holds assets on others’ behalf.) Yes, notaries are intermediaries, who can both block transactions and allow conflicts to occur, but this is true of today’s custodians as well. A “notary change transaction” can be seen as the transfer of assets from one custodian to another. And Corda transactions are signed by just one notary for the same reason that we like asset exchanges to occur in one place – to prevent either party being out of pocket.

Looking at Corda in this way, we can see how it improves on the traditional custodial model:

• It defines a standard computational paradigm and format for expressing financial assets and other contractual commitments.
• It provides open source software for interpreting and executing these commitments, guaranteeing that transacting parties and custodians agree on the outcome of every transaction.
• Complex multiparty custodians that protect against abuse can be created (using software only!) by leveraging fault tolerant consensus algorithms.
• A standard process (“notary change”) is defined for the transfer of assets between custodians, and no custodian is allowed to refuse.
• Custodians cannot use an asset under their custody without the owner’s consent, since transactions must also be signed by their inputs’ owners.

I’m far from being a banker, but to me this all sounds rather promising. And perhaps Corda could equally well be applied to other industries with complex custodial structures, such as insurance or shipping. While Corda’s design may not provide the full disintermediation of a blockchain, it proposes a powerful transformation for industries in which intermediaries play an essential role.

Once we go down this line of thinking, a question inevitably arises: If we’re already trusting notaries with the life-and-death job of verifying uniqueness, why not rely on them for correctness and authorization as well? Corda already has the notion of a “validating notary”, which fully verifies transactions before adding its signature. Instead of regular Corda nodes downloading and checking their transactions’ ancestors, why not just ask a notary instead? This could help with scalability and confidentiality, since most nodes would see no transactions other than their own. We might even suggest that a network’s notaries fully trust each other, so there’s no need to worry about ancestors. Each state’s notary could vouch for its validity, verifying only the transaction that created it with other notaries’ help.

Let Corda be Corda

All this takes us back to where we started: Corda isn’t really a competitor for conventional blockchains, MultiChain included. Corda is Corda – an interesting new type of distributed ledger, which has been optimized for the needs of those who are funding it. I have no idea whether Corda will ultimately succeed or fail, because I don’t know its real-world costs and benefits compared to the current way of doing things. But no matter what happens in the future, it is certainly worth studying in terms of philosophy and design.

As for MultiChain, we’re taking a different approach. To steal a line from The West Wing, we’re determined to “let blockchain be blockchain”. Blockchains are what they are, and we have no plans to turn them into something different. As the data infrastructure for a shared application, a blockchain represents a specific trade-off when compared to a centralized database – a gain in disintermediation at the cost of reduced confidentiality. And we’re working hard on making MultiChain 2.0 the best possible blockchain platform for application developers to use.

Please post any comments on LinkedIn.

Today we’re pleased to unveil the second preview release of MultiChain 2.0. This makes substantial progress on the MultiChain 2.0 roadmap, and includes an important extra feature relating to asset permissions.

Per-asset permissions

Let’s start with the surprise. This release adds the ability to separately control the send and receive permissions for each asset issued on the blockchain. This control is important in environments where each asset has different characteristics in terms of regulation, user identification requirements and so on.

At the time a new asset is issued, it can optionally be specified as receive- and/or send-restricted. Receive-restricted assets can only appear in transaction outputs whose address has receive permissions for that asset. Similarly, send-restricted assets can only be spent in transaction inputs by addresses which have per-asset send permissions. (Note that in all cases, addresses need global send and receive permissions to appear in inputs and outputs respectively.)

The send and receive permissions for an asset can be granted or revoked by any address which has admin or activate permissions for that asset. By default, these permissions are only assigned to the asset issuer, but the issuer (or any subsequently added asset administrator) can extend them to other addresses as well.

Blockchain parameter upgrades

One of the major features in development for MultiChain 2.0 is blockchain upgrading, to allow many of a chain’s parameters to be changed over time. This is vital because blockchains are designed to run for the long term, and it’s hard to predict how computer systems will be used many years after their creation.

MultiChain 1.0.x already provides a facility for upgrading a single parameter – the chain’s protocol version. This release of MultiChain 2.0 takes a significant step forwards, allowing changes to seven additional parameters related to blockchain performance and scaling. These include the target block time, maximum block size, maximum transaction size and maximum size of metadata.

As with other crucial operations relating to governance, upgrading a chain’s parameters can only be performed by the chain’s administrator(s), subject to a customizable level of consensus. We’re continuing to work on this feature, so look out for more upgradable parameters in future releases of MultiChain 2.0.

Inline metadata

MultiChain 1.0.x already supports unformatted (binary) transaction metadata, which can be embedded raw or wrapped in a stream item. The first preview release of MultiChain 2.0 extended this to allow metadata to be optionally represented in text or JSON format. In all of these cases the metadata appears in a separate transaction output containing an OP_RETURN, which makes the output unspendable by subsequent transactions.

This release of MultiChain 2.0 introduces a new type of metadata which we call “inline”. Inline metadata is stored within a regular spendable transaction output, and so is associated directly with that output’s address and/or assets. As with other forms of metadata, inline metadata can be in binary, text or JSON formats, and is easily writable and readable via a number of different APIs.

Inline metadata becomes truly powerful when used in conjunction with custom rules regarding transaction validity. One example is to send assets with an expiry date, or with a list of restrictions on where they can go next. In this release, custom validation rules can only be defined by modifying MultiChain’s C++ source code. However, once filters are implemented as part of the MultiChain 2.0 roadmap, these rules will be written in JavaScript and installed on a blockchain using regular API calls.

The road ahead

With this second preview/alpha release, we’ve completed about half of work scheduled for the open source Community edition of MultiChain 2.0. You can download and try out alpha 2 by visiting the MultiChain 2.0 preview releases page. On this page you’ll also find documentation for the new and enhanced APIs.

We’ve already started working on the next major feature for MultiChain 2.0, which we’re calling off-chain stream items. In an off-chain item, only a hash of the item’s payload is embedded inside the chain, alongside the item’s keys and some other metadata. The payload itself is stored locally by the publisher and propagated to the stream’s subscribers using peer-to-peer file sharing techniques, with the on-chain hash providing verification. The result is a huge improvement in the scalability and performance of blockchains used to record large amounts of information, where some of this information is only of interest to certain participants. While not originally planned for MultiChain 2.0, this feature rose up our list of priorities in response to user demand.

As always, we welcome your feedback on the progress of MultiChain 2.0, and look forward to delivering the next preview release in due course.

Please post any comments on LinkedIn.

It’s exactly two years since we published “Avoiding the pointless blockchain project“, a checklist of questions to ask when assessing permissioned blockchain use cases. The post obviously struck a nerve and continues to attract thousands of monthly readers on our site and others. People are still hungry for content that goes beyond the blockchain hype to assess this technology objectively.

The good news is that, judging by our incoming inquiries, the market’s understanding of blockchains has greatly improved over the last two years. I would estimate that 60% of the blockchain use cases we now hear are commercially and technically sound. Nonetheless there is still plenty of confusion – companies determined to use a blockchain when a regular database would fit better, startups using “blockchains” in their branding but nowhere else, and widely reported but pointless blockchain projects which use a single node or a group of nodes under a single party’s control.

To recap what I’ve written before, the core value of a blockchain is to enable a database or ledger to be directly shared across boundaries of trust, without putting any single party in charge. A blockchain lets a group of actors achieve real-time reconciliation of validated, authenticated and timestamped transactions, without the cost, hassle and risk of relying on a trusted intermediary. The chain provides meaningful value when it’s maintained by consensus between multiple nodes, each of which is controlled by a party with different interests. This protects against individual participants (or small groups thereof) from corrupting or deleting past transactions.

MultiChain 1.0 was released a few months ago, and we’re delighted to now share the details of some of the early MultiChain-powered blockchains in production. Each application described below was independently built by a third party using the regular MultiChain software and APIs. All are running in a network of four nodes or more, with multiple active validators. Most importantly, in each case the blockchain is addressing a real business problem that could not be solved by a regular database.

Workflow management for infrastructure projects

Construtivo is a Brazilian software company which builds solutions for the design and construction phase of large infrastructure projects. For the past 15 years, Construtivo’s general approach has been to deliver software-as-a-service (SaaS), in which the company acts as the central trusted intermediary for managing project data. This is the traditional approach to ensuring that all stakeholders maintain a consistent view of a project’s status and progress.

To satisfy their customers’ desire for greater transparency and auditability, Construtivo have now integrated MultiChain into their solution, providing the option of storing crucial project data on a blockchain alongside Construtivo’s database. Several infrastructure projects in South America are already making use of this option. Each project has its own chain, with nodes run by both Construtivo and stakeholders such as contractors and engineering companies. Depending on the project’s requirements, the chain can record plans, contracts, and other workflow-related information, and can be browsed through a web-based interface.

The typical MultiChain network for an infrastructure project has 4 nodes, with an average transaction size of 15K. All nodes in each chain participate in the validation process, with control over user permissions remaining in Construtivo’s hands. As with most of our users, Construtivo researched a number of blockchain platforms to find a suitable fit. When asked why they settled on MultiChain, Rodrigo Trindade, systems analyst at Construtivo, cited its speed, simplicity and ease of integration with their application.

Shared ledger for a catastrophe bond

Solidum Partners is an investment advisory company which specializes in creating catastrophe bonds. These are financial instruments which pay investors a high rate of yield compared to regular commercial bonds, but have a risk of partial or no repayment if a particular event occurs. In essence, purchasers of catastrophe bonds are acting like insurance companies, providing the capital to cover unlikely losses and making a tidy profit so long as those losses don’t materialize.

In order to be easy to trade, non-physical securities like catastrophe bonds are traditionally held by a trusted intermediary on their owners’ behalf. Trades in the security are “settled” virtually via an update of the intermediary’s records. For Solidum, the intermediary of choice had been Euroclear, which holds over $30 trillion in financial assets on behalf of investors, or more than 10% of the world’s total. Naturally, with around 4,000 employees at 15 offices around the world, Euroclear doesn’t provide this service for free.

Due to recent changes at a banking partner, Solidum lost access to Euroclear and had to seek another way. So they issued a new $15 million catastrophe bond directly onto a MultiChain blockchain, along with dollar denominated tokens that could be used for transacting. If you like, they performed two private placement Initial Coin Offerings (ICOs), but with real underlying assets instead of a white paper and the hope of future value.

The blockchain enables safe “delivery-vs-payment” transactions, in which two users exchange dollars and bond units in a single step – a feat which traditionally requires help from a trusted intermediary. Aside from avoiding this middleman’s fees, using a permissioned blockchain gave Solidum easy and direct control over who can participate in the system, without triggering the same heavy regulation as Euroclear and its peers.

Each participant in the network has their own MultiChain node, giving them direct control over their on-chain assets. While a trustee knows the real-world identity behind each address on the blockchain, participants do not know each other’s. (Unlike many financial use cases, the level of activity is not high enough for this veil of confidentiality to be broken.) After completing AML and KYC checks, users are given access to the chain by Solidum and can then transact with each other directly. The network currently has around 10 nodes, 4 of which are permanently online and participate in the consensus process.

When asked why they chose MultiChain, Cedric Edmonds, partner at Solidum, cited its simple built-in support for delivery-vs-payment exchange transactions, as well as its general stability and ease of use.

Transaction notarization for e-commerce

Cryptologic, a blockchain consultancy based in Rosario, Argentina, have built and deployed a system for notarizing e-commerce transactions, in order to help resolve disputes between buyers and sellers. Their first customer is MercadoLibre, Latin America’s most popular e-commerce site, which has almost $1 billion in annual revenues.

Under usual circumstances, when a customer makes a purchase from an online merchant, they have to trust that merchant to record the transaction securely and permanently. But in practice, nothing stops employees of the merchant from deleting or modifying transaction records, and this can serve as a back door for delayed delivery or goods to end up in the wrong hands. By contrast, if each transaction is recorded on a blockchain whose contents are publicly visible, and whose control is spread among a number of different parties, then this record becomes far more difficult to retroactively change.

To preserve confidentiality, transaction data is hashed before being embedded in the chain. The hashes provide a mechanism for timestamping and notarization, and are sufficient to settle later disputes if either party reveals the unhashed transaction. The network currently contains 7 permanent nodes, spread between Cryptologic, various government offices, and a partner abroad. Since transactions contain hashes only, they are fairly small, and the network has seen a peak rate of 50 transactions per second (still well below MultiChain’s maximum throughput).

When asked why they chose MultiChain, Maximiliano Cañellas, CTO at Cryptologic, said they found it really easy to use, with great features like streams, and that the product is very stable, having run for 10 months without interruption.

General lessons learned

These are some early examples of permissioned blockchains in production. The networks are still small, with modest transaction volumes that are far from the limits of products like MultiChain. So it’s important not to extrapolate too much.

Nonetheless, it’s interesting to note what these applications have in common. First and most importantly, they all derive from a genuine desire for decentralization, rather than using a blockchain for a blockchain’s sake. In all three cases, there were clear reasons to choose a blockchain architecture over messaging or a centralized database.

Second, none of the chains have yet transitioned to a decentralized model for governance. All still rely on a single administrator, who onboards new users and grants them permission to transact. It remains to be seen how often decentralized governance (as supported by MultiChain’s admin consensus model) is viable or necessary in practice. Perhaps it is sufficient for the blockchain to provide a transparent view of all administrator activity, while leaving control of this activity with a single party.

Finally, the nature of these applications confirms our view that blockchains are a general purpose technology for shared databases, and not restricted to particular industries or verticals. The lion’s share of media coverage might be received by specific use cases, such as interbank settlement, supply chain finance and shared identity. But in reality, blockchains can be applied whenever we seek to avoid centralized control over a digital system of record. It’s time to think more broadly about the types of problems that this technology can solve.

Please post any comments on LinkedIn.

Today we’re delighted to share the first preview release of MultiChain 2.0, which implements one major part of the MultiChain 2.0 roadmap published earlier this year – a richer data model for streams.

Streams have proven to be a popular feature in MultiChain, providing a natural abstraction for general purpose data storage and retrieval on a blockchain. A MultiChain chain can contain any number of named streams, each of which can have individual write permissions or be open for writing by all. In MultiChain 1.0, each stream item has one or more publishers (who sign it), an optional key for efficient retrieval, a binary data payload up to 64 MB in size, and a timestamp derived from the block in which it’s embedded.

This preview release of MultiChain 2.0, numbered alpha 1, takes streams functionality to a whole new level:

• JSON items. As an optional alternative to raw binary data, stream items can now contain any JSON structure, which is stored on the blockchain in the efficient UBJSON serialization format. Since the MultiChain API already uses JSON throughout, these JSON structures can be read and written in a natural and obvious way.
• Text items. Stream items may also contain Unicode text, stored efficiently on the blockchain in UTF-8 encoding. Text items can also be read and written directly via the MultiChain API.
• Multiple keys. Each stream item can now have multiple keys instead of only one. This enables much more flexible schemes for tagging, indexing and retrieval.
• Multiple items per transaction. Multiple items can now be written to the same stream in a single atomic transaction. This allows multiple stream items to: (a) be naturally grouped together under a single transaction ID, (b) take up less space on the blockchain and (c) require fewer signature verifications.
• JSON merging. There are new APIs to summarize the items in a stream with a particular key or publisher. The first type of summary offered is a merge of all of the JSON objects in those items. The outcome of the merge is a new object containing all the JSON keys from the individual objects, where the value corresponding to each JSON key is taken from the last item in which that key appears. The merge can be customized in various ways, e.g. to control whether sub-objects are merged recursively and if null values should be included.

The purpose of JSON merging is to enable a stream to serve as a flexible database for applications built on MultiChain, with the stream key or publisher (as appropriate) acting as a “primary key” for each database entry. The advantage over a regular database is that the stream contains a fully signed and timestamped history of how each entry was changed over time, with the blockchain securing this history immutably through multiparty consensus.

As in previous versions, each node can freely decide which streams to subscribe to, or can subscribe to all streams automatically. If a node is subscribed to a stream, it indexes that stream’s content in real time, allowing efficient retrieval by publisher, key, block, timestamp or position – and now summarization by key or publisher.

Aside from stream items, MultiChain 2.0 alpha 1 also supports JSON and text in raw transaction metadata, as alternatives to the raw binary data supported in MultiChain 1.0.

Finally, this release allows the custom fields of issued assets and created streams to contain any JSON object, instead of the text-only key/value pairs offered in MultiChain 1.0. For forwards compatibility, MultiChain 1.0.2 includes the ability to read (but not write) these richer asset and stream custom fields.

To try out these new features, visit the MultiChain 2.0 preview releases page and download alpha 1. The page also provides detailed documentation on the new APIs and parameters available.

We’d love to hear your feedback on this new functionality. And of course we’re already hard at work on the next major set of enhancements for MultiChain 2.0, scheduled for release early next year.

Please post any comments on LinkedIn.

Today we’re delighted to announce the release of MultiChain 1.0 into production and the addition of 14 new members to the MultiChain Partner Program. These include two multinational consulting companies: Cognizant and Indra Sistemas, as well as twelve other companies: Aicumen, Bambusoft, Chainfrog, CrimsonLogic, Encrypgen, Hypatia Technologies, Maroon Studios, Medici Ventures, Project Radium, SolarLab, The Apollo Group and Tilkal.

Apart from that, we’re already hard at work developing MultiChain 2.0 and hope to have a first preview release available (with the richer data model for streams) before the end of the year.

More details can be found in the press release below.

MultiChain Launches Production-Ready Version 1.0 with Fourteen New Partners

August 2, 2017 – Coin Sciences Ltd is delighted to announce the production-ready release of MultiChain 1.0, along with fourteen new members of the MultiChain Partner Program, bringing the total number to 43.

MultiChain 1.0 is available for immediate download for Linux, Windows and Mac, after two and a half years of intensive feedback-driven development. This includes a four-month beta period, during which MultiChain was optimized to support over 1,000 transactions per second on a mid-range server. Since its first alpha release in June 2015, MultiChain has received over 60,000 downloads, more than half of which were during 2017.

The new members of the MultiChain Partner Program include two multinational consulting companies: Cognizant and Indra Sistemas. Twelve more SMBs have also joined: Aicumen, Bambusoft, Chainfrog, CrimsonLogic, Encrypgen, Hypatia Technologies, Maroon Studios, Medici Ventures, Project Radium, SolarLab, The Apollo Group and Tilkal. Members of the program enjoy a close working relationship with the MultiChain engineering team, can use MultiChain branding in their marketing materials, and are promoted on the MultiChain website, which now receives 35,000 visitors monthly.

The MultiChain Partner Program has now been split into two tracks – Platform Partners who develop applications for third parties on the MultiChain platform, and Product Partners who are using MultiChain in their own proprietary solutions. The partners in each track are listed at https://www.multichain.com/platform-partners/ and https://www.multichain.com/product-partners/ respectively.

“We’re delighted to have reached this milestone,” said Dr Gideon Greenspan, CEO and Founder of Coin Sciences Ltd. “Developing the first production release of MultiChain has been an immense challenge, and we’ve learned a great deal about our users and their requirements along the way. Work has already begun on MultiChain 2.0, which will be the first version of MultiChain to come in two editions – Community (open source) and Enterprise (commercial). We look forward to continued growth in usage of the product and cooperating with all our partners to help them leverage it for their needs.”

“We used MultiChain to build a platform for transferring digital assets between different organizations (from commerce to public administration) in a permissioned network where all the participants collaborate,” said Víctor Sánchez Hórreo, Manager of Blockchain and Digital Transformation at Minsait (Indra Sistemas). “The assets act as a key tool to enable social and economic projects, and the features of MultiChain regarding permission management, quick deployment and asset creation fit very well with our needs.”

“Because of its Bitcoin ancestry, MultiChain’s reliability, even during its alpha phase, was great,” said Joel Weight, Chief Technology Officer at Medici Ventures. “The addition of a key-based permission layer and built-in asset support make it the right solution for some of our products.”

“Chainfrog chose to use MultiChain in their music royalties collection pilot because it is based on the mature Bitcoin source base, is incredibly easy to deploy and the APIs are clearly documented,” said Dr Keir Finlow-Bates, CEO and Founder of ChainFrog. “From their blog posts it is obvious that the Coin Sciences team know their blockchains inside out.”

Please post any comments on LinkedIn.

Here at Coin Sciences, we’re best known for MultiChain, a popular platform for creating and deploying permissioned blockchains. But we began life in March 2014 in the cryptocurrency space, with the goal of developing a “bitcoin 2.0″ protocol called CoinSpark. CoinSpark leverages transaction metadata to add external assets (now called tokens) and notarized messaging to bitcoin. Our underlying thinking was this: If a blockchain is a secure decentralized record, surely that record has applications beyond managing its native cryptocurrency.

After less than a year, we stopped developing CoinSpark, due to both a push and a pull. The push was the lack of demand for the protocol – conventional companies were (understandably) reluctant to entrust their core processes to a public blockchain. But there was also a pull, in terms of the developing interest we saw in closed or permissioned distributed ledgers. These can be defined as databases which are safely and directly shared by multiple known but non-trusting parties, and which no single party controls. So in December 2014 we started developing MultiChain to address this interest – a change in direction that Silicon Valley would call a “pivot”.

Two years since its first release, MultiChain has proven an unqualified success, and will remain our focus for the foreseeable future. But we still take an active interest in the cryptocurrency space and its rapid pace of development. We’ve studied Ethereum’s gas-limited virtual machine, confidential CryptoNote-based systems like Monero, Zcash with its (relatively) efficient zero knowledge proofs, and new entrants such as Tezos and Eos. We’ve also closely observed the crypto world’s endless dramas, such as bitcoin’s block size war of attrition, the failures of numerous exchanges, Ethereum’s DAO disaster and Tether’s temporary untethering. Crypto news is the gift that keeps on giving.

Crypto and the enterprise

Aside from sheer curiosity, there’s a good reason for us to watch so closely. We fully expect that many of the technologies developed for cryptocurrencies will eventually find their way into permissioned blockchains. And I should stress here the word eventually, because the crypto community has (to put it a mildly) a far higher risk appetite than enterprises exploring new techniques for integration.

It’s important to be clear about the similarities and differences between cryptocurrencies and enterprise blockchains, because so much anguish is caused by the use of the word “blockchain” to describe both. Despite the noisy objections of some, I believe this usage is reasonable, because both types of chain share the goal of achieving decentralized consensus between non-trusting entities over a record of events. As a result, they share many technical characteristics, such as digitally signed transactions, peer-to-peer networking, transaction constraints and a highly robust consensus algorithm that requires a chain of blocks.

Despite these similarities, the applications of open cryptocurrency blockchains and their permissioned enterprise counterparts appear to be utterly distinct. If you find this surprising or implausible, consider the following parallels: The TCP/IP networking protocol is used to connect my computer to my printer, but also powers the entire Internet. Graphics cards make 3D video games more realistic, but can also simulate neural networks for “deep learning”. Compression based on repeating sequences makes web sites faster, but also helps scientists store genetic data efficiently. In computing, multi-purpose technologies are the norm.

So here at Coin Sciences, we believe that blockchains will be used for both cryptocurrencies and enterprise integration over the long term. We don’t fall on either side of the traditional (almost tribal) divide between advocates of public and private chains. Perhaps this reflects an element of wishful thinking, because a thriving cryptocurrency ecosystem will develop more technologies (under liberal open source licenses) that we can use in MultiChain. But I don’t think that’s the only reason. I believe there is a compelling argument in favor of cryptocurrencies, which can stand on its own.

In favor of crypto

What is the point of cryptocurrencies like bitcoin? What do they bring to the world? I believe the answer is the same now as in 2008, when Satoshi Nakamoto published her famous white paper. They enable direct transfers of economic value over the Internet, without a trusted intermediary, and this is an incredibly valuable thing. But unlike Satoshi’s original vision, I do not see this as a better way to buy coffee in person or kettles online. Rather, cryptocurrencies are a new class of asset for people looking to diversify their financial holdings in terms of risk and control.

Let me explain. In general people can own two types of asset – physical and financial. For most of us physical assets are solid and practical items, like land, houses, cars, furniture, food and clothing, while a lucky few might own a boat or some art. By contrast, financial assets consist of a claim on the physical assets or government-issued money held by others. Unlike physical assets, financial assets are useless on their own, but can easily be exchanged for useful things. This liquidity and exchangeability makes them attractive despite their abstract form.

Depending on who you ask, the total value of the world’s financial assets is between $250 and $300 trillion, or an average of $35-40k per person alive. The majority of this sum is tied up in bonds – that is, money lent to individuals, companies and governments. Most of the rest consists of shares in public companies, spread across the stock exchanges of the world. Investors have plenty of choice.

Nonetheless, all financial assets have something in common – their value depends on the good behavior of specific third parties. Furthermore, with the exception of a few lingering bearer assets, they cannot be transferred or exchanged without a trusted intermediary. These characteristics create considerable unease for these assets’ owners, and that feeling gains credence during periods of financial instability. If a primary purpose of wealth is to make people feel safe in the face of political or personal storms, and the wealth itself is at risk from such a storm, then it’s failing to do its job.

So it’s natural for people to seek money-like assets which don’t depend on the good behavior of any specific third party. This drive underlies the amusingly-named phenomenon of gold bugs – people who hold a considerable portion of their assets in physical gold. Gold has been perceived as valuable by humans for thousands of years, so it’s reasonable to assume this will continue. The value of gold cannot be undermined by governments, who often succumb to the temptation to print too much of their own currency. And just as in medieval times, gold can be immediately used for payment without a third party’s assistance or approval.

Despite these qualities, gold is far from ideal. It’s expensive to store, heavy to transport, and can only be handed over through an in-person interaction. In the information age, surely we’d prefer an asset which is decentralized like gold but is stored digitally rather than physically, and can be sent across the world in seconds. This, in short, is the value proposition of cryptocurrencies – teleportable gold.

On intrinsic value

The most immediate and obvious objection to this thesis is that, well, it’s clearly ridiculous. You can’t just invent a new type of money, represented in bits and bytes, and call it Gold 2.0. Gold is a real thing – look it’s shiny! – and it has “intrinsic value” which is independent of its market price. Gold is a corrosion-resistant conductor of electricity and can be used for dental fillings. Unlike bitcoin, if nobody else in the world wanted my gold, I could still do something with it.

There’s some merit to this argument, but it’s weaker than it initially sounds. Yes, gold has some intrinsic value, but its market price is not derived from that value. In July 2001 an ounce of gold cost $275, ten years later it cost $1840, and today it’s back around the $1200 mark. Did the practical value of dental fillings and electrical wiring rise sevenfold in ten years and then plummet in the subsequent six?

Clearly not. The intrinsic value argument is about something more subtle – it places a lower bound on gold’s market price. If gold ever became cheaper than its functional substitutes, such as copper wiring or dental amalgam, electricians and dentists would snap it up. So if you buy some gold today, you can be confident that it will always be worth something, even if it’s (drastically) less than the price you paid.

Cryptocurrencies lack the same type of lower bound, derived from their practical utility (we’ll discuss a different form of price support later on). If everyone in the world lost interest in bitcoin, or it was permanently shut down by governments, or the bitcoin blockchain ceased to function, then any bitcoins you hold would indeed be worthless. These are certainly risks to be aware of, but their nature also points to the source of a cryptocurrency’s value – the network of people who have an interest in holding and transacting in it. For bitcoin and others, that network is large and continuing to grow.

Indeed, if we look around, we can find many types of asset which are highly valued but have negligible practical use. Examples include jewelry, old paintings, special car license plates, celebrity autographs, rare stamps and branded handbags. We might even say that, in terms of suitability for purpose, property in city centers is drastically overpriced compared to the suburbs. In these cases and more, it’s hard to truly justify why people find something valuable – the reason is buried deep in our individual and collective psyches. The only thing these assets have in common is their relative scarcity.

So I wouldn’t claim that bitcoin’s success was a necessary or predictable consequence of its invention, however brilliant that may have been. What happened was a complete surprise to most people, myself included, like the rise of texting, social media, sudoku and fidget spinners. There’s only one reason to believe that people will find cryptocurrencies valuable, and that is the fact that they appear to be doing so, in greater and greater numbers. Bitcoin and its cousins have struck a psychoeconomic nerve. People like the idea of owning digital money which is under their ultimate control.

Against crypto maximalism

At this point I should clarify that I am not a “cryptocurrency maximalist”. I do not believe that this new form of money will take over the world, replacing the existing financial landscape that we depend on. The reason for my skepticism is simple: Cryptocurrencies are a poor solution for the majority of financial transactions.

I’m not just talking about their sky-high fees and poor scalability, which can be technically resolved with time. The real problem with bitcoin is its core raison d’être – the removal of financial intermediaries. In reality, intermediaries play a crucial role in making our financial activity secure. Do consumers want online payments to be irreversible, if a merchant has ripped them off? Do companies want a data loss or breach to cause immediate bankruptcy? One of my favorite Twitter memes is this from Dave Birch (although note that bitcoin is not truly anonymous or untraceable):

Help! I want my anonymous untraceable electronic money back, part 97: South Korea https://t.co/LoImbsZnEV

— David G.W. Birch (@dgwbirch) July 5, 2017

Help. I want my anonymous untraceable electronic money back, part 97: Ethereum tokens https://t.co/Qi5w04dFAo

— David G.W. Birch (@dgwbirch) June 18, 2017

While it’s wonderful to send value directly across the Internet, the price of this wizardry is a lack of recourse when something goes wrong. For the average Joe buying a book or a house, this trade-off is simply a bad deal. And the endless news stories about stolen cryptocurrency and hacked bitcoin exchanges aren’t going to change his mind. As a result, I believe cryptocurrencies will always be a niche asset, and nothing more. They will find their place inside or outside of the existing financial order, alongside small cap stocks and high yield bonds. Not enough people are thinking about the implications of this boring and intermediate outcome, which to me seems most likely of all.

A pointed historical analogy can be drawn with the rise of e-commerce. In the heady days of the dot com boom, pundits were predicting that online stores would supersede their physical predecessors. Others said that nobody would want to buy unseen goods from web-based upstarts. Twenty years later, Amazon, Ebay and Alibaba have indeed built their empires, but physical stores are still with us and attractive to buy. In practice, most of us purchase some things online, and other things offline, depending on the item in question. There are trade-offs between these two forms of commerce, just as there are between cryptocurrencies and other asset classes. He who diversifies wins.

Now about that price

If cryptocurrencies will be around in the long term, but won’t destroy the existing financial order, then the really interesting question is this: Exactly how big are they going to get? Fifty years from now, what will be the total market capitalization of all the cryptocurrency in the world?

In my view, the only honest answer can be: I’ve no idea. I can make a strong case for a long-term (inflation-adjusted) market cap of $15 billion, since that’s exactly where crypto was before this year’s (now deflating) explosion. And I can make an equally strong case for $15 trillion, since the total value of the world’s gold is currently $7 trillion, and cryptocurrencies are better in so many ways. I’d be surprised if the final answer went outside of this range, but a prediction this wide is as good as no prediction at all.

Most financial assets have some kind of metric which acts to anchor their price. Even in turbulent markets, they don’t stray more than 2-3x in either direction before rational investors bring them back into line. For example, the exchange rates between currencies gravitate towards purchasing power parity, defined as the rate at which a basket of common goods costs the same in every country. Bonds gravitate towards their redemption price, adjusted for interest, inflation and risk, which depends on the issuing party. Stocks gravitate towards a price/earnings ratio of 10 to 25, because of the alternatives available to income-seeking investors. (One exception appears to be high-growth technology stocks, but even these eventually come back down to earth. Yes, Amazon, your day will come.)

When it comes to the world of crypto, there is no such grounding. Cryptocurrencies aren’t used for pricing common goods, and they don’t pay dividends or have a deadline for redemption. They also lack the pedigree of gold or artwork, whose price has been discovered over hundreds of years. As a result, crypto prices are entirely at the mercy of Keynesian animal spirits, namely the irrational, impulsive and herd-like decisions that people make in the face of uncertainty. To paraphrase Benjamin Graham, who wrote the book on stock market investing, Mr Crypto Market is madder than a madman. The geeks among us might call it chaos theory in action, with thousands of speculators feeding off each other in an informational vacuum.

Of course, some patterns can be discerned in the noise. I don’t want to write (or be accused of writing) a guide to cryptocurrency investing, so I’ll mention them only in brief: reactions to political uncertainty and blockchain glitches, periods of media-driven speculation, profit-taking by crypto whales, 2 to 4 year cycles, deliberate pump-and-dump schemes, and the relentless downward pressure caused by proof-of-work mining. But if I could give one piece of advice, it would be this: Buy or sell to ensure you’ll be equally happy (and unhappy) whether crypto prices double or halve in the next week. Because either can happen, and you have no way of knowing which.

If the price of a cryptocurrency isn’t tied to anything and moves unpredictably, could it go down to zero? Barring a blockchain’s catastrophic technical failure, I think the answer is no. Consider those speculators who bought bitcoin in 2015 and sold out during the recent peak, making a 10x return. If the price of bitcoin goes back to its 2015 level, it would be a no-brainer for them to buy back in again. In the worst case, they’ll lose a small part of their overall gains. But if history repeats itself, they can double those gains. And maybe next time round, the price will go even higher.

This rational behavior of previous investors translates into a cryptocurrency’s price support, at between 10% and 25% (my estimate) of its historical peak. That’s exactly what happened during 2015 (see chart below) when bitcoin’s price stabilized in the $200-$250 range after dropping dramatically from over $1000 a year earlier. At the time there was no good reason to believe that it would ever rise again, but the cost of taking a punt became too low to resist.

So I believe that cryptocurrencies will be with us for the long term. As long as bitcoin is worth some non-trivial amount, it can be used as a means of directly sending money online. And as long as it serves this purpose, it will be an attractive alternative investment for people seeking to diversify. The same goes for other cryptocurrencies that have reached a sufficient level of interest and support, such as Ethereum and Litecoin. In Ethereum’s case, this logic applies whether or not smart contracts ever find serious applications.

On that subject, I should probably (and reluctantly) mention the recent wave of token Initial Coin Offerings (ICOs) on Ethereum. For the most part, I don’t see these as attractive investments, because their offer price may well be a high point to which they never return. And the sums involved are often ridiculous – if $18 million was enough to fund the initial development of Ethereum, I don’t see why much simpler projects are raising ten times that amount. My best guess is that many ICO investors are looking for something to do with their newly-found Ether riches, which they prefer not to sell to drive down the price. Ironically, after being collected by these ICOs, much is being sold anyway.

Back to reality

There’s a certain symmetry between people’s reactions to cryptocurrencies and enterprise blockchains. In both cases, some shamelessly drive the hype, claiming that bitcoin will destroy the financial system, or that enterprise chains will replace relational databases. Others are utterly dismissive, seeing cryptocurrencies as elaborate Ponzi schemes and permissioned blockchains as a technological farce.

In my view, these extreme positions are all ignoring a simple truth – that there are trade-offs between different ways of doing things, and in the case of both cryptocurrencies and enterprise blockchains, these trade-offs are clear to see. A technology doesn’t need to be good for everything in order to succeed – it just needs to be good for some things. The people who are doing those things have a tendency of finding it.

So when it comes to both public and private blockchains, it’s time to stop thinking in binary terms. Each type of chain will find its place in the world, and provide value when used appropriately. In the case of cryptocurrencies, as an intermediary-free method for digital value transfer and an alternative asset class. And in the case of enterprise blockchains, as a new approach to database sharing without a trusted intermediary.

That, at least, is the bet that we’re making here.

Disclosure: The author has a financial interest in various cryptocurrencies. Coin Sciences Ltd does not.

Please post any comments on LinkedIn.

Today we’re delighted to release the second beta of MultiChain 1.0 for Linux, Windows and Mac (for now the Mac version requires compilation). This concludes the planned development of MultiChain 1.0 – with the exception of any bug fixes, the final release of MultiChain 1.0 over the summer will be unchanged.

This month also marks two years since the first alpha release of MultiChain in June 2015. As with any new product, we weren’t sure how the market would react, and knew there was only one way to find out – release a minimum viable product, meaning an initial version which provides significant value but is preliminary by design. Thankfully, unlike our first product CoinSpark, MultiChain received a strong and immediate positive response. This was accompanied by a tsunami of sensible feature requests, many of which we’ve now implemented. In parallel to the product’s development, usage has also grown remarkably by every measure. For example, the MultiChain website received under 3,000 visitors in July 2015, and now brings in ten times that number monthly.

MultiChain performance

Over the past two years we’ve invested a lot of effort in optimizing MultiChain, which was forked from Bitcoin Core, the reference implementation for the public bitcoin network. Below is a comparison of transaction throughput for a single-node setup using five versions of the product:

Average transactions per second, including API overhead and building, signing, mining and verifying transactions and blocks.
Tests performed using the ab HTTP server benchmarking tool sending two concurrent requests to the sendtoaddress API.
Server specifications: Intel Core i7-4770, 4 cores @ 3.4 MHz, 32 GB RAM, Seagate 2 TB 7200 RPM SATA, CentOS 6.4.

Naturally, the biggest jump came in alpha 22 when we transitioned to a database-driven wallet. But since that release, we’ve almost doubled MultiChain’s speed again. We hope we’ve demonstrated that bitcoin’s limit of 4 transactions per second is due to its particular network parameters, and has no relation to blockchains in general.

Of course, performance optimization is a never-ending task, and there’s no reason why MultiChain can’t reach 10,000 tx/sec on a 16-core processor with the appropriate architectural changes. However, based on conversations with our users and partners, it seems that few expect to need more than 1,000 tx/sec for the next few years. So we’re refocusing our development efforts on new features, which brings us nicely onto the subject of MultiChain 2.0.

MultiChain 2.0 overview

Version 2.0 of MultiChain will be the first to come in two editions – Community (open source) and Enterprise (commercial). I’m going to focus here on the free Community edition, since we’re only discussing the details of MultiChain Enterprise with our partners. In any event, the Community and Enterprise editions will be highly compatible, in that: (a) applications built on the Community edition will run without modification on MultiChain Enterprise, and (b) both editions will be able to connect and transact with each other on the same chain.

The three key areas of enhanced functionality in both editions of MultiChain 2.0 will be:

• Richer data model for streams, including JSON documents.
• Custom programmable transaction filters for on-chain validation.
• Seamless updating of a blockchain’s protocol and parameters.

Let’s turn to discuss each of these in detail.

Richer data model for streams

MultiChain streams were introduced in September 2016 and have proven extremely popular. As described in this post, streams provide a simple and natural abstraction for general purpose data storage, indexing and retrieval on a blockchain. A MultiChain blockchain can contain any number of named streams, each of which can either be open to all for writing, or writable only from certain addresses.

In MultiChain 1.0, each stream item has one or more publishers (who sign it), an optional key, a binary data payload up to 64 MB in size, and a timestamp (derived from the block in which it’s embedded). Each node can freely decide which streams to subscribe to, or can subscribe to all streams automatically. If a node is subscribed to a stream, it indexes that stream’s content in real time, allowing efficient retrieval by publisher, key, block, timestamp or position.

MultiChain 2.0 will enrich this streams functionality in a number of ways:

• JSON items. As well as binary data, stream items will support structured JSON objects, stored on the blockchain in an efficient serialization format such as UBJSON. Since the MultiChain API already uses JSON throughout, these JSON objects will be writable and readable in a natural and obvious way.
• Multiple keys. Stream items will support multiple keys, enabling a single piece of data to be indexed in multiple ways for retrieval using liststreamkeyitems. We’re constantly evaluating how much database functionality to include within MultiChain, and don’t expect to support indexing of the sub-elements within JSON stream items in version 2.0. Allowing multiple keys per stream item provides a reasonable workaround.
• Atomic writes of multiple items. MultiChain 1.0 allows a single transaction to write to multiple streams, but not to write multiple items to the same stream. MultiChain 2.0 will remove this restriction.
• JSON merging. Any ordered list of JSON objects can be naturally flattened or summarized to create a “merged” object. The merged object contains all the keys which appear in the individual objects, where the value corresponding to each key is taken from the last object in which that key appears. If you like, the merged object is the final state of a database row, whose columns are defined by the first object and extended or updated by later objects. MultiChain 2.0 will add APIs to easily and rapidly retrieve the merged object for the JSON items in a stream with a particular key or publisher.

These features are derived from common ways in which developers are currently using streams. In other words, we’re observing what many people are building on top of MultiChain at the application level, and bringing that functionality into MultiChain itself – a pattern that we intend to continue applying. Now that stream items will include type information, they can easily be extended in future to support other data formats such as XML, HDF5 and MIME-identified content. Not to mention the possibilities of transparent on-chain compression and encryption.

MultiChain 2.0 will also support JSON objects for raw transaction metadata (i.e. not stream items) as well as the metadata for asset issuance and stream creation events, instead of the text-only key/value pairs implemented in MultiChain 1.0. The listassets API will offer JSON merging across all of an asset’s issuance events, so that each issuance’s metadata can effectively update the asset’s final description.

Custom transaction filters

We’ve thought a lot about how to add custom programmable rules to MultiChain. While Ethereum’s “smart contract” paradigm is popular, it has a number of key shortcomings for high-throughput permissioned blockchains. First, smart contracts introduce a global dependency across the blockchain’s entire state, which drastically impairs concurrency and performance. Second, smart contracts cannot stop incorrect transactions from being embedded in a blockchain, but only prevent those transactions from updating the blockchain database’s state. While in the long term we expect an Ethereum-compatible virtual machine to be offered as a high-level abstraction within MultiChain, we don’t think it’s the right solution for low-level validation.

MultiChain 2.0 will introduce a different paradigm called transaction filters, which validate individual transactions without reference to any global state. We expect filters to be written in Javascript and executed within an embedded runtime engine such as v8, which is used in Google’s Chrome browser and the Node.js platform. Of course, we’ll need to ensure that filter code runs identically on every node in a blockchain, blocking any sources of non-determinism such as reading the time, using random numbers, accessing network or disk, or performing math operations that depend on the host server’s architecture. Creating a deterministic Javascript runtime environment is a challenge, but (without giving too much away) we believe it will be useful for several other MultiChain features in future.

Filters will be passed a JSON object describing an individual transaction, structured like the output of decoderawtransaction but with extra fields. For example, each transaction input in the JSON will include a structure describing the previous transaction output it spends, and each address will be accompanied by a list of permissions currently held by that address. A filter’s job is to return a Boolean value indicating whether the transaction is acceptable and if not, provide a textual error explaining why. MultiChain’s API will include commands for creating filters, testing them on previous or new transactions, and activating them subject to administrator consensus.

Unlike smart contracts, if a bug is discovered in the code for a filter, it can easily be replaced by a new version. Nonetheless, like all Turing-complete code, filters still run the risk of entering an infinite loop. This problem will be mitigated in two ways:

• Filters can only be installed and activated by the chain’s administrators, subject to consensus. This gives each administrator the opportunity to examine a filter’s code in depth before voting for it to be activated.
• All well-behaved nodes will validate new transactions using the active filters before forwarding them on to their peer nodes. As a result, if a transaction sends a filter into an infinite loop, the transaction should not propagate beyond the node which created it.

We expect one popular application for filters to be validating stream items. For example, a filter could ensure that certain fields in a stream’s JSON items contain numbers in a specific range. In MultiChain 1.0 this type of validation has to be done at the application level, either when writing stream items (if the source is trusted) or when reading them. By contrast, MultiChain 2.0 will enable these rules to be embedded within the blockchain itself, rather like check constraints in a relational database.

MultiChain 2.0 will include two additional features to make filters even more powerful. First, it will introduce user-defined permissions, which exist alongside the eight permissions defined by MultiChain. As with regular permissions, these will be granted to specific addresses by administrators (and in some cases, by users with activate privileges) and included alongside addresses in the JSON object passed to a filter. For example, a filter could ensure that only addresses with a particular user-defined permission can write certain types of data to a stream, or transact in a particular asset above a certain threshold.

Second, MultiChain 2.0 will support custom (binary or JSON) metadata within regular transaction outputs. This will enable any output to act as a general database row, “owned” by the address within. Filters will see any metadata within a transaction’s spent and created outputs as part of its JSON description. As a result, MultiChain will become a universal shared database engine, where a transaction’s validity is determined by a customizable function of the rows it creates and deletes. (If this sounds a little abstract, we’ll be sure to provide some concrete examples.)

Blockchain updating

Since blockchains are designed to run for many years, their characteristics might need to be changed over time. The current version of MultiChain already provides a fair degree of flexibility, allowing permissions changes (including of administrators and miners by consensus), new assets and streams to be created, and nodes to be seamlessly added or removed from the network. Nonetheless, in MultiChain 1.0 a blockchain’s basic parameters, such as the maximum block size and target confirmation time, are fixed when the chain is created and cannot be subsequently changed.

MultiChain 2.0 will add the ability to update a blockchain, allowing many (but not all) of its parameters to be modified while the chain continues to run. Like other important operations, updating a blockchain will require a customizable level of administrator consensus, where this level itself is a parameter that can be changed. Updates will come into effect from a certain block, and apply thereafter to every subsequent block until the next update.

Blockchain parameters that can be updated will include:

• Protocol version. This will enable a blockchain created with one version of MultiChain to be upgraded to support the features in a new version, such as JSON stream items or transaction filters. Indeed, the protocol version 10008 introduced in MultiChain 1.0 alpha 29 (and used in the beta) has already been future-proofed with undocumented support for this type of upgrade. Once a MultiChain 1.0 blockchain is upgraded to the 2.0 protocol, it will also gain access to the other parameter changes described here.
• Blockchain scaling. Blockchains that become popular may outgrow the initial values set for their target confirmation time or maximum transaction and block sizes. MultiChain 2.0 will allow these values to be increased or decreased as necessary.
• Permissioning model. MultiChain 2.0 will allow the updating of many parameters relating to permissioning and governance, including: (a) anyone-can-* parameters that control the ways in which a blockchain is open or closed, (b) admin-consensus-* parameters that determine the levels of administrator consensus required for certain operations, and (c) the mining-diversity parameter that controls the strictness of the round-robin consensus algorithm.

Once this updating functionality is implemented, there should be no reason why a blockchain created in MultiChain cannot run for many decades or more.

Looking ahead

We’ve already started work on MultiChain 2.0, and look forwards to delivering on this roadmap. No doubt other enhancements will be included as well. As with MultiChain 1.0, we’ll have alpha releases along the way, so that developers can use and learn new features as they are implemented (and of course, report any problems or shortcomings). Naturally, we’ll continue to maintain version 1.0 throughout this period, fixing any bugs that appear.

I’d like to finish by thanking our development team, led by Dr Michael Rozantsev, for their continued excellence and hard work. We see MultiChain as a straightforwards software engineering project, in which code quality and testing counts above all. It’s my privilege to work with people who can turn a complex product vision into stable working software with such remarkable efficiency and speed.

Please post any comments on LinkedIn.

“The highest good, than which there is no higher, is the blockchain, and consequently it is immutably good, hence truly eternal and truly immortal.”
— Saint Augustine, De natura boni, i, 405 C.E. (with minor edits)

If you ask someone well-informed about the characteristics of blockchains, the word “immutable” will invariably appear in the response. In plain English, this word is used to denote something which can never be modified or changed. In a blockchain, it refers to the global log of transactions, which is created by consensus between the chain’s participants. The basic notion is this: once a blockchain transaction has received a sufficient level of validation, some cryptography ensures that it can never be replaced or reversed. This marks blockchains as different from regular files or databases, in which information can be edited and deleted at will. Or so the theory goes.

In the raucous arena of blockchain debate, immutability has become a quasi-religious doctrine – a core belief that must not be shaken or questioned. And just like the doctrines in mainstream religions, members of opposing camps use immutability as a weapon of derision and ridicule. The past year has witnessed two prominent examples:

• Cryptocurrency advocates claiming that immutability can only be achieved through decentralized economic mechanisms such as proof-of-work. From this perspective, private blockchains are laughable because they depend on the collective good behavior of a known group of validators, who clearly cannot be trusted.
• Scorn poured on the idea of an editable (or mutable) blockchain, in which retroactive modifications can be made to the transaction history under certain conditions. Mockers posed the question: What could possibly be the point of a blockchain if its contents can easily be changed?

For those of us on the sidelines, it’s fun to watch the mudslinging. Not least because both of these criticisms are plain wrong, and stem from a fundamental misunderstanding of the nature of immutability in blockchains (and indeed any computer system). For those short on time, here’s the bottom line:

In blockchains, there is no such thing as perfect immutability. The real question is: What are the conditions under which a particular blockchain can and cannot be changed? And do those conditions match the problem we’re trying to solve?

To put it another way, a blockchain’s transactions are not written into the mind of God (with apologies to Augustine above). Instead, the chain’s behavior depends on a network of corporeal computer systems, which will always be vulnerable to destruction or corruption. But before we get into the details of how, let’s proceed by recapping some basics of blockchains themselves.

Blockchains in brief

A blockchain runs on a set of nodes, each of which may be under the control of a separate company or organization. These nodes connect to each other in a dense peer-to-peer network, so that no individual node acts as a central point of control or failure. Each node can generate and digitally sign transactions which represent operations in some kind of ledger or database, and these transactions rapidly propagate to other nodes across the network in a gossip-like way.

Each node independently verifies every new incoming transaction for validity, in terms of: (a) its compliance with the blockchain’s rules, (b) its digital signature and (c) any conflicts with previously seen transactions. If a transaction passes these tests, it enters that node’s local list of provisional unconfirmed transactions (the “memory pool”), and will be forwarded on to its peers. Transactions which fail are rejected outright, while others whose evaluation depends on unseen transactions are placed in a temporary holding area (the “orphan pool”).

At periodic intervals, a new block is generated by one of the “validator” nodes on the network, containing a set of as-yet unconfirmed transactions. Every block has a unique 32-byte identifier called a “hash”, which is determined entirely by the block’s contents. Each block also includes a timestamp and a link to a previous block via its hash, creating a literal “block chain” going back to the very beginning.

Just like transactions, blocks propagate across the network in a peer-to-peer fashion and are independently verified by each node. To be accepted by a node, a block must contain a set of valid transactions which do not conflict with each other or with those in the previous blocks linked. If a block passes this and other tests, it is added to that node’s local copy of the blockchain, and the transactions within are “confirmed”. Any transactions in the node’s memory pool or orphan pool which conflict with those in the new block are immediately discarded.

Every chain employs some sort of strategy to ensure that blocks are generated by a plurality of its participants. This ensures that no individual or small group of nodes can seize control of the blockchain’s contents. Most public blockchains like bitcoin use “proof-of-work” which allows blocks to be created by anyone on the Internet who can solve a pointless and fiendishly difficult mathematical puzzle. By contrast, in private blockchains, blocks tend to be signed by one or more permitted validators, using an appropriate scheme to prevent minority control. Our product MultiChain uses a technique called “mining diversity” which requires a minimum proportion of the permitted validators to participate in order to create a valid chain.

Depending on the consensus mechanism used, two different validator nodes might simultaneously generate conflicting blocks, both of which point to the same previous one. When such a “fork” happens, different nodes in the network will see different blocks first, leading them to have different opinions about the chain’s recent history. These forks are automatically resolved by the blockchain software, with consensus regained once a new block arrives on one of the branches. Nodes that were on the shorter branch automatically rewind their last block and replay the two blocks on the longer one. If we’re really unlucky and both branches are extended simultaneously, the conflict will be resolved after the third block on one branch, or the one after that, and so on. In practice, the probability of a fork persisting drops exponentially as its length increases. In private chains with a limited set of validators, the likelihood can be reduced to zero after a small number of blocks.

Nonetheless, it’s important to remember that each node is running on a computer system owned and controlled by a particular person or organization, so the blockchain cannot force it to do anything. The purpose of the chain is to help honest nodes to stay in sync, but if enough of its participants choose to change the rules, no earthly power can stop them. That’s why we need to stop asking whether a particular blockchain is truly and absolutely immutable, because the answer will always be no. Instead, we should consider the conditions under which a particular blockchain can be modified, and then check if we’re comfortable with those conditions for the use case we have in mind.

Mutability in public chains

Let’s return to the two examples cited in the introduction, in which the doctrine of immutability has been used as a basis for ridicule. We’ll begin with the claim that the consensual validation procedures used in permissioned blockchains cannot bring about the “true immutability” promised by public chains.

This criticism is most easily addressed by pointing to the vulnerability of public blockchains themselves. Take, for example, the Ethereum blockchain, which suffered a devastating exploit in June 2016. Someone found a coding loophole in a smart contract called “The DAO”, in which almost $250 million had been invested, and began draining its funds at speed. While this clearly violated the intentions of the contract’s creators and investors, its terms and conditions relied on the mantra that “code is law”. Law or not, less than a month later, the Ethereum software was updated to prevent the hacker from withdrawing the cryptocurrency “earned”.

Of course, this update could not be enforced, since every Ethereum user controls their own computer. Nonetheless, it was publicly supported by Vitalik Buterin, Ethereum’s founder, as well as many other community leaders. As a result, most users complied, and the blockchain with the new rules kept the name “Ethereum”. A minority disagreed with the change and continued the blockchain according to its original rules, earning the title “Ethereum Classic”. A more accurate choice of names might be “Ethereum compromised” and “Ethereum the pure”. Either way, democracy is democracy, and (the pragmatic and popular) “Ethereum” is now worth over ten times (the idealistic but sidelined) “Ethereum Classic”.

Now let’s consider a less benevolent way in which public blockchain immutability can be undermined. Recall that block creation or “mining” in bitcoin and Ethereum uses a proof-of-work scheme, in which a mathematical problem must be solved in order to generate a block and claim its reward. The value of this reward inevitably turns mining into an arms race, with miners competing to solve the problems faster. To compensate, the network periodically adjusts the difficulty to maintain a constant rate of block creation, once every 10 minutes in bitcoin or 15 seconds in Ethereum.

In the last 5 years, bitcoin’s difficulty has increased by a factor of 350,000×. Today, the vast majority of bitcoin mining takes place on expensive specialized hardware, in locations where the weather is cold and electricity is cheap. For example, $1,089 will buy you an Antminer S9, which mines blocks 10,000 times faster than any desktop computer and burns 10 times more electricity. This is all a long way from the democratic ideals with which bitcoin was created, even if it does make the blockchain extremely secure.

Well, kind of secure. If someone wanted to undermine the immutability of the bitcoin blockchain, here’s how they would do it. First, they would install more mining capacity than the rest of the network put together, creating a so-called “51% attack”. Second, instead of openly participating in the mining process, they would mine their own “secret branch”, containing whichever transactions they approve and censoring the rest. Finally, when the desired amount of time had passed, they would anonymously broadcast their secret branch to the network. Since the attacker has more mining power than the rest of the network, their branch will contain more proof-of-work than the public one. Every bitcoin node will therefore switch over, since the rules of bitcoin state that the more difficult branch wins. Any previously confirmed transactions not in the secret branch will be reversed, and the bitcoin they spent could be sent elsewhere.

By now, most bitcoin believers will be laughing, because I wrote “install more mining capacity than the rest of the network put together” as if this is trivial to achieve. And they have a point, because of course it’s not easy, otherwise lots of people would already have done it. You need a lot of mining equipment, and a lot of electricity to power it, both of which cost a ton of money. But here’s the inconvenient fact that most bitcoiners brush over: For the government of any mid-size country, the money required is still small change.

Let’s estimate the cost of a 51% attack which reverses a year of bitcoin transactions. At the current bitcoin price of $1500 and reward of 15 bitcoins (including transaction fees) per 10-minute block, miners earn around $1.2 billion per year ($1500 × 15 × 6 × 24 × 365). Assuming (reasonably) that they are not losing money overall, or at least not losing much, this means that total miner expenses must also be in the same range. (I’m simplifying here by amortizing the one-time cost of purchasing mining equipment, but $400 million will buy you enough Antminer 9s to match the current bitcoin network’s mining capacity, so we’re in the right ball park.)

Now think about the reports that bitcoin is being used by Chinese citizens to circumvent their country’s capital controls. And consider further that the Chinese government’s tax revenues are approximately $3 trillion per year. Would a non-democratic country’s government spend 0.04% of its budget to shut down a popular method for illegally taking money out of that country? I wouldn’t claim that the answer is necessarily yes. But if you think the answer is definitely no, you’re being more than a little naive. Especially considering that China reportedly employs 2 million people to police Internet content, which totals $10 billion/year if we assume a low wage of $5,000. That puts the $1.2 billion cost of reversing a year of bitcoin transactions in perspective.

Even this analysis understates the problem, because the Chinese government could undermine the bitcoin network much more easily and cheaply. It appears that the majority of bitcoin mining takes place in China, due to low-cost hydroelectric power and other factors. Given a few tanks and platoons, China’s army could physically seize these bitcoin mining operations, and repurpose them to censor or reverse transactions. While the wider bitcoin world would undoubtedly notice, there’s nothing it could do without fundamentally altering the governance structure (and therefore nature) of bitcoin itself. What was that about censorship free money?

None of this should be construed as a criticism of bitcoin’s design, or a prediction that a network catastrophe will actually happen. The bitcoin blockchain is a remarkable piece of engineering, perhaps even perfect for the purpose its creator(s) had in mind. And if I had to put money on it, I would bet that China and other governments probably won’t attack bitcoin in this way, because it’s not in their ultimate interest to do so. More likely, they’ll focus their wrath on its more untraceable cousins like Dash, Zcash and Monero.

Nonetheless, the mere possibility of this form of interference puts the cryptocurrency immutability doctrine in its place. The bitcoin blockchain and its ilk are not immutable in any perfect or absolute sense. Rather, they are immutable so long as nobody big enough and rich enough decides to destroy them. Still, by relying on the economic cost of subverting the network, cryptocurrency immutability satisfies the specific needs of people who don’t want to trust governments, companies and banks. It may not be perfect, but it’s the best they can do.

Rewriteable private chains

Now let’s move on to private blockchains, designed for the needs of governments and large companies. We can begin by noting that, from the perspective of these organizations, immutability based on proof-of-work is a commercial, legal and regulatory non-starter, because it allows any (sufficiently rich) actor to anonymously attack the network. For institutions, immutability can only be grounded in the good behavior of other similar institutions, with whom they can sign a contract and sue if need be. As a bonus, private blockchains are far less costly to run, since blocks only need a simple digital signature from the nodes that approve them. So long as a majority of validator nodes are following the rules, the end result is stronger and cheaper immutability than any public cryptocurrency can offer.

Of course, immutability is still easy to undermine if all the participants in a chain decide to do so together. Let’s imagine a private blockchain used by six hospitals to aggregate data on infections. A program in one hospital writes a large and erroneous data set to the chain, which is a source of inconvenience for the other participants. A few phone calls later, the IT departments of all the hospitals agree to “rewind” their nodes back one hour, delete the problematic data, and then allow the chain to continue as if nothing happened. If all the hospitals agree to do this, who’s going to stop them? Indeed, apart from the staff involved, who will even know that it happened? (It should be noted that some consensus algorithms like PBFT don’t provide an official mechanism for rollbacks, but this doesn’t help with governance since nodes are still free to bypass the rules.)

Now consider a case where most of a private blockchain’s participants agree to rewind and remove some transaction, but a few withhold their consent. Since every organization’s node is under its ultimate control, nobody can force the minority to join the consensus. However, by sticking to their principles, these users will find themselves on a fork being ignored by everyone else. Like the virtuous proponents of Ethereum Classic, their place in heaven may well be assured. But back here on earth, they will be excluded from the consensus process for which the chain was deployed, and might as well give up completely. The only practical application of transactions outside the consensus is to serve as evidence in a court of law.

With this in mind, let’s talk about the second case in which the doctrine of blockchain immutability has been used to ridicule ideas. Here, we’re referring to Accenture’s idea of using a chameleon hash to enable a block buried deep in a chain to be easily replaced. The primary motivation, as described by David Treat, is to allow an old problematic transaction to be quickly and efficiently removed. Under the scheme, if a block substitution does occur, a “scar” is left behind which all participants can see. (It should be noted that any later transactions that depend on the deleted one would need to be removed as well.)

It’s hard to overstate how many people poured scorn on this idea when it was announced. Twitter and LinkedIn were aghast and aflutter. And I’m not just talking about the crypto crowd, which takes sporting pleasure in mocking anything related to enterprise blockchains. The idea was broadly slammed by private blockchain advocates as well.

And yet, under the right conditions, the idea of allowing blockchains to be modified retroactively via chameleon hashes can make perfect sense. To understand why, we begin with a simple question: in this type of blockchain, who would actually have the power to replace old blocks? Clearly, it can’t be any unidentified network participant, because that would render the chain ungovernable.

The answer is that a chameleon hash can only be used by those who hold its secret key. The key is required to enable a new version of a block, with different transactions, to be given the same chameleon hash as before. Of course, we probably don’t want centralized control in a blockchain, so we can make the scheme stronger by having multiple chameleon hashes per block, each of whose key is held by a different party. Or we might use secret sharing techniques to divide a single chameleon hash key between multiple parties. Either way, the chain can be configured so that a retroactive block substitution can only occur if a majority of key holders approve it. Is this starting to sound familiar?

Allow me to render the parallel more explicit. Let’s say that we share control over chameleon hashes between those same validating nodes which are responsible for block creation. This means that an old block can only be replaced if a majority of validating nodes agree to do so. And yet, as we discussed earlier, any blockchain can already be retroactively modified by a majority of validating nodes, via the rewind and replay mechanism. So in terms of governance, chameleon hashes subject to a validator majority make no difference at all.

If so, why bother with them? The answer is: performance optimization, because chameleon hashes allow old blocks to be substituted in a chain far more efficiently than before. Imagine that we need to remove a transaction from the start of a blockchain that has been running for 5 years. Perhaps this is due to the European Union’s right to be forgotten legislation, which allows individuals to have their personal data removed from companies’ records. Nodes can’t just wipe the offending transaction from their disks, because that would change the corresponding block’s hash and break a link in the chain. The next time the blockchain was scanned or shared, everything would fall apart.

To solve this problem without chameleon hashes, nodes would have to rewrite the early block without the problematic transaction, calculate the block’s new hash, then change the hash embedded in the next block to match. But this would also affect the next block’s own hash, which must be recalculated and updated in the subsequent block, and so on all the way along the chain. While this mechanism is possible in principle, it could take hours or days to complete in a blockchain with millions of blocks and transactions. Even worse, while engaged in this process, a node may be incapable of processing new incoming network activity. So chameleon hashes provide a far more computationally efficient way to achieve the same goal. If you imagine a bad transaction as a rock buried many miles underground, chameleon hashes can teleport the rock to the surface, instead of making us dig all the way down, retrieve the rock, and fill in the hole.

Immutability is nuanced

By reviewing the risks of proof-of-work blockchains and the technical value of chameleon hashes, I hope to have convinced you that blockchain immutability is far more nuanced than a “yes or no” question. To quote Simon Taylor quoting Ian Grigg, the question must always be “who are you and what do you want to achieve?”

For cryptocurrency believers who want to avoid government-issued money and the traditional banking system, it makes perfect sense to believe in a public proof-of-work blockchain, whose immutability rests on economics rather than trusted parties. Even if they must live with the possibility of a large government (or other wealthy actor) bringing down the network, they can take solace in the fact that this would be a painful and expensive operation. And no doubt they hope that cryptocurrencies will only get more secure, as their value and mining capacity continues to grow.

On the other hand, for enterprises and other institutions that want to safely share a database across organizational boundaries, proof-of-work immutability makes no sense at all. Not only is it astoundingly expensive, but it allows any sufficiently motivated participant to anonymously seize control of the chain and censor or reverse transactions. What these users need is immutability grounded in the good behavior of a majority of identified validator nodes, backed by contracts and law.

Finally, for most permissioned blockchain use cases, we probably don’t want validator nodes to be able to easily and cheaply substitute old blocks in the chain. As Dave Birch said at the time, “the way to correct a wrong debit is with a correct credit”, rather than pretending that the debit never took place. Nonetheless, for those cases where we do need the extra flexibility, chameleon hashes help make blockchains a practical choice.

Please post any comments on LinkedIn.

Watch a video of a half hour talk given at the CityChain17 conference in London this month, organized by MBN Solutions. Below is a list of the topics covered, with links to more in-depth information in brackets:

• The basics of blockchains, compared to traditional messaging and centralized databases (more).
• Dispelling some myths about how blockchains are different from, or better than, regular databases.
• The key trade-off (disintermediation vs confidentiality) when considering using a blockchain (more).
• An in-depth look at two proposed applications of blockchains, only one of which makes sense.
• The four general categories of strong blockchain use cases that we know so far (more).
• Platforms which call themselves “blockchains” but don’t fulfill the technology’s promise (more).
• A brief description of the MultiChain open source blockchain platform (more).

I should also clarify that we’re not affiliated with IBM, but the conference was hosted at an IBM location.

Today we’re delighted to announce the first beta release of MultiChain 1.0 for Linux and Windows, after more than two years of intensive development. As we’ve said before, our definition is very specific: "beta" means that there are no known bugs or major shortcomings. So the purpose of the beta period is to ensure than any unknown issues are discovered through our own testing, as well as that of our growing user base.

The beta period will last a few months, with the final release of MultiChain 1.0 due in summer 2017. Apart from testing, we’ll also be using this time for further performance optimization. While we (and several partners) have benchmarked MultiChain at ~500 tx/sec on mid-range hardware, we’re planning to push that further for the 1.0 release.

We’re also announcing the addition of 14 companies to the MultiChain Platform Partner Program, bringing the total number to 27. The new members include three multinational consultancies: Boston Consulting Group, PricewaterhouseCoopers LLP and Worldline. Eleven other smaller companies, many of which are focused on blockchain application development, have also joined: Auxesis Group, Crossword Cybersecurity, Cryptologic, Enuke Software, Enuma Technologies, InfoCorp Technologies, Kunstmaan, Minddeft Technologies, Primechain Technologies, RecordsKeeper and Satoshi Citadel Industries. The partner program involves both technical and marketing cooperation, including a listing on the MultiChain website, which now receives over 25,000 visitors monthly.

Apart from partners using MultiChain to deliver client projects, we’re also seeing a different type of collaboration emerge, where software vendors integrate MultiChain into their platform to enable peer-to-peer data sharing and collaboration. Today we’re pleased to jointly announce that we’re working with Seal Software, the market leading platform for contract discovery and analytics. Seal will leverage MultiChain to allow a multiparty contract’s terms and conditions to be shared, negotiated and managed directly on a peer-to-peer basis between the parties involved. This follows our recent announcement of a similar collaboration with Wolfram Research, the company behind Mathematica.

Thank you for joining us on our journey so far. The full text of today’s press release is included below.

MultiChain Adds New Partners and Enters Beta

March 30, 2017 – Coin Sciences Ltd is delighted to announce the addition of fourteen companies to the MultiChain Platform Partner Program, a new collaboration with Seal Software, and the first beta release of MultiChain 1.0.

New members of the Platform Partner Program include three multinational consulting companies: Boston Consulting Group, PricewaterhouseCoopers LLP and Worldline. Eleven other smaller companies have also joined: Auxesis Group, Crossword Cybersecurity, Cryptologic, Enuke Software, Enuma Technologies, InfoCorp Technologies, Kunstmaan, Minddeft Technologies, Primechain Technologies, RecordsKeeper and Satoshi Citadel Industries. This brings the total number of program members to 27, which includes founding partners Accenture, D+H and Mphasis. A full list is now available at: http://www.multichain.com/platform-partners/

Coin Sciences Ltd is also pleased to announce a new collaboration with Seal Software, the award winning platform for contract discovery and analytics. Seal is to integrate MultiChain as a key component of its platform, combining Seal’s machine learning framework for intelligent contracts with MultiChain’s advanced streams functionality. This will enable a single view of a multiparty contract’s terms and conditions to be shared, negotiated and managed on a peer-to-peer basis between the parties involved.

Together with these new partnerships, Coin Sciences is today announcing the first beta release of MultiChain 1.0 for Linux and Windows. The beta release represents the culmination of over 2 years of intensive development, driven relentlessly by feedback from developers building on the platform. During the beta period, additional testing and optimization will take place, with the final version 1.0 release due in summer 2017.

“Since launching the platform partner program in October, dozens of companies have reached out to us,” said Dr Gideon Greenspan, CEO and Founder of Coin Sciences Ltd. “Many had already built projects for their customers on MultiChain, and welcomed the opportunity to create a formal relationship. Others, such as Seal Software and Wolfram Research, sought to integrate MultiChain into their software platforms to enable peer-to-peer data sharing and collaboration. With version 1.0 of MultiChain now entering beta, we look forward to continued cooperation with all our partners, helping them leverage MultiChain for their customers’ needs.”

“Bringing two advanced technologies together like this takes the use of AI- based contract analytics to a whole new level,” said Kevin Gidney, Seal Software’s CTO and co-founder. “I am delighted to be working with Coin Sciences and their MultiChain platform on this opportunity.”

“Joining the MultiChain Platform Partner Program underpins our commitment to maintaining a deep understanding of the different blockchain fabrics available, helping us advise our clients on electing the right solution,” said Seamus Cushley, Director of the PwC EMEA Blockchain team. “PwC has developed blockchain solutions across a wide range of industries with a number of underlying fabrics, including successful engagements using MultiChain. Joining this program offers us early access to releases and priority access to their internal engineering team and we look forward to working with Coin Sciences more closely in the future.”

“We are very pleased to partner with Coin Sciences Ltd,” said Kaj Burchardi, Managing Director of Boston Consulting Group’s Platinion Netherlands division. “In our experience, the MultiChain platform provides an efficient opportunity to implement use cases in permissioned distributed ledger environments. We are convinced that the recent open source code release will give the platform another push in terms of implementations.”

“We are using MultiChain for building several blockchain powered solutions, including shared KYC / AML, syndication of loans / consortium lending, trade finance, asset registry, asset re-hypothecation, secure documents, cross-border payments and peer-to-peer payments,” said Shinam Arora, CEO of Primechain Technologies Pvt. Ltd. “We find MultiChain very powerful yet easy to use.”

Please post any comments on LinkedIn.

Today we’re delighted to jointly announce a collaboration with Wolfram Research, the industry-leading company behind the Mathematica platform and the Wolfram|Alpha answer engine. Over the coming year, MultiChain will be integrated into the Wolfram Language and across Wolfram’s line of products. For example, Mathematica users will be able to store and retrieve data in a zero-configuration private blockchain deployed in the Wolfram Cloud, in their own blockchain running on local MultiChain nodes, or in a chain which combines nodes from both.

We’re particularly pleased with this collaboration because it demonstrates our long-running view that, as a technology, blockchains are in no way specific to the finance sector. The perceived association between banks and blockchains is an accident of history, stemming from the fact that most public blockchains, like bitcoin, happen to enable a new type of money. By contrast, private or permissioned blockchains are a shared database technology, allowing a set of participants or organizations to safely collaborate on a database that crosses boundaries of trust.

Of course, banks work extensively with general-purpose databases such as Oracle or SQL Server, and the same will apply for blockchains as well. For example, MultiChain is already being used for dozens of projects in the finance sector. But it’s important to be clear: Blockchains embody a particular set of trade-offs compared to centralized databases, which make them suitable for some use cases and unsuitable for others. Quite rightly, many “distributed ledger” startups focused on specific problems in capital markets are not using blockchains at all.

As for MultiChain, we hope this will be the first of many collaborations enabling software platforms to offer a decentralized data layer to their users. We’re already working with several other companies to explore the possibility in depth. The underlying problem is this: How can multiple organizations ensure they agree on a developing set of facts, without giving control over those facts to a single party? Blockchains combine peer-to-peer networking, public key cryptography, consensus algorithms and a new transaction model, to provide a practical solution.

The full text of the joint press release is included below.

Wolfram to Integrate with MultiChain Blockchain Platform

March 8, 2017 – Wolfram Research and Coin Sciences Ltd today announced a collaboration in which Coin Sciences’ market-leading MultiChain blockchain platform will be made accessible from the Wolfram Language and across the Wolfram family of products, including the award-winning Mathematica. This will enable Wolfram Language users to write applications that make use of blockchain functionality.

“A blockchain is a decentralized ledger that has append-only memory,” said Christian Pasquel, Wolfram’s Development Team Lead. “This is a very simple concept, but it is a powerful building block that will enable a new class of computational applications—we expect everyone who uses the Wolfram Language will benefit from this simple yet powerful functionality.”

Initially, Wolfram will offer users direct access to a zero-configuration private blockchain, accessible in the Wolfram Cloud. Users will be able to create their own blockchain(s)—combining local nodes with those in the Wolfram Cloud, have it store and retrieve data and also expose information about that blockchain to other parties over the web.

“We’re delighted that Wolfram Research has chosen to integrate MultiChain functionality into their flagship product,” said Dr. Gideon Greenspan, founder and CEO of Coin Sciences Ltd. “We have always viewed blockchains as a general-purpose technology for creating peer-to-peer shared databases, so this integration with the Wolfram Language is a perfect fit. Despite the focus which blockchains have received in the finance sector, they are equally useful for decentralized data aggregation and collaboration in many other fields, such as science, engineering, economics and government.”

The integration with MultiChain is slated for release with a Wolfram Language version update later this year.

About Wolfram Research, Inc.

Wolfram is a powerhouse in technical innovation and has been defining the computational future for three decades. As the creator of Mathematica, Wolfram|Alpha and the Wolfram Language, Wolfram is the leader in developing technology and tools that inject sophisticated computation and knowledge into everything. Learn more at http://www.wolfram.com.

About Coin Sciences Ltd

Coin Sciences Ltd develops the popular MultiChain (http://www.multichain.com) blockchain platform. MultiChain includes features such as permissions management, native assets, data streams and simple configuration and deployment. It has been used successfully for blockchain projects in many of the world’s largest banks, consulting firms, financial technology and IT companies.

Please post any comments on LinkedIn.

Two years after starting to develop MultiChain, we’re delighted to release its source code under the GNU General Public License (GPLv3). The code, along with compilation instructions for Ubuntu, is now available at Github. You are free to browse and review it, compile it for yourself, or fork MultiChain in accordance with the GPL license.

Why now?

The code was originally scheduled for release with the first beta of MultiChain 1.0, but we decided to bring it forwards, as source code access has become crucial for many of our users and platform partners. Releasing the code allows enterprise users of MultiChain to perform independent security audits, and guarantees freedom of choice in the unlikely event that we stop developing the product.

So why did we wait so long? First, we needed to invest time in tidying up the code for public consumption, and preferred until recently to focus our efforts on pushing the product forwards. With the feature set for version 1.0 nearing completion, we could spare the distraction. Second, we didn’t want to be too helpful to some of our competitors who seemed rather desperate to see MultiChain’s code, judging by the (ahem) peculiar phone calls and email requests we’ve received. Now that the product is reasonably mature and well known, this is less of a concern.

Business models

If MultiChain is open source, how will we generate the revenue necessary to support its long-term development? To start with, we are already offering Service Level Agreements (SLAs) to customers who need guaranteed response and solution times for their questions and problems. Even though MultiChain is still officially in alpha, we already know of cases where it’s being used in production in the finance and government sectors.

In parallel to offering SLAs, we’ve started preparing the groundwork for a premium version of MultiChain, which will include extra features relating to security, scalability, analytics and performance. If you’re already working with the free version of MultiChain, there are two important things to know about the premium product. First, it will be possible to connect free and premium nodes in a single network, so each participant can independently decide which version to use. Second, any applications built on MultiChain today will work unmodified on the premium version – all APIs and parameters will remain backwards compatible.

Roadmap to 1.0 beta

In the meantime, we still have more to do before MultiChain 1.0 reaches beta. A full list can be found in the TODO file inside the source code repository, but here are some of the most important items:

• Add support for automatic “checkpoints” in a node, to permanently lock down changes in a blockchain’s governance model (admin and mining permissions).
• Allow control over the mining of empty blocks. This is useful for minimizing disk usage in blockchains with periods of low activity.
• Add a “mining turnover” parameter, which balances between (a) all permitted nodes mining blocks at random, and (b) round-robin mining which prevents forks but can still recover quickly if a mining node goes down.
• Finish the mechanism for notifying external processes of new transactions relating to a wallet address and/or subscribed stream/asset.
• Increase the maximum size of transaction metadata (whether raw or as part of a stream item) from the current limit of 8 MB to at least 32 MB (and hopefully more).
• Review and reduce the size of logs and other files whose primary purpose is to help with debugging.
• Complete the port of MultiChain to Mac OS.

The first three of these have already been implemented (see the development branch on Github). We’re hoping to complete the rest, along with smaller tweaks and changes, by the end of Q1 2017.

The beta phase

We define a “beta” version as “without known shortcomings”, i.e. when we’re not aware of a single bug or important unaddressed issue in the product. So the purpose of the beta phase, which will probably last 6 months or so, is to enable any hidden problems to be discovered through our user base and internal test suite, both of which continue to grow. No doubt we’ll also receive feature requests during this period, but we’ll only implement those which are very low risk in terms of product stability. Major new features will have to wait until MultiChain 1.1, 1.5 or 2.0, as appropriate.

However, one aspect of development will continue during the beta phase – performance optimization. MultiChain’s transaction throughput, which can reach 800 tx/sec under ideal conditions, is already more than enough for most blockchain applications. Nonetheless, some use cases require more, and there is no reason why MultiChain cannot reach thousands of tx/sec with the appropriate optimizations. Naturally, we won’t be making any significant architectural changes during the beta phase. Instead, we will focus on local optimizations, such as caching intermediate results.

Beyond 1.0 and Premium

Apart from the well-defined path to MultiChain 1.0 and its premium version, what’s the longer term roadmap for the MultiChain platform? How do we see the product developing over the next five to ten years?

I should start by clarifying that, as a technology, we don’t see blockchains as specific to banks or the finance sector. While platforms such as MultiChain can indeed be used to implement shared ledgers of financial assets, their applications go far wider. We view blockchains as a fundamentally new type of database, which can be directly shared between separate companies or organizations, without requiring a central intermediary. This ability to span trust boundaries sets blockchains apart from today’s common database platforms, whether they are of the SQL, NoSQL or NewSQL variety. Indeed, in the long term, we should probably call these “peer-to-peer databases” rather than “blockchains”, because a product’s purpose is more important than a description of its underlying technology.

Version 1.0 of MultiChain provides three high-level abstractions for peer-to-peer database application development: permissions (to control access and activity), assets (ownership tokens that are transferred or exchanged), and streams (general purpose data storage and retrieval). Over the coming years, we’ll be studying the strongest use cases for this new type of database, to see what else should be added to this list.

We already know of some obvious possibilities, such as virtual machines and zero-knowledge asset transactions. But the more interesting abstractions will probably be those that we can’t yet imagine. What is the blockchain equivalent of foreign keys in relational databases, map-reduce in big data stores, or the HyperLogLog of in-memory databases? As we continue developing MultiChain in conversation with our users and partners, we intend to find out.

Please post any comments on LinkedIn.

About 18 months have passed since the finance sector woke up, en masse, to the possibilities of permissioned blockchains, or to use the more general term, “distributed ledgers”. The period since has seen a tsunami of activity, including research reports, strategic investments, pilot projects, and the formation of many consortia. No one can accuse the banking world of not taking the potential of this technology seriously.

Naturally, the explosive growth in blockchain projects has driven the development of permissioned blockchain platforms, on which those projects are built. For example, our product MultiChain has tripled in usage over the past year, whether we measure web traffic, monthly downloads or commercial inquiries. And of course, there are many other platforms, such as BigChainDB, Chain, Corda, Credits, Elements, Eris, Fabric, Ethereum (deployed in a closed network), HydraChain and Openchain. Not to mention still more startups who have developed some kind of blockchain platform but have not made it publicly available.

For companies wishing to explore and understand a new technology, an abundance of choice is generally a good thing. However, in the case of blockchains, which still remain loosely defined and poorly understood, this cornucopia comes with a significant downside: many of the available “blockchain” platforms don’t actually address the core problem they are meant to solve. And what is that problem? Allow me to quote the succinct video definition by Richard Gendal Brown, CTO of R3, in full:

A distributed ledger is a system that allows parties who don’t fully trust each other to come to consensus about the existence, nature and evolution of a set of shared facts without having to rely on a fully trusted centralized third party.

To take an extreme example, consider a bunch of Lego bricks tied together with string. If we use the term “block chain” to describe this fashion item, who’s to say that we’re not describing it accurately? And yet, that particular chain of blocks will not help multiple parties to safely and directly share a database without a central intermediary. Similarly, many “blockchain” platforms do something related to chains of blocks, but also lack the necessary properties to serve as the basis for a peer-to-peer database.

Another chain of blocks that does not help with database sharing – source.

Minimum viable blockchain

In order to understand the basic requirements of a distributed ledger, it helps to clarify how these systems differ from regular databases, which are controlled by a single entity. For example, let’s consider a simple system for tracking who owns a particular company’s shares. The ledger, as implemented in a database, has one row for each owner containing two columns: the owner’s identifier, such as their name, and the corresponding quantity of shares.

Here are six crucial ways in which this system could fail its users:

• Forgery: Transferring shares from one person to another without the sender’s permission.
• Censorship: Refusing to fulfill someone’s request to transfer some shares elsewhere.
• Reversal: Undoing a transfer that took place at some point in the past.
• Illegitimacy: Changing the total quantity of shares in the system without a corresponding action by the issuer.
• Inconsistency: Giving different responses to inquiries from different users.
• Downtime: Not responding to incoming requests for information at all.

Because of all these possibilities, the shareholders must maintain a high level of trust in whoever is managing this ledger on their behalf. Building and running an organization worthy of that trust comes with substantial hassle and cost.

Blockchains or distributed ledgers remove the need for this kind of central database operator, by allowing the users of a database to interact directly with each other on a peer-to-peer basis. In our example, the stockholders could safely hold their shares on a blockchain which they collectively manage, and make transfers to each other instantly over that chain. (The disadvantage is a significant loss of confidentiality between the chain’s users, which we won’t address here but I’ve previously discussed at length.)

All this brings us back to the question of blockchain platforms. In order to serve as a viable basis for peer-to-peer database sharing, a blockchain has to protect its participants against all six types of database failure – forgery, censorship, reversal, illegitimate transactions, inconsistency and downtime. While many products in the market fulfill these requirements, quite a few of them come up short. I call these blockchains “half-baked” because they may address some of these risks, but not all. In some respects at least, the database’s users remain dependent on the good behavior of a single participant, which is precisely the scenario we want to avoid.

These half-baked blockchains come in any number of varieties, but three archetypes stand out as the most common or obvious. I’m not going to name individual products because, well, I don’t want to offend. The blockchain startup community is small enough that most of us know each other through conferences and other meetings, and the interactions tend to be positive. Nevertheless, if blockchains (in the sense of useful peer-to-peer databases) are ever going to emerge as a coherent product category, it’s important to distinguish between half-baked and real solutions.

The one validator blockchain

One pattern we’ve seen a few times is a blockchain in which only one participant can generate the blocks in which transactions are confirmed. Transactions are sent to this one node instead of being broadcast to the network as a whole, so their acceptance is subject to this party’s whims rather than some kind of majority consensus. Still, once a block has been built by this central party, it is broadcast to the other nodes in the network, who can independently confirm the validity of the transactions within, and record the new block locally and permanently.

To return to our six forms of database malfunction, this type of blockchain is far from useless. Transactions must be digitally signed by the entity whose funds they move, so they cannot be forged by the central party. They cannot be reversed because each node maintains its own copy of the chain. And transactions cannot perform illegal operations like creating assets out of thin air, because every node independently validates each transaction for correctness. Finally, each node maintains its own copy of the database, so its content is always available for reading.

Unfortunately, four out of six is not enough. The validating node can easily censor individual transactions, by refusing to include them in the blocks it creates. Even if the operators of this node are honest, a system or communications failure can render it unavailable, causing all transaction processing to come to a halt. In addition, depending on the setup, the validating node may be able to transmit different versions of the blockchain to different participants. In terms of censorship and consistency, the database still contains a single point of failure, on which all the other nodes rely.

One platform offers a twist on this scheme, in which blocks are centrally generated by a single node, but a quorum of other designated nodes signs them to indicate consensus. In terms of the risk of inconsistency, this certainly helps. The nodes in the quorum will only lend their signatures to a single version of the blockchain, which can therefore be considered as authoritative. Nonetheless, the quorum nodes cannot help if the block generator censors transactions, or loses its connection to the Internet. Ultimately, this type of blockchain still uses a hub-and-spoke architecture, rather than a peer-to-peer network.

The shared state blockchain

Technically speaking, there are many similarities between blockchains and more traditional distributed databases such as Cassandra and MongoDB. In both cases, transactions can be initiated by any node in the network, and must reach all the other nodes as part of a consensus about the database’s developing state. Both blockchains and distributed databases have to cope with latency (communication delays which stem from the distance between nodes) and the possibility of some nodes and/or communication links intermittently failing.

Distributed databases have been around for a while, so any blockchain platform developer would do well to understand their consensus algorithms and the strategies they use to globally order transactions and resolve conflicts. Nonetheless, it’s important not to take the comparison too far, because blockchains must contend with a crucial additional challenge – an absence of trust between the database’s nodes. Whereas distributed databases focus on providing scalability, robustness and high performance within a single organization’s boundaries, blockchains must be redesigned in order to safely traverse those boundaries.

To return to our six types of database risk, a node in a distributed database need only worry about downtime, i.e. the possibility of other nodes becoming unavailable. Nodes can safely assume that every transaction and message on the network is valid, and are not concerned with forgery, censorship, reversal, illegitimacy or inconsistency. Their worst problem is dealing with two simultaneous but valid transactions, initiated on different nodes, which affect the same piece of data. Solving these conflicts is by no means trivial, but it’s still a lot easier than worrying about “Byzantine faults“, in which some nodes deliberately act to disrupt the functioning of others.

A database can only be shared safely across trust boundaries if nodes treat all activity on the network with a certain degree of suspicion. For example, every transaction which modifies the database must be individually digitally signed since, in a peer-to-peer architecture, there is no other way to know its true point of origin. Similarly, every incoming message, such as the announcement of a new block, has to be critically assessed for its content and context. Unlike in distributed databases, nodes must not be able to immediately and directly modify another node’s state.

Some “blockchain” platforms have been developed by starting with a distributed database, and sprinkling some features on top to make them more blockchainy. For example, by grouping transactions into blocks and storing hashes (digital fingerprints) of those blocks in the database, they aim to add a form of immutability. But unless each node can be sure that its list of hashes cannot be modified by another node, this type of immutability is easily gamed. The standard response to these criticisms is that every security problem can be solved with sufficient time and coding. But this is rather like holding some prisoners in an open field, and trying to stop them escaping with tripwires and ditches. It’s far safer to use a purpose-built concrete structure, whose doors are locked and whose windows are barred.

The one cloud blockchain

By far the strangest phenomenon I’ve seen is blockchain platforms which can only be accessed through their developer’s cloud-based platform-as-a-service. To be clear, we’re not talking about some of a blockchain’s participants choosing to host their nodes on their cloud provider of choice, such as Microsoft Azure or Amazon Web Services. Rather, this is a blockchain which can only be accessed through APIs exposed by the servers of a company “hosting” it.

Let us grant, for argument’s sake, that a centralized blockchain provider genuinely has a group of nodes running under its control. What difference does this make to the users of the system who are sending API requests and receiving responses? The participants have no way of assessing if everyone’s transactions have been processed without omission or error. Perhaps the central service is malfunctioning, or perhaps it is censoring or reversing some transactions deliberately. And if you believe the blockchain provider has no reason to do this, why not use them to host a regular centralized database instead? You’ll get a more mature product with better performance, and suffer none of the risks of working with new technologies. In short, centralized blockchains are about as useful as Lego on a string.

Solving the mystery

We’ve now seen three types of platform which market themselves as “blockchains”, and indeed make some use of a chain of blocks, but which don’t solve the fundamental problem for which these systems are designed. To recap, this is to enable a single database to be safely and directly shared across trust boundaries, without a central intermediary.

Apart from pointing at this peculiar phenomenon, I believe it’s instructive to consider what might underlie it. Why are so many blockchain startups building products which don’t fulfill the promise of this technology, often achieving no more than traditional centralized or distributed databases? Why are so many talented people wasting so much of their time?

I can see two main classes of explanation – technical and commercial. To start with the technical, it is rather tricky to create distributed consensus systems which can tolerate one or more nodes behaving maliciously in unpredictable ways. In the case of MultiChain, we somewhat cheated, by using bitcoin’s battle-hardened reference implementation as a starting point, and then replacing proof of work by a structurally similar consensus algorithm called “mining diversity”. Teams developing a blockchain node from scratch have to think deeply about asynchronous and adversarial processes – a combination which few programmers have experience of. I can certainly understand the temptation to take a shortcut, such as using a single node to generate blocks, or piggybacking on an existing distributed database, or only running nodes in a trusted environment. Choosing any of these undoubtedly makes life easier for developers, even if this undermines the entire point.

As for commercial reasons, every startup seems to be approaching the blockchain opportunity from a different angle. Here at Coin Sciences, we’re focused on becoming a (database) software vendor, so we’re distributing MultiChain for free while developing a premium node with additional features. Other startups want to sell subscription services, so they will naturally build a platform which customers cannot host themselves. Some are hoping to centrally control a blockchain or help their partners to do so (an odd ambition for a disintermediation technology!) and are naturally drawn to consensus algorithms that rely on a single node. And finally, there are companies whose primary goal is to sell consulting services, in which case their platform need not function at all, so long as its website brings in some large customers.

Perhaps another issue is that some blockchain companies are being run by people who are undoubtedly bursting with talent, but lack a deep understanding of the technology itself. In startups carving out a new field, it’s probably vital for strategic decisions to be taken by people who understand the nature of that field and how it differs from what came before. Not a few blockchain startups appear to have painted themselves into a corner by pursuing a product vision which is attractive to their customers, but cannot actually be built.

As a user of blockchains, how can you avoid being caught by these fallacies? When evaluating a particular blockchain platform, be sure to ask whether it fulfills the six requirements of safe peer-to-peer database sharing: prevention of downtime and inconsistency, as well as transaction forgery, censorship, reversal and illegitimacy. And beware of explanations that consist of too much mumbling or hand waving – they probably mean that the answer is no.

Please post any comments on LinkedIn.

Last Friday saw the launch of Zcash, a new public blockchain and associated cryptocurrency that attracted a lot of attention. By now, there are hundreds of cryptocurrencies, so any budding young entrant needs a serious differentiator to rise above the fray. In the case of Zcash, this is easy – Zcash users can send money to each other in absolute privacy. For a cryptocurrency based on a blockchain, this is a remarkable technical achievement. (Though it should be noted that other chains such as Monero and Dash aim at the same goal using simpler but less effective means.)

As I’ve written about before, in a general sense blockchains (whether public or private) represent a trade-off in which disintermediation is gained at the cost of confidentiality. Blockchains provide a clever new way for participants to safely share a database, even if they do not trust each other, without requiring a central intermediary. But there’s a price to pay for this peer-to-peer decentralization – the “node” belonging to every participant in the chain must verify every transaction for itself, and this in turn means that it sees what everyone else is doing.

Two ways to chain

In the case of public blockchains and cryptocurrencies, the shared database serves primarily as a record of who controls (and so effectively owns) how much cryptocurrency, with an optional sprinkling of “metadata” (bitcoin) or contractual logic (Ethereum) on top. By contrast, in private blockchains, we tend to see two major classes of use case: (a) the ownership and transfer of external assets represented by tokens on the chain, and (b) more general applications relating to data storage and retrieval. For example, in our own product MultiChain, these two classes of use case are implemented using native assets and data streams respectively.

When it comes to general data storage, the blockchain provides a number of services: proving where a piece of data comes from, timestamping it, and notarizing it immutably to prevent modification by a minority of blockchain participants. But the blockchain need have nothing to say about the data itself – each application can decide what a piece of data means, and whether it is valid. Bad data can simply be ignored at the application level, without causing harm to the blockchain’s state as a whole.

By contrast, if blockchains are directly transferring tokenized assets, they must apply internal rules regarding the validity of those transfers. To put it simply, an event such as “Alice pays Bob one Euro” will only be approved by the chain if Alice has at least one Euro to her name. While different types of blockchain express this rule in different ways (bitcoin transaction constraints vs Ethereum smart contracts), they all share the property that Alice’s finances must be known by every node in the chain. This allows them to assess whether her payment is valid, know how much Bob has as a result, and evaluate any future payments from Bob to Charlie and others.

At this point, readers familiar with blockchains will point out that Alice and Bob are not directly identified by name on a chain. Instead, each transacts under one or more “addresses”, which are long alphanumeric strings of gibberish that bear no relation to their real-world identities. While this is true, in reality it does not help a great deal, because there are several ways in which the connection between users and their addresses can be inferred.

First and most simply, in order to transact with someone on a blockchain, I need to know at least one of their addresses. So if I send them some money, I can see where that money goes next, and if they’re paying me, I can see where it came from. Second, if I happen to know something about a participant from the real world (e.g. what types of assets they trade at what time of day), I can search the chain’s activity for corresponding patterns, and then infer their address with a high level of confidence. Finally, once I know one address of a participant, I can often work out which other addresses they own and use, by monitoring the full flow of funds on the chain. While this is not trivial to achieve, it is certainly possible with sufficient motivation, as proven by companies such as Chainalysis and Skry who make a living providing this type of “network analysis” for bitcoin.

Saved by encryption?

The contrast between assets and data touches directly on the question of encryption. In the case of general data storage on a blockchain, we can encrypt the information stored, while still gaining the benefits of data provenance, timestamping and immutability. None of these features need insight into the data itself. Therefore it is perfectly valid for two participants to use a blockchain to store information which only they can read, while still gaining the benefit of other participants committing to the origin of that data and its existence at a certain point in time.

By contrast, encryption of this nature cannot be used by transactions that represent transfers of tokenized assets. If Alice and Bob were to encrypt their transaction, then the assets in question could not be used safely by any other participant in the chain, because nobody else would know where the assets actually are. The assets would cease to have any collective meaning on the chain, which destroys the entire point.

In the finance sector, this conflict between privacy and liquidity is the core difficulty of using blockchains to transfer assets, dashing the hopes of many a startup in the space. While the technical feasibility of moving assets over a blockchain has been proven by countless pilot projects, in practice this causes too much activity to be revealed between peers. Information leakage is a disadvantage at the best of times, but it’s a complete showstopper when a chain’s participants are in fierce competition, or where regulation forbids it.

As a result, many prominent “distributed ledger” startups have moved away from the idea of on-chain settlement, reverting to more traditional bilateral transactions which are encrypted and notarized on a blockchain under the “general data storage” paradigm. This can prevent disputes and double spends, but settlement itself remains external to the chain. While the blockchain is still providing some value, it is less transformative than originally hoped. No doubt there have been more than a few red-faced meetings between startups and their investors.

And yet, after all the disappointment, salvation may finally be at hand. Enter the zero knowledge blockchain.

Introducing zero knowledge

Before discussing this new type of blockchain, it’s helpful to understand the principle of zero knowledge itself. In a general sense, a zero knowledge proof is one which demonstrates the truth of a certain statement, without revealing any additional information beyond what it’s trying to prove.

To take an example, let’s say I have a color blind friend who owns two pens, which are identical except that one is green and one is blue. My friend cannot distinguish between them, and I want to convince her that they are indeed different. Of course, I can’t do this by simply telling her the colors, because she can’t assess if I’m lying or not.

So what can I do? (Why not take a minute and try to work out the answer yourself…) Well, I can ask her to take a piece of paper, and draw two lines on it in another room. When doing this, she can freely decide whether to use the same pen for both lines, or one pen for each. From her perspective the result looks the same either way. Then she comes back in with the paper, and I tell her whether she used one pen or two. Of course, if the pens were the same color, I would have no way of knowing. So the fact that I get it right proves they are different.

Well, not quite. There’s a problem with this logic. Even if the pens were identical I would still have a 50% chance of giving the right answer, because there are only two possibilities (she used one pen or two). So one lucky guess proves nothing at all. In order to strengthen my case, the game must be played over multiple rounds. After every round, my chance of being consistently right goes down by half. So with 5 rounds, I have a 1 in 32 chance of successfully faking. With 10 rounds, it’s 1 in 1024, and with 20 rounds, 1 in 1048576 – in other words, one in a million. Depending on my friend’s relative level of boredom and suspicion, she can reach any probabilistic level of proof that she desires, although never absolute certainty.

Bring on the snarks

Zero knowledge proofs in blockchains apply a similar principle, though of course they’re not about the color of pens. Rather, they aim to prove the statement “this transfer of assets is valid”, without revealing anything important about the transfer itself. Zcash uses a relatively new technique for zero knowledge proofs called zk-SNARKs, the full explanation of which is (to put it mildly) beyond the scope of this piece. But the basic idea is this: any computational condition can be represented by an arithmetic circuit, which takes some data as input and gives an answer of “true” or “false” in response. A zk-SNARK uses a model of this circuit to let me prove, to any desired degree of certainty, that I possess an input which gives a true response, without revealing the input itself. Philosophically at least, this is like proving that two pens are different colors, without revealing what those colors are.

A zk-SNARK uses a neat little trick to avoid the interactivity that is typical of zero knowledge proofs, in which a skeptical party repeatedly presents a challenge to the one making a claim. In the case of our pens, this challenge is my friend’s choice between using one or two pens in each round. This type of interactivity is not feasible on a blockchain because there is no trusted central party to set the challenges. Instead, a zk-SNARK uses an approximation of a “random oracle” in which the challenges are created deterministically by some code, but behave for all intents and purposes as if they were random. Not by coincidence, this combination of determinism and unpredictability uses the same kind of hash function that secures a blockchain itself.

Zero knowledge proofs have been around for a while, but zk-SNARKs introduce a number of innovations that render them usable in blockchains. Most importantly, zk-SNARKs reduce the size of the proofs and the computational effort required to verify them. Zerocoin, a previous attempt at using zero knowledge proofs in blockchains, requires 45 kb transactions, each of which takes half a second to check (figures taken from the white paper on which Zcash is based). This is drastically worse than bitcoin, whose transactions are typically 0.3 kb in size and can be verified in under a millisecond. By contrast, Zcash transactions weigh in at 1kb and can be checked in under 6 milliseconds. This puts Zcash in the same scalability league as bitcoin – a remarkable achievement. If we took our hats off to the creator(s) of bitcoin, we should take our socks and shoes off for this.

Caution advised

Before you convert all your bitcoin to Zcash, there are some caveats to bear in mind. First, Zcash’s cryptography relies on a trusted setup process, in which two long public keys are derived from a single randomly-generated private one. It is absolutely vital that this private key is destroyed, since anyone who possesses it can forge the proofs on which the system relies. In the case of Zcash, the private key was created in an elaborate ceremony, described in detail here. The ceremony involved several well known characters from the cryptocurrency world, each of whom (we are told) had only a partial view of the private key. In turn, this means that Zcash can only be compromised if all of the ceremony’s participants colluded maliciously. It is up to the reader to decide how confident they feel about that.

Second, even though it is relatively quick to verify an anonymous Zcash transaction, creating each of these transactions carries a serious computational burden. According to the Zcash Speed Center, it currently takes 48 seconds on a high-end server, and over 3 GB of memory. This makes it impractical to transact anonymously from mobile devices and older desktops and laptops. Zcash partially works around this limitation by supporting both regular visible cryptocoins (with fast transactions) and anonymous “notes” (with slow ones), with a built-in method for converting between the two.

Third, even if we assume that the underlying cryptography is sound, there could be bugs lurking in the Zcash code which allow anonymous notes to be conjured out of thin air. This would allow the Zcash monetary base to be limitlessly inflated, ultimately rendering the cryptocurrency worthless. Unlike transparent cryptocurrencies like bitcoin, this catastrophic event cannot be detected, because the entire point of Zcash is keeping transactions hidden. Nonetheless, according to Zooko Wilcox, the Zcash CEO, work is already under way to find a solution, so we can look forward to seeing it.

Finally, as with any cryptocurrency based on proof-of-work, the potential for 51% attacks remains. This means that a group of “miners” with over half of the network’s computational power can collude to reverse transactions that everyone else thought were complete (bad miners still cannot fake transactions which steal others’ funds). Zcash smartly relies on Equihash, a different hashing algorithm from bitcoin’s SHA-256, meaning that the huge mass of existing bitcoin mining power cannot be turned against Zcash. Equihash is also designed to be more resistant to the “ASICs” (special purpose microprocessors) that have turned bitcoin mining into an oligopoly, but only time will tell if hardware engineers can find a workaround, and at what cost.

Zero knowledge private blockchains

So far, we’ve focused our discussion on the public Zcash blockchain and cryptocurrency. But what about external assets moving over private or permissioned blockchains and shared ledgers? Can the same zero knowledge techniques be used?

On a technical level, the answer is undoubtedly yes. Compared with the theoretical and technological tour de force that underlies Zcash, it’s trivial to extend the protocol to support assets issued on a chain. All that’s required is to extend the conditions proven by a zk-SNARK to enforce the preservation of multiple assets, instead of a single cryptocurrency. Or even more simply, create multiple distinct anonymous subsystems on a single blockchain, each representing a different type of asset, and transact within each subsystem exactly as Zcash does today. This second method would require no understanding of zk-SNARKs at all.

How would an asset’s life cycle look in this model? First, a trusted entity issues tokens representing the asset, by sending a visible blockchain transaction certifying those tokens’ value. The same entity would then perform a second transaction which converts the visible tokens into anonymized Zcash-style “notes”, effectively moving the asset underground. These notes can then be secretly transferred from the issuer to others, and onwards among the chain’s participants. As with Zcash, the transfer transactions can be verified as valid by all blockchain participants without revealing their content. Finally, when a holder wishes to redeem a note, they convert it back into visible tokens using another Zcash-style transaction, send those tokens to the original issuer, and receive the equivalent real-world asset in return. We might also allow notes to be directly redeemed anonymously, in which case blockchain participants would not know how much of the asset remains in circulation.

So zero knowledge transactions promise to untie the Gordian knot which has prevented blockchains from being used for settlement in the finance sector. To recap, in a regular blockchain transaction, when an asset is sent from one bank to another, the details of that transaction are visible to every other bank on the chain. By contrast, in a zero knowledge transaction, the others only know that a valid transaction has taken place, but nothing about the sender, recipient, asset class (if we’re clever) and quantity. Even the volume of transactions can be obfuscated by participants regularly creating fake transactions in which they send assets to themselves.

In terms of privacy, this is as good as a gold bar travelling in a briefcase from one bank to another, but without the cost and time of physically moving the gold. And it’s better than using a trusted intermediary such as a custodial bank, because there isn’t even that single party who sees everything going on. For the first time, zero knowledge blockchains allow asset transfers to be digitally performed on a peer-to-peer basis, in perfect secrecy.

Don’t throw out that database (yet)

Assuming that Zcash’s technical fundamentals are sound, I fully expect it to reach the top tier of cryptocurrencies in terms of developer interest and market capitalization. But is there a similarly bright future for zero knowledge transactions in private blockchains? Will they make the transition from the laboratory to production-quality systems moving real money around the world?

It is, of course, far too early to tell. But there are a number of questions that need answering before permissioned blockchain advocates can point to zero knowledge transactions and triumphantly declare victory.

First, and most importantly, is this safe? Can we really be confident that both the underlying cryptography and its coded implementations are strong enough to prevent a malicious party from generating assets out of thin air? As mentioned earlier, unlike transparent blockchains, it is not yet possible to detect if the monetary base of a zero knowledge blockchain has been compromised. Still, there is no surer test of this technology than releasing it as an open public blockchain that is available for all to see and attack, and this is exactly what Zcash is doing. After several years of seeing Zcash running smoothly, institutions may become convinced that zero knowledge blockchains can genuinely safeguard their assets. As with all matters blockchain, patience is required.

A related issue is the novelty of zero knowledge cryptography itself. It’s true that regular blockchains rely on advanced cryptography – namely, asymmetric encryption (public/private keys) and cryptographic hash functions (digital fingerprints). And it’s also true that the great majority of blockchain programmers and application developers don’t understand the mathematical principles which underlie these techniques. But the broader point is this: if treated as black boxes, these methods have been widely employed for decades, by a huge number of developers and users (heard of https?) and everyone believes that they work. By contrast, until recently zero knowledge proofs were only known to a small community of academics, and didn’t have broad applications on the Internet or elsewhere. We can expect this obscurity to reduce the willingness of a bank’s CIO or risk officer to move their core processes to zero knowledge blockchains, at least for the next five years. And let’s not even start to imagine how long it will take regulators to get comfortable with assets moving around in this way.

Talking about regulation brings up another practical issue with zero knowledge blockchains. Anonymous transactions in a blockchain contain statements regarding asset transfers and ownership, but those statements are only visible to selected parties (namely, those directly involved). Even if we give a regulator full visibility into a zero knowledge blockchain and its participants’ identities, it has no way of knowing what is truly happening within. Of course, the regulator could ask all of the participants to identify and reveal their transactions, and they can do this efficiently using Zcash-style “viewing keys”. Nonetheless, if the parties to any particular transaction want to keep it secret, the regulator is stuck, and does not know who to fine. There is no custodial bank from whom it can obtain the full picture, and the only option for enforcement is to shut down the entire chain.

So what’s the bottom line? For now at least, I suggest simply following the progress of the public Zcash blockchain, to see how it develops and grows. If the history of Ethereum is repeated, there will be surprises and vulnerabilities lurking under the surface, waiting to be exploited by greedy opportunists. Nonetheless, in the longer term, make no mistake: zero knowledge transactions are a game-changing breakthrough for blockchains. If the underlying cryptographic principles prove sound, expect them to significantly broaden the range of use cases to which blockchains can be applied.

Please post any comments on LinkedIn.

It’s our pleasure to launch the MultiChain Platform Partner Program, starting with 13 member companies from Europe, Asia and North America. The program facilitates our working relationship with the growing number of consultants and integrators building blockchain solutions on the MultiChain platform.

The initial list of members includes three large multinational solution providers: Accenture, D+H and Mphasis. These are joined by ten more companies, most of which specialize in blockchain application development: ANX International, Cubichain Technologies, DXMarkets, Hashcove, KrypC Technologies, Lexington Innovations, Motivian, regio iT, SettleMint and Vanbex Group.

Members of the program have knowledge and experience working with MultiChain, and enjoy a direct line to our team for prioritized assistance and problem resolution. In addition, they gain early access to selected MultiChain releases and can use MultiChain branding in their marketing materials. Best of all, they are promoted through the MultiChain website, which now receives over 20,000 visitors per month (still growing fast!)

So if you’re looking to implement a blockchain project, we recommend talking to these partners – a detailed list with descriptions and locations is available here. And if you’re a MultiChain solution developer and wish to join the program, please don’t hesitate to get in touch. Below is the full text of today’s press release:

October 27, 2016 – Coin Sciences Ltd is today launching the MultiChain Platform Partner Program, with the initial participation of 13 members from Europe, Asia and North America. The program is designed to facilitate our deepening relationship with the growing number of IT consulting companies and blockchain solution providers building on the MultiChain platform.

Companies participating in the MultiChain Platform Partner Program launch include three large multinational solution providers: Accenture, D+H and Mphasis. These are joined by ten more companies, most of which specialize in blockchain application development: ANX International, Cubichain Technologies, DXMarkets, Hashcove, KrypC Technologies, Lexington Innovations, Motivian, regio iT, SettleMint and Vanbex Group. A full list of participants with descriptions and locations is now available at: http://www.multichain.com/platform-partners/

MultiChain is a popular private blockchain solution, currently available as a free download for Linux and Windows. It provides a comprehensive set of features for developing and deploying blockchain applications, including permissions management, native assets, data streams and simple per-chain configuration. MultiChain extends the bitcoin protocol and Bitcoin Core APIs, making it compatible with a vast range of tools and open source code built for bitcoin, including software libraries, online explorers, mobile wallets and hardware security devices.

Members of the Platform Partner Program enjoy a close working relationship with the MultiChain engineering team, prioritized problem reporting and resolution, and early access to selected MultiChain releases. In addition, partners are promoted through the MultiChain website, which now receives over 20,000 visitors per month, and can use MultiChain branding in their own materials.

“In the past year, MultiChain has grown rapidly in usage, providing the core of blockchain projects by dozens of integrators and solution providers,” said Dr Gideon Greenspan, CEO and Founder of Coin Sciences Ltd. “This includes several companies who have built an entire toolset and application suite on the platform. The MultiChain Platform Partner Program allows us to formalize our relationship with these companies, offering them both marketing exposure and deep technical assistance.”

“We’ve been working with MultiChain for nearly two years, and have seen the platform evolve rapidly to adapt to clients’ needs, including everything from building proofs-of-concept to supporting pilot projects,” said David Treat, managing director and head of Accenture’s Financial Services blockchain practice. “MultiChain is a simple, powerful, well-documented platform that makes it easier to start a blockchain-based project. We’re pleased to work with them in the development process and in bringing blockchain-based innovations to market, particularly in financial services.”

“MultiChain is a simplified blockchain platform, which makes it extremely easy to start a permissioned blockchain-based project and build multiple use cases between organizations,” said Nitin Narkhede, Vice President and Head of the Blockchain Centre of Excellence at Mphasis. “Mphasis is glad to be associated with MultiChain in the development process and bring pilots and full-scale blockchain-based implementations to the market, especially in financial services.”

“MultiChain is the most stable and reliable blockchain software product that has been purposely-built for an enterprise environment,” said Marcelo Garcia Casil, Co-Founder and CEO of DXMarkets. “We have used it for more than a year now in a number of projects with great success.”

Please post any comments on LinkedIn.

Today we’re proud to release the latest version of MultiChain, which implements a crucial new set of functionality called “streams”. Streams provide a natural abstraction for blockchain use cases which focus on general data retrieval, timestamping and archiving, rather than the transfer of assets between participants. Streams can be used to implement three different types of databases on a chain:

1. A key-value database or document store, in the style of NoSQL.
2. A time series database, which focuses on the ordering of entries.
3. An identity-driven database where entries are classified according to their author.

These can be considered as the ‘what’, ‘when’ and ‘who’ of a shared database.

Streams basics

Any number of streams can be created in a MultiChain blockchain, and each stream acts as an independent append-only collection of items. Each item in a stream has the following characteristics:

• One or more publishers who have digitally signed that item.
• An optional key for convenient later retrieval.
• Some data, which can range from a small piece of text to many megabytes of raw binary.
• A timestamp, which is taken from the header of the block in which the item is confirmed.

Behind the scenes, each item in a stream is represented by a blockchain transaction, but developers can read and write streams with no awareness of this underlying mechanism. (More advanced users can use raw transactions to write to multiple streams, issue or transfer assets and/or assign permissions in a single atomic transaction.)

Streams integrate with MultiChain’s permissions system in a number of ways. First, streams can only be created by those who have permission to do so, in the same way that assets can only be issued by certain addresses. When a stream is created, it is open or closed. Open streams are writeable by anybody who has permission to send a blockchain transaction, while closed streams are restricted to a changeable list of permitted addresses. In the latter case, each stream has one or more administrators who can change those write permissions over time.

Each blockchain has an optional ‘root’ stream, which is defined in its parameters and exists from the moment the chain is created. This enables a blockchain to be used immediately for storing and retrieving data, without waiting for a stream to be explicitly created.

As I’ve discussed previously, confidentiality is the biggest challenge in a large number of blockchain use cases. This is because each node in a blockchain sees a full copy of the entire chain’s contents. Streams provide a natural way to support encrypted data on a blockchain, as follows:

1. One stream is used by participants to distribute their public keys for any public-key cryptography scheme.
2. A second stream is used to publish data, where each piece of data is encrypted using symmetric cryptography with a unique key.
3. A third stream provides data access. For each participant who should see a piece of data, a stream entry is created which contains that data’s secret key, encrypted using that participant’s public key.

This provides an efficient way to archive data on a blockchain, while making it visible only to certain participants.

Retrieving from streams

The core value of streams is in indexing and retrieval. Each node can choose which streams to subscribe to, with the blockchain guaranteeing that all nodes which subscribe to a particular stream will see the same items within. (A node can also be configured to automatically subscribe to every new stream created.)

If a node is subscribed to a stream, information can be retrieved from that stream in a number of ways:

• Retrieving items from the stream in order.
• Retrieving items with a particular key.
• Retrieving items signed by a particular publisher.
• Listing the keys used in a stream, with item counts for each key.
• Listing the publishers in a stream, with item counts.

As mentioned at the start, these methods of retrieval allow streams to be used for key-value databases, time series databases and identity-driven databases. All retrieval APIs offer start and count parameters, allowing subsections of long lists to be efficiently retrieved (like a LIMIT clause in SQL). Negative values for start allow the most recent items to be retrieved.

Streams can contain multiple items with the same key, and this naturally solves the tension between blockchain immutability and the need to update a database. Each effective database ‘entry’ should be assigned a unique key in your application, with each update to that entry represented by a new stream item with its key. MultiChain’s stream retrieval APIs can then be used to: (a) retrieve the first or last version of a given entry, (b) retrieve a full version history for an entry, (c) retrieve information about multiple entries, including the first and last versions of each.

Note that because of a blockchain’s peer-to-peer architecture, items in a stream may arrive at different nodes in different orders, and MultiChain allows items to be retrieved before they are ‘confirmed’ in a block. As a result, all retrieval APIs offer a choice between global (the default) or local ordering. Global ordering guarantees that, once the chain has reached consensus, all nodes receive the same responses from the same API calls. Local ordering guarantees that, for any particular node, the ordering of a stream’s items will never change between API calls. Each application can make the appropriate choice for its needs.

Streams and the MultiChain roadmap

With the release of streams, we’ve completed the last major piece of work for MultiChain 1.0, and are now firmly on the path to beta. We expect to spend the next few months expanding our internal test suite (already quite large!), finishing the Windows and Mac ports, adding some more useful APIs, updating the Explorer for streams, tweaking aspects of the consensus mechanism, releasing our web demo, and generally tidying up code and help messages. Most importantly, we’ll continue to fix any bugs as soon as they’re discovered, so that our mistakes don’t interrupt your work.

In the longer term, where do streams fit into the MultiChain roadmap? Taking a step back, MultiChain now offers three areas of high-level functionality:

• Permissions to control who can connect, transact, create assets/streams, mine/validate and administrate.
• Assets including issuance, reissuance, transfer, atomic exchange, escrow and destruction.
• Streams with APIs for creating streams, writing, subscribing, indexing and retrieving.

After the release of MultiChain 1.0 (and a premium version), what’s next in this list? If you look at the API command which is used to create streams, you’ll notice an apparently superfluous parameter, with a fixed value of stream. This parameter will allow MultiChain to support other types of high-level entity in future.

Possible future values for the parameter include evm (for an Ethereum-compatible virtual machine), sql (for an SQL-style database) or even wiki (for collaboratively edited text). Any shared entity whose state is determined by an ordered series of changes is a potential candidate. Each such entity will need: (a) APIs which provide the right abstraction for updating its state, (b) appropriate mechanisms for subscribed nodes to track that state, and (c) APIs for efficiently retrieving part or all of the state. We’re waiting to learn which other high-level entities would be most useful, to be implemented by us or by third parties via a plug-in architecture.

What about smart contracts?

In a general sense, MultiChain takes the approach in which data is embedded immutably in a blockchain, but the code for interpreting that data is in the node or application layer. This is deliberately different from the “smart contracts” paradigm, as exemplified by Ethereum, in which code is embedded in the blockchain and runs in a virtual machine. In theory, because smart contracts are Turing complete, they can reproduce the behavior of MultiChain or any other blockchain platform. In practice, however, Ethereum-style smart contracts have many painful shortcomings:

• Every node has to perform every computation, whether it’s of interest or not. By contrast, in MultiChain each node decides which streams to subscribe to, and can ignore the data contained by others.
• The virtual machine used for smart contracts has drastically worse performance than code which has been natively compiled for a given computer architecture.
• Smart contract code is immutably embedded in a chain, preventing features from being added and bugs from being fixed. This was demonstrated forcefully in the demise of The DAO.
• Transactions sent to a smart contract cannot update a blockchain’s state until their final ordering is known, because of the nature of general purpose computation. This leads to delays (until a transaction is confirmed in a block) as well as possible reversals (in the event of a fork in the chain). By contrast, MultiChain can treat each type of unconfirmed transaction in the appropriate way: (a) incoming assets immediately update a node’s unconfirmed balance, (b) incoming stream items are instantly available, with their global ordering subsequently finalized, (c) permissions changes are applied immediately and then replayed in incoming blocks.

Nonetheless, as I’ve said before, we’re certainly not ruling out smart contracts as a useful paradigm for blockchain applications, if and when we see strong use cases. However, in MultiChain smart contracts would be implemented in a stream-like layer on top of the blockchain, rather than the lowest transaction level. This will preserve MultiChain’s superior performance for simpler blockchain entities like assets and streams, while offering slower on-chain computation where it’s really needed. But there are fewer such cases than you might think.

Please post any comments on LinkedIn.

Technical addendum

All commands related to streams are documented in full in the MultiChain API page, but here is a brief summary:

• Create a stream using create stream or createfrom ... stream
• Add an item to a stream with publish or publishfrom
• Retrieve a list of streams using liststreams
• Start or stop tracking a stream with subscribe and unsubscribe
• Retrieve stream items using liststreamitems, liststreamkeyitems and liststreampublisheritems
• List stream keys and publishers with liststreamkeys and liststreampublishers
• For large stream items, retrieve the full data using gettxoutdata (see maxshowndata below)
• Control per-stream permissions with calls like grant [address] stream1.write
• View a stream’s permissions using listpermissions stream1.*

Some other developer notes relating to streams:

• The create permission allows an address to create streams.
• Relevant per-stream permissions are write, admin and activate
• New blockchain parameters: root-stream-name (leave empty for none), root-stream-open, anyone-can-create, admin-consensus-create, max-std-op-returns-count
• New runtime parameters: autosubscribe to automatically subscribe to new streams created and maxshowndata to limit the amount of data in API responses (see gettxoutdata above).
• The maximum size of a stream item’s data is fixed by the max-std-op-return-size blockchain parameter, as well as the smaller of the maximum-block-size and max-std-tx-size values minus a few hundred bytes.
• Nodes using the old wallet format cannot subscribe to streams, and should be upgraded.

After two months of intensive development and testing, we’re proud to release the latest alpha of MultiChain, with a completely rewritten in-node wallet. This new wallet transforms the performance and scalability of creating, receiving and storing transactions in MultiChain.

Before we get into the details, let me provide some context. When we began developing MultiChain, we made the decision to use Bitcoin Core, the standard node for the public bitcoin network, as a starting point. In programming terms, this means that MultiChain is a “fork” of the bitcoin software. Our primary reasoning was that bitcoin was (and continues to be) the highest valued and most battle-tested cryptocurrency ecosystem, by quite some way.

On the plus side, this decision helped us get to market quickly, compared to coding up a blockchain node from scratch. Despite the many differences between public and private blockchains, they share a large amount of technical common ground, including the peer-to-peer protocol, transaction and block structure, digital signature creation and verification, consensus rules, key management, and the need for a node API. Forking from Bitcoin Core allowed us to leverage its maturity and focus on what MultiChain adds to blockchains – configurability, permissioning and native asset support. As a result, we were able to release the first alpha in June 2015, just 6 months after starting development.

However, alongside these benefits, we also had to accept the fact that some aspects of Bitcoin Core are poorly architected. While they work just fine at small scales, their performance degrades dramatically as usage grows. With the public bitcoin network still restricted to a few transactions per second, this won’t be an issue for most Bitcoin Core users for a long time. But with private blockchains aiming for hundreds or thousands of transactions per second, we knew that, sooner or later, these bottlenecks would need to be removed.

Bitcoin Core’s wallet

The “wallet” within Bitcoin Core was always the most crucial of these pain points. Its job is to store the transactions which are of particular relevance to the node, because they involve a blockchain address which it owns or a “watch-only” address whose activity it is tracking. For example, every transaction which sends funds to or from a node must be stored in that node’s wallet. And every time a node creates a transaction, it must search for one or more “unspent outputs” of previous wallet transactions which the new transaction will spend.

So what’s wrong with the wallet we inherited from Bitcoin Core? Actually, three things:

• All wallet transactions are held in memory. This causes slow startup times and rapidly increasing memory usage.
• Many operations perform an inefficient “full scan” of every transaction in the wallet, whether old or new.
• Every transaction in the wallet is stored in full, including any arbitrary “metadata” which has no meaning from the node’s perspective and is already stored in the blockchain on disk. This is very wasteful.

The consequence is that, with around 20,000 transactions stored, Bitcoin Core’s wallet slows down significantly. After 200,000 or so, it practically grinds to a halt. Even worse, since a MultiChain blockchain allows up to 8 MB of metadata per transaction (compared to bitcoin’s 80 bytes), the wallet’s memory requirements can balloon rapidly even with a small number of transactions.

It’s important to clarify that these shortcomings apply only to Bitcoin Core’s wallet, rather than its general transaction processing capacity. In other words, it can comfortably process and store millions (or even billions) of transactions which don’t relate to its own addresses, since these are held on disk rather than in memory. For example, many popular bitcoin exchanges and wallets use Bitcoin Core as-is, but store their own transactions externally rather than inside the node.

MultiChain’s new wallet

We could have made the same demand of MultiChain users, to store their own transactions outside of the node. However this didn’t feel like the right solution because it would greatly complicate the setup and maintenance for each of a chain’s participants. So instead, we bit the bullet and rewrote the wallet from the ground up.

How does the new wallet differ? If you have any experience with databases, the answers may be obvious:

• Rather than keeping the wallet transactions in memory, they are stored on disk in a suitable format, with transactions of interest retrieved when necessary.
• Instead of performing full wallet scans, the transactions are “indexed” in various ways to enable those which fulfill particular criteria to be rapidly located.
• Any piece of transaction metadata which is larger than 256 bytes is not stored in the wallet. Instead, the wallet contains a pointer to that metadata’s position in the blockchain itself.

In other words, we’ve rebuilt the in-node wallet to be properly database-driven (using LevelDB), rather than relying on a naïve in-memory structure that can’t be searched efficiently. Unsurprisingly, the difference (as measured on a 3.4 GHz Intel Core i7) is rather dramatic:

The graphs show that, once the old wallet contains 250,000 transactions, its send rate drops to 3 tx/sec and it adds 600 MB to the node’s memory usage. By contrast, the new wallet sustains over 100 tx/sec and only adds 90 MB. We stopped testing the old wallet at this point, but even with 6-8 million stored transactions, the new wallet continues to send over 100 tx/sec, and it tops out at around 250 MB of RAM used (due to database caching).

These tests were performed under realistic conditions, with multiple addresses and assets (and therefore many unspent transaction outputs) in the node’s wallet. In an idealized scenario (one address, one asset, few UTXOs), the sustained send rate was over 400 tx/s. Either way, as part of this rewrite, we have also properly abstracted all of the wallet’s functionality behind a clean internal interface. This will make it easy to support other database engines in future, for even greater robustness and speed.

To reiterate, all of these numbers refer to the rate at which a node can create, send and store transactions in its local wallet, rather than its throughput in terms of processing transactions created by others. For general network throughput, MultiChain can currently process 200 to 800 tx/sec, depending on the hardware it’s running on. (Be skeptical of any blockchain software promising numbers like 100,000 tx/sec on regular hardware, because the bottleneck is digital signature verification, which takes real time to perform. If nodes are not verifying individual transaction signatures, a blockchain cannot possibly be used across trust boundaries, making it no better than a regular distributed database.)

To finish, I’d like to mention the next major feature coming to MultiChain, which required this wallet rewrite. This feature, called streams, provides a high-level abstraction and API for general purpose data storage on a blockchain. You can think of a stream as a time-series or key-value database, with the added blockchain-related benefits of decentralization, digital signatures, timestamping and immutability. We know of many blockchain use cases that could use this functionality, and we’re already hard at work on building it. Watch this space.

Please post any comments on LinkedIn.

Technical addendum

Starting in MultiChain alpha 22, you can verify which version of the wallet is currently running by examining the walletdbversion field of the getinfo or getwalletinfo API calls. A value of 1 means the original Bitcoin Core wallet, and 2 means the new MultiChain wallet.

If you run the new version of MultiChain on an existing chain, it will not immediately switch to the new wallet. You can upgrade the wallet by stopping the node and then re-running multichaind with the parameters -walletdbversion=2 –rescan. You can downgrade similarly using –walletdbversion=1 –rescan.

By default, when you start a node on a new chain, it will automatically use the new wallet. You can change this by running multichaind for the first time with the parameter –walletdbversion=1.

With the new wallet, all MultiChain APIs work exactly the same way as before, with the exception of the old transaction querying APIs getreceivedbyaddress, listreceivedbyaddress and listtransactions (use listwallettransactions or listaddresstransactions instead). In addition, the new wallet does not support API calls and parameters relating to Bitcoin Core’s poorly implemented and soon-to-be-deprecated “accounts” mechanism, which was never properly supported by MultiChain. These calls are safely disabled with an error message.

Last week witnessed a catastrophic event in the Ethereum ecosystem, when The DAO, a smart contract less than two months old, began rapidly leaking funds to an unknown party. Looking at the current set of Ethereum contracts, filled with casinos and self-declared Ponzi schemes, this might not seem like a big deal. That is, until you learn that over 12 million units of ether, the Ethereum cryptocurrency, had been invested in The DAO by almost 20,000 people. That’s around 15% of all the ether in existence, valued at over $250 million on June 17th.

Two days later, The DAO’s assets dipped below $100 million. Two things contributed to this precipitous fall. First, a third of its funds (as denominated in ether) had already been taken. And second, the resulting panic sent the market price of ether crashing down from its peak of over $21 to a more sobering $10.67. (At the time of publication, the price had recovered to around $14.) This second effect was a natural consequence of the first, since much of ether’s recent increase in value was driven by people buying it to invest in The DAO.

The DAO had promised to act as a new type of decentralized crowdsourcing vehicle, like Kickstarter or Indiegogo but without the middleman and regulation. It was designed to let participants pool their cryptocurrency, collectively vote on projects looking for funding, then invest and reap the future rewards. Before catastrophe struck, over 100 projects had already been proposed, most of which were related to Ethereum itself. In addition, The DAO allowed participants to withdraw their uninvested funds at any time, positioning itself as a low risk investment.

Ironically, the individual or group which drained The DAO did so by exploiting subtle errors in this withdrawal mechanism. Like all smart contracts in Ethereum, The DAO is just a piece of computer code, which is “immutably” (i.e. permanently and irreversibly) embedded in the blockchain and executed by every node in response to incoming transactions. And like any self-respecting smart contract, The DAO provides full transparency by making its source code easily accessible online. This means that anybody can independently verify its functionality but also, crucially, look for vulnerabilities. And yet, the immutable nature of blockchains prevents any such problems from being fixed.

At the end of May, several critical issues were highlighted on the outstanding Hacking Distributed blog, alongside a call for a moratorium on project proposals for The DAO. This is what we might call the ‘white hat’ approach, in which exploits are reported for the good of the community. Nonetheless nobody seemed too worried, as the problems related to skewed economic incentives rather than a risk of outright theft. Simultaneously, however, it appears that others were poring over The DAO’s code with greater self-interest – namely, to look for a way to make a ton of money. And on June 17th, someone succeeded.

Draining The DAO

In a general sense, the attack arose from the interaction between vulnerabilities in The DAO’s code and other code which was designed to exploit them. You see, when looked at in isolation, The DAO did not contain any obvious mistakes, and indeed it was only released after an extensive security audit. But with the benefit of hindsight and many more eyes, a significant number of errors have since been found.

I won’t provide a full technical description of the exploit’s mechanism here, since others have already published superb and detailed post mortems (see here, here and here). But I will explain one particular vulnerability that was present, because it has been discovered in many other smart contracts and serves as an instructive example.

Let’s say that a smart contract holds funds on behalf of a number of users, and allows those users to withdraw their funds on request. The logic for the process might look something like this:

1. Wait for a user to request a withdrawal.
2. Check if that user’s balance is sufficient.
3. If so, send the requested quantity to the user’s address.
4. Check that the payment was successful.
5. If so, deduct the quantity from the user’s balance.

This all looks eminently sensible, and rather like an ATM which gives you some cash and deducts the appropriate amount from your bank balance.

So how can this simple process go wrong? Well, it turns out that if an Ethereum address belongs to a contract rather than a regular user, then this contract can run some code in response to receiving funds. And this code can, in turn, trigger other pieces of code on the Ethereum blockchain. Crucially, it can even trigger the same piece of code that caused it to be paid in the first place.

This means that, during step 3 above, the receiving address can send a new request for withdrawal, beginning a new process at step 1 before the previous process has completed. Since the user’s balance is only reduced in step 5, a new withdrawal will be approved based on the previous balance, and the same amount will be paid out again. In response to this second payment, the receiving contract can request a third, and then a fourth, and so on until the funds are drained or some other limit is reached. At this point, the user’s balance will finally be reduced by the appropriate amount, entering the negative territory which step 2 was supposed to prevent.

The equivalent would be an ATM which delivers banknotes that trigger a free repeat withdrawal when waved at the screen. The first customer to find out could empty the ATM entirely.

This ability for a piece of code to wind up calling itself is called recursion, and is a very useful technique in general computer programming. However in the case of The DAO, it paved the way for this ruinous exploit. Nonetheless, if this had been the only problem, the attack’s potential would have been contained, because Ethereum applies a limit on how deeply recursion can occur. Unfortunately, several further bugs in The DAO amplified the effects, leading to the eventual loss of tens of millions of dollars.

Of course, if just a few lines of The DAO’s code had been written differently, none of this could have happened. For example, in the 5-step process above, if the user’s balance is reduced before the funds are sent, then recursive calling would be perfectly safe. But sadly, even if its creators’ intentions were pure, The DAO’s actual code was deeply flawed. And computers have a nasty habit of blindly following the instructions they are given, even if a five year old can see that the results don’t make sense. Having been embedded immutably in the Ethereum blockchain, the faulty DAO was granted stewardship over hundreds of millions of dollars by a horde of naïve investors, and then spectacularly went up in flames. The DAO turned out to be a complete and utter shambles, and it can never be fixed.

The trouble with code

Tempting as it might be, I’m not here to haul The DAO’s programmers over the technical coals. Looking at the underlying source code, it seems reasonably well architected, with good function and variable names and clear internal documentation. While none of this proves its quality, there tends to be a high correlation between how code looks and how well it functions, for the same reason that CVs with poor punctuation warn of sloppy employees. In any event I don’t doubt that The DAO’s authors are competent developers – indeed, the fact that it passed an extensive code review suggests that the basic logic was sound.

So if the problem is not the people who worked on this project, or the work they produced, what is it? It is the fact that writing large pieces of bug-free code is extremely hard, if not impossible. I’ve worked with some truly outstanding programmers in my career, the sort who can crank out code at ten times the average developer’s pace, and with ten times fewer defects. And yet, even these remarkable individuals make mistakes which lead to software malfunctions. Donald Knuth, possibly the greatest computer programmer of all time, made a famous promise to provide an exponentially increasing financial reward to each person who found a bug in his TeX typesetting software. And he’s sent out more than a few checks.

To be clear, I’m not talking about silly slip-ups with names like “off-by-one”, “uninitialized variable” and “operator precedence”. These often cause a visible failure the first time a program is run, and can be easily spotted by reviewing the local piece of code in which they reside. And I’m not even talking about security vulnerabilities like “unvalidated inputs”, “SQL injection” and “buffer overflows”, which might not show up in a program’s regular usage, but should nonetheless be front of mind for every experienced developer.

Rather, I’m talking about trickier problems like “race conditions” and “deadlocks”. These arise from conflicts between parallel processes and tend to only show up intermittently, making them hard to detect and reproduce. As a result, they can only be understood by considering a system as a whole and how its constituent parts interact. This is much harder than regular programming, because it requires developers to think beyond the individual piece of code that they’re working on. It’s not unusual for coders to spend several days “debugging” in order to nail one of these problems down. And this is precisely the sort of holistic thinking that was needed to foresee how The DAO might be vulnerable.

With all of these difficulties, one might legitimately wonder why our increasingly code-driven world isn’t crumbling around us. Luckily, most software has three critical factors working in its favor – gradual adoption, regular updates and time.

Here’s how it works: A new software product is created to answer an emerging market need. At first, the market is small, so only a few people know they need the product. And since the product is new, an even smaller number of them will actually find it. These “early adopters” are a brave and hardy bunch who enjoy living on the technological edge, despite the associated risks. So they try out the new product, see some stuff they like, ask for a bunch of things that are missing and, best of all, report any problems encountered. Every good software entrepreneur knows to shower these people with love and assistance, and thank them for every single morsel of feedback they provide. Because while it sucks to hear about a defect in your product, it sucks a lot more not to hear about it.

Ideally, within a month or less, a new version of the product is released, fixing the reported bugs and adding some requested features. The early adopters are happy and more feedback flows in, as the latest version is put through its paces, and round it goes again. As the market grows, the number of people using the product increases. And as the product steadily improves, more and more of these people tell others about it. Even better, the more people that use the product, the more likely it is that someone, somewhere, will create that precise and unlikely situation in which an obscure bug will appear. With a bit of luck, they will let you know, and you will scratch your head in disbelief, ask for more information, eventually find and resolve the problem, and breathe a sigh of relief.

With few exceptions, this is how today’s software development works, because it is the most efficient way to create outstanding products. Of course, a good software team will also develop an extensive internal test suite, to catch as many errors as possible before they reach users, and ensure that new versions don’t break anything that previously worked. But still, most of us also rely on our user bases, because there is simply no way that we can afford to imagine and test every possible way in which our products might be used. And if you think this doesn’t apply to the big guys, you couldn’t be more wrong. How many “automatic updates” have been downloaded to your Windows, Mac or Linux system in the past year? And if you’re using Chrome or Firefox, your web browser now updates itself automatically and silently, an average of once per month.

This iterative process takes considerable time, by which I mean a few years or more. Still, after a product has been in development for long enough, and its user base has grown large enough, and those users have been (unknowingly) testing it in enough different situations, something magic happens. This magic is called “maturity”, and it’s what every software product must strive to achieve. Maturity means that a product works really well for pretty much everybody that uses it, and there are no shortcuts to getting there. But if you get the timing right, your product will mature at around the time that your target market coalesces, i.e. when large numbers of customers are actually willing to stump up and pay for it. And then, as they say, verily shall ye profit.

On immutable code

So here we come to the fundamental problem with smart contracts, as demonstrated so forcefully by The DAO:

By design, smart contracts are immutably embedded in a blockchain, and so cannot be updated. This prevents them from reaching maturity.

In previous posts, I’ve discussed other problems with smart contracts, such as their effect on blockchain performance and the fact that they are less powerful than many people imagine. For these and other reasons, we have not (yet) implemented smart contracts in the MultiChain blockchain platform. But until I witnessed the failure of The DAO, I hadn’t given enough thought to a much more fundamental issue: any non-trivial smart contract is likely to contain defects that cannot be fixed.

For the modern software developer, unfixable code is an out-and-out nightmare, setting the bar higher than most are able to reach. But we do encounter this kind of code in some situations, such as the design of the microprocessors which lie at the heart of every computer and smartphone. This code, written in languages like Verilog and VHDL, defines the physical layout of a silicon chip, which cannot be changed once manufactured. In situations like these, we tend to see several characteristics: (a) the code is written in a language that was designed with safety in mind, (b) large numbers of people work on it for several years, (c) it is subject to extensive automated testing and formal verification, and (d) if the final product is shipped with a defect, the cost of a recall falls squarely on the shoulders of the party responsible (see for example the infamous Pentium bug).

It goes without saying that none of this applies to the creators of The DAO, or indeed any other smart contract. But code immutability isn’t the only challenge for smart contract developers. A number of other factors conspire to make Ethereum considerably more dangerous than most computing environments:

• As discussed earlier, most contracts reveal their source code, to gain the trust of potential users. This makes bugs easy to find and exploit. While regular code can be fixed when a problem is found, with immutable code only attackers get to benefit.
• As in most programming languages, one “function” (piece of code) on the blockchain is able to “call” (trigger) another, to create cascading effects. However Ethereum is unusual in enabling direct function calls between the code written by parties who do not know each other and whose interests may collide. This is a perfect recipe for adversarial and unexpected behavior.
• As mentioned previously, if one Ethereum contract sends funds to another, the latter has the opportunity to execute some code in response. This code can be deliberately designed to cause the send operation to fail, potentially triggering all sorts of further havoc.
• When one function calls another, and this second function calls a third, a “stack” of calls and sub-calls is created. Keeping track of this stack carries a computational cost, so Ethereum includes a “call stack limit” which restricts how deep it can go. This is fair enough. But if the limit is reached by a particular function call, the Ethereum environment silently skips that call, rather than safely terminating the entire transaction and unwinding its effects. In other words, some code in a smart contract just might not be executed, and this non-execution can be deliberately caused by triggering that contract from a sufficiently deep stack. This strikes me as a truly abominable design choice, breaking the mental model that every software developer is accustomed to. Whoever made this decision probably should be hauled over the coals, though there is thankfully now a suggestion to change it.
• Ethereum also has a “gas limit”, which prevents abuse in public blockchains by making transactions pay for the computational resources they consume. The sender of a transaction decides how much gas they are willing to spend, and if this runs out before the transaction completes, it is safely aborted. While this is probably the best solution to a difficult problem, it can have unpleasant consequences. Some contracts turn out to need more gas than anticipated, while others cannot be run at all.
• The public Ethereum network’s cryptocurrency allows defects in smart contracts to send real money to the wrong place, with no easy method of recovery. While Ethereum miners seem to be voting in favor of a “soft fork” to freeze the funds drained from The DAO, this is not a sustainable solution.

To summarize, compared to regular centralized computer systems, Ethereum is a much more tricky environment to code for safely. And yet its principle of immutability serves to prevent buggy software from being updated. In other words, smart contracts are software whose bugs are visible, cannot be fixed, and directly control real people’s money. This, rather obviously, is a highly toxic mix.

Proponents of Ethereum-style smart contracts in private blockchains might be tempted to celebrate The DAO’s demise, but I don’t think this response is merited. With the exception of the last two points above, all of the issues with Ethereum apply equally to permissioned blockchains, which still rely on immutable smart contracts – although in this case the immutability is guaranteed by a group of identified parties rather than anonymous miners. If you want to claim that private blockchains allow buggy smart contracts to be more easily rewound, replaced or ignored, then what you’re really saying is that smart contracts serve no purpose in these blockchains at all. Put simply, if something is not meant to be immutable, it shouldn’t be stored in a blockchain. Instead, stick to good old fashioned legal documents and centralized application logic, using the chain for: (a) immutably storing the data on which that logic depends, and (b) representing the final consensual outcome of applying it. (This design pattern has been named Simple Contracts by others.)

Nonetheless the risks in the public Ethereum network are undoubtedly worse, because badly written smart contracts can rapidly and irreversibly send large amounts of real value (in the form of cryptocurrency) to users whose identity is unknown. Indeed, is there any better way for an evil genius to make a killing than: (a) writing a smart contract which looks right and fair, (b) allowing it to run safely and consistently for several years, (c) waiting for it to accumulate a large sum of money from investors, and then (d) triggering some obscure vulnerability to siphon off those funds. While I’m not suggesting that The DAO’s failure was deliberate, it will surely inspire others to make similar “mistakes”.

If I had to summarize the factors underlying Ethereum’s design, I might use the phrase “inexperienced genius”. Genius, because I believe it is a genuinely brilliant invention, adding two key innovations to the cryptocurrency systems that came before: (a) the Ethereum Virtual Machine which executes smart contracts and its method for assigning cost to computation, and (b) the use of Patricia trees to enable compact proofs of any aspect of a blockchain’s state. And yet, inexperienced as well, because some of Ethereum’s design choices are so obviously terrible, such as the silent-but-violent call stack limit, or the ability of a payment recipient to recursively trigger the code which paid it.

None of this would be a problem if Ethereum was being treated as an experiment, worthy of exploration but with critical issues remaining to be resolved. The equivalent perhaps of bitcoin during its first couple of years, when its total market capitalization didn’t go beyond a few million dollars. Unfortunately, as a result of speculation and inflated expectations, Ethereum hasn’t been given the same opportunity to find its proverbial feet. Instead, at less than one year old, it’s carrying a billion dollars in market value. Ethereum is like a toddler being forced to cook dinner, or an economics freshman chairing the Federal Reserve. I believe it’s time to recognize that the immaturity problem of individual smart contracts also applies to Ethereum as a whole.

Ethereum’s way forward

While I’m yet to see strong use cases for smart contracts in private or permissioned blockchains, I think they probably do have a place in public chains with associated cryptocurrencies. That is, if you accept the basic premise of censorship-free financial systems, which help the financially excluded and ransomware authors in equal measure. Putting this debate aside, there is certainly technical merit in a cryptocurrency which supports arbitrary logic, of the sort that cannot be implemented on “first generation” blockchains like bitcoin. For now at least, Ethereum is the first and only convincing attempt to build such a system, with a ton of money and momentum behind it.

Nonetheless, as a developer ecosystem, Ethereum appears to be fundamentally broken. While The DAO is its most costly and high profile failure, many other contracts are suffering from similar problems. So how can Ethereum clean up its act?

• Send a clear message that, at least for the next two years, nobody should send any funds to a smart contract unless they are happy to lose them in the name of self-education.
• Fix some glaring issues with the Ethereum Virtual Machine (“EVM”), namely: (a) removing the call stack limit, (b) providing a way to send ether without triggering code, and (c) allowing contracts to be marked as “non-reentrant”, meaning that their functions cannot be called while they are already in the middle of something.
• Develop a new programming language for smart contracts, which uses a more restrictive method for expressing computation that is amenable to formal proofs of correctness. Decades of research have already been invested in this field, so there is much existing work to be leveraged. (This won’t require changes to the EVM itself, since the chosen language could still be compiled into regular “bytecode”.)
• Build up an official set of secure smart contracts and functions, which have been peer-reviewed to death and proven themselves reliable in many different situations. This is akin to the standard libraries that are available for many mature programming languages. (Though at this point it’s tempting to ask: why not just hard-code the functionality of these libraries into the EVM, and enjoy much better performance as a result? Answer: Because Ethereum was specifically designed to move away from blockchains with hard-coded feature sets. But still, it does make you wonder.)

The current option, of manually intervening in response to the failure of specific smart contracts, will not be viable on a larger scale if Ethereum is to maintain its identity as a trustless and decentralized computing platform. Indeed, some make a credible case that this single judgment-based act of governance has already destroyed Ethereum’s reputation. And we should note that The DAO’s terms and conditions explicitly state that nothing “may modify or add any additional obligations or guarantees beyond those set forth in The DAO’s code”. In other words, whoever drained The DAO was acting in accordance with its published terms, and is therefore presumably on the right side of the law.

We must also accept the possibility that, after several more years of good work, Ethereum might still prove too difficult for developers to work with safely. In that case, it will languish as a matchmaking service between anonymous scammers and their foolish marks. But that wouldn’t mean it was a waste of time – at the very least, Ethereum is a fascinating experiment, from which the blockchain community will learn a lot.

In the meantime, for users of private blockchains, I can only repeat what I’ve said before:

If your application doesn’t require smart contracts, then use a simpler blockchain architecture.

Whereas this advice was previously justified in terms of performance, it is now reinforced by the apparent difficulty of getting smart contracts right. And if you’re not sure whether your use case requires smart contracts, feel free to email us with some details, and we’ll be happy to let you know.

Please post any comments on LinkedIn.

Almost a year after first releasing MultiChain, we’ve learnt a huge amount about how blockchains, in a private and non-cryptocurrency sense, can and cannot be applied to real-world problems. Allow me to share what we know so far.

To begin with, the first idea that we (and many others) started with, appears to be wrong. This idea, inspired by bitcoin directly, was that private blockchains (or “shared ledgers”) could be used to directly settle the majority of payment and exchange transactions in the finance sector, using on-chain tokens to represent cash, stocks, bonds and more.

This is perfectly workable on a technical level, so what’s the problem? In a word, confidentiality. If multiple institutions are using a shared ledger, then every institution sees every transaction on that ledger, even if they don’t immediately know the real-world identities of the parties involved. This turns out to be a huge issue, both in terms of regulation and the commercial realities of inter-bank competition. While various strategies are available or in development for mitigating this problem, none can match the simplicity and efficiency of a centralized database managed by a trusted intermediary, which maintains full control over who can see what. For now at least, it seems that large financial institutions prefer to keep most transactions hidden in these intermediary databases, despite the costs involved.

I base this conclusion not only on our own experience, but also on the direction taken by several prominent startups whose initial goal was to develop shared ledgers for banks. For example, both R3CEV and Digital Asset are now working on “contract description languages”, in Corda and DAML respectively (earlier examples include MLFi and Ricardian Contracts). These languages allow the conditions of a complex financial contract to be represented formally and unambiguously in a computer readable format, while avoiding the shortcomings of Ethereum-style general purpose computation. Instead, the blockchain plays only a supporting role, storing or notarizing the contracts in encrypted form, and performing some basic duplicate detection. The actual contract execution does not take place on the blockchain – rather, it is performed only by the contract’s counterparties, with the likely addition of auditors and regulators.

In the near term, this is probably the best that can be done, but where does it leave the broader ambitions for permissioned blockchains? Are there other applications for which they can form a more significant part of the puzzle?

This question can be approached both theoretically and empirically. Theoretically, by focusing on the key differences between blockchains and traditional databases, and how these inform the set of possible use cases. And in our case, empirically, by categorizing the real-world solutions being built on MultiChain today. Not surprisingly, whether we focus on theory or practice, the same classes of use case arise:

• Lightweight financial systems.
• Provenance tracking.
• Interorganizational record keeping.
• Multiparty aggregation.

Before explaining these in detail, let’s recap the theory. As I’ve discussed before, the two most important differences between blockchains and centralized databases can be characterized as follows:

1. Disintermediation. Blockchains enable multiple parties who do not fully trust each other to safely and directly share a single database without requiring a trusted intermediary.
2. Confidentiality: All participants in a blockchain see all of the transactions taking place. (Even if we use pseudonymous addresses and advanced cryptography to hide some aspects of those transactions, a blockchain will always leak more information than a centralized database.)

In other words, blockchains are ideal for shared databases in which every user is able to read everything, but no single user controls who can write what. By contrast, in traditional databases, a single entity exerts control over all read and write operations, while other users are entirely subject to that entity’s whims. To sum it up in one sentence:

Blockchains represent a trade-off in which disintermediation is gained at the cost of confidentiality.

In examining the four types of use case below, we’ll repeatedly come back to this core trade-off, explaining why, in each case, the benefit of disintermediation outweighs the cost of reduced confidentiality.

Lightweight financial systems

Let’s start with the class of blockchain applications that will be most familiar, in which a group of entities wishes to set up a financial system. Within this system, one or more scarce assets are transacted and exchanged between those entities.

In order for any asset to remain scarce, two related problems must be solved. First, we must ensure that the same unit of the asset cannot be sent to more than one place (a “double spend”). Second, it must be impossible for anyone to create new units of the asset on a whim (“forgery”). Any entity which could do either of these things could steal unlimited value from the system.

A common solution to these problems is physical tokens, such as metal coins or securely printed paper. These tokens trivially solve the problem of double spending, because the rules of physics (literally) prevent one token from being in two places at the same time. The problem of forgery is solved by making the token extremely difficult to manufacture. Still, physical tokens suffer from several shortcomings which can render them impractical:

• As pure bearer assets, physical tokens can be stolen with no trace or recourse.
• They are slow and costly to move in large numbers or over long distances.
• It is tricky and expensive to create physical tokens that cannot be forged.

These shortcomings can be avoided by leaving physical tokens behind, and redefining asset ownership in terms of a ledger managed by a trusted intermediary. In the past, these ledgers were based on paper records, and today they tend to run on regular databases. Either way, the intermediary enacts a transfer of ownership by modifying the ledger’s content, in response to an authenticated request. Unlike settlement with physical tokens, questionable transactions can quickly and easily be reversed.

So what’s the problem with ledgers? In a nutshell, concentration of control. By putting so much power in one place, we create a significant security challenge, in both technical and human terms. If someone external can hack into the database, they can change the ledger at will, stealing others’ funds or destroying its contents completely. Even worse, someone on the inside could corrupt the ledger, and this kind of attack is hard to detect or prove. As a result, wherever we have a centralized ledger, we must invest significant time and money in mechanisms to maintain that ledger’s integrity. And in many cases, we require ongoing verification using batch-based reconciliation between the central ledger and those of each of the transacting parties.

Enter the blockchain (or “shared ledger”). This provides the benefits of ledgers without suffering from the problem of concentration. Instead, each entity runs a “node” holding a copy of the ledger and maintains full control over its own assets, which are protected by private keys. Transactions propagate between nodes in a peer-to-peer fashion, with the blockchain ensuring that consensus is maintained. This architecture leaves no central attack point through which a hacker or insider could corrupt the ledger’s contents. As a result, a digital financial system can be deployed more quickly and cheaply, with the added benefit of automatic reconciliation in real time.

So what’s the downside? As discussed earlier, all participants in a shared ledger see all of the transactions taking place, rendering it unusable in situations where confidentiality is required. Instead, blockchains are suitable for what I call lightweight financial systems, namely those in which the economic stakes or number of participants is relatively low. In these cases, confidentiality tends to be less of an issue – even if the participants pay close attention to what each other are doing, they won’t learn much of value. And it is precisely because the stakes are low that we prefer to avoid the hassle and cost of setting up an intermediary.

Some obvious examples of lightweight financial systems include: crowdfunding, gift cards, loyalty points and local currencies – especially in cases where assets are redeemable in more than one place. But we are also seeing use cases in the mainstream finance sector, such as peer-to-peer trading between asset managers who are not in direct competition. Blockchains are even being tested as internal accounting systems, in large organizations where each department or location must maintain control of its funds. In all these cases, the lower cost and friction of blockchains provides an immediate benefit, while the loss of confidentiality is not a concern.

Provenance tracking

Here’s a second class of use case that we repeatedly hear from MultiChain’s users: tracking the origin and movement of high-value items across a supply chain, such as luxury goods, pharmaceuticals, cosmetics and electronics. And equally, critical items of documentation such as bills of lading or letters of credit. In supply chains stretching across time and distance, all of these items suffer from counterfeiting and theft.

The problem can be addressed using blockchains in the following way: when the high-value item is created, a corresponding digital token is issued by a trusted entity, which acts to authenticate its point of origin. Then, every time the physical item changes hands, the digital token is moved in parallel, so that the real-world chain of custody is precisely mirrored by a chain of transactions on the blockchain.

If you like, the token is acting as a virtual “certificate of authenticity”, which is far harder to steal or forge than a piece of paper. Upon receiving the digital token, the final recipient of the physical item, whether a bank, distributor, retailer or customer, can verify the chain of custody all the way back to the point of origin. Indeed, in the case of documentation such as bills of lading, we can do away with the physical item altogether.

While all of this makes sense, the astute reader will notice that a regular database, managed (say) by an item’s manufacturer, can accomplish the same task. This database would store a record of the current owner of each item, accepting signed transactions representing each change of ownership, and respond to incoming requests regarding the current state of play.

So why use a blockchain instead? The answer is that, for this type of application, there’s a benefit to distributed trust. No matter where a centralized database is held, there will be people in that place who have the ability (and can be bribed) to corrupt its contents, marking forged or stolen items as legit. By contrast, if provenance is tracked on a blockchain belonging collectively to a supply chain’s participants, no individual entity or small group of entities can corrupt the chain of custody, and end users can have more confidence in the answers they receive. As a bonus, different tokens (say for some goods and the corresponding bill of lading) can be safely and directly exchanged, with a two-way swap guaranteed at the lowest blockchain level.

What about the problem of confidentiality? The suitability of blockchains for supply chain provenance is a happy result of this application’s simple pattern of transactions. In contrast to financial marketplaces, most tokens move in a single direction, from origin to endpoint, without being repeatedly traded back-and-forth between the blockchain’s participants. If competitors rarely transact with each other (e.g. toy manufacturer to toy manufacturer, or retailer to retailer), they cannot learn each others’ blockchain “addresses” and connect those to real-world identities. Furthermore, the activity can be easily partitioned into multiple ledgers, each representing a different order or type of good.

Interorganizational record keeping

Both of the previous use cases are based on tokenized assets, i.e. on-chain representations of an item of value transferred between participants. However there is a second group of blockchain use cases which is not related to assets. Instead, the chain acts as a mechanism for collectively recording and notarizing any type of data, whose meaning can be financial or otherwise.

One such example is an audit trail of critical communications between two or more organizations, say in the healthcare or legal sectors. No individual organization in the group can be trusted with maintaining this archive of records, because falsified or deleted information would significantly damage the others. Nonetheless it is vital that all agree on the archive’s contents, in order to prevent disputes.

To solve this problem, we need a shared database into which all of the records are written, with each record accompanied by a timestamp and proof of origin. The standard solution would be to create a trusted intermediary, whose role is to collect and store the records centrally. But blockchains offer a different approach, giving the organizations a way to jointly manage this archive, while preventing individual participants (or small groups thereof) from corrupting it.

One of the most enlightening conversations I’ve had in the past two years was with Michael Mainelli of Z/Yen. For 20 years his company has been building systems in which multiple entities collectively manage a shared digital audit trail, using timestamping, digital signatures and a round robin consensus scheme. As he explained the technical details of these systems, it became clear that they are permissioned blockchains in every respect. In other words, there is nothing new about using a blockchain for interorganizational recordkeeping – it’s just that the world has finally become aware of the possibility.

In terms of the actual data stored on the blockchain, there are three popular options:

• Unencrypted data. This can be read by every participant in the blockchain, providing full collective transparency and immediate resolution in the case of a dispute.
• Encrypted data. This can only be read by participants with the appropriate decryption key. In the event of a dispute, anyone can reveal this key to a trusted authority such as a court, and use the blockchain to prove that the original data was added by a certain party at a certain point in time.
• Hashed data. A “hash” acts as a compact digital fingerprint, representing a commitment to a particular piece of data while keeping that data hidden. Given some data, any party can easily confirm if it matches a given hash, but inferring data from its hash is computationally impossible. Only the hash is placed on the blockchain, with the original data stored off-chain by interested parties, who can reveal it in case of a dispute.

As mentioned earlier, R3CEV’s Corda product has adopted this third approach, storing hashes on a blockchain to notarize contracts between counterparties, without revealing their contents. This method can be used both for computer-readable contract descriptions, as well as PDF files containing paper documentation.

Naturally, confidentiality is not an issue for interorganizational record keeping, because the entire purpose is to create a shared archive that all the participants can see (even if some data is encrypted or hashed). Indeed in some cases a blockchain can help manage access to confidential off-chain data, by providing an immutable record of digitally signed access requests. Either way, the straightforward benefit of disintermediation is that no additional entity must be created and trusted to maintain this record.

Multiparty aggregation

Technically speaking, this final class of use case is similar to the previous one, in that multiple parties are writing data to a collectively managed record. However in this case the motivation is different – to overcome the infrastructural difficulty of combining information from a large number of separate sources.

Imagine two banks with internal databases of customer identity verifications. At some point they notice that they share a lot of customers, so they enter a reciprocal sharing arrangement in which they exchange verification data to avoid duplicated work. Technically, the agreement is implemented using standard master–slave data replication, in which each bank maintains a live read-only copy of the other’s database, and runs queries in parallel against its own database and the replica. So far, so good.

Now imagine these two banks invite three others to participate in this circle of sharing. Each of the 5 banks runs its own master database, along with 4 read-only replicas of the others. With 5 masters and 20 replicas, we have 25 database instances in total. While doable, this consumes noticeable time and resources in each bank’s IT department.

Fast forward to the point where 20 banks are sharing information in this way, and we’re looking at 400 database instances in total. For 100 banks, we reach 10,000 instances. In general, if every party is sharing information with every other, the total number of database instances grows with the square of the number of participants. At some point in this process, the system is bound to break down.

So what’s the solution? One obvious option is for all of the banks to submit their data to a trusted intermediary, whose job is to aggregate that data in a single master database. Each bank could then query this database remotely, or run a local read-only replica within its own four walls. While there’s nothing wrong with this approach, blockchains offer a cheaper alternative, in which the shared database is run directly by the banks which use it. Blockchains also bring the added benefit of redundancy and failover for the system as a whole.

It’s important to clarify that a blockchain is not acting just as a distributed database like Cassandra or RethinkDB. Unlike these systems, each blockchain node enforces a set of rules which prevent one participant from modifying or deleting the data added by another. Indeed, there still appears to be some confusion about this – one recently released blockchain platform can be broken by a single misbehaving node. In any event, a good platform will also make it easy to manage networks with thousands of nodes, joining and leaving at will, if granted the appropriate permissions.

Although I’m a little skeptical of the oft-cited connection between blockchains and the Internet of Things, I think this might be where a strong such synergy lies. Of course, each “thing” would be too small to store a full copy of the blockchain locally. Rather, it would transmit data-bearing transactions to a distributed network of blockchain nodes, who would collate it all together for further retrieval and analysis.

Conclusion: Blockchains in Finance

I started this piece by questioning the initial use case envisioned for blockchains in the finance sector, namely the bulk settlement of payment and exchange transactions. While I believe this conclusion is becoming common wisdom (with one notable exception), it does not mean that blockchains have no other applications in this industry. In fact, for each of the four classes of use case outlined above, we see clear applications for banks and other financial institutions. Respectively, these are: small trading circles, provenance for trade finance, bilateral contract notarization and the aggregation of AML/KYC data.

The key to understand is that, architecturally, our four classes of use case are not specific to finance, and are equally relevant to other sectors such as insurance, healthcare, distribution, manufacturing and IT. Indeed, private blockchains should be considered for any situation in which two or more organizations need a shared view of reality, and that view does not originate from a single source. In these cases, blockchains offer an alternative to the need for a trusted intermediary, leading to significant savings in hassle and cost.

Please post any comments on LinkedIn.

As the developers of a popular blockchain platform, we sometimes get asked whether Ethereum-like smart contracts are on the MultiChain roadmap. The answer I always give is: no, or at least not yet.

But in the hype-filled world of blockchains, smart contracts are all the rage, so why ever not? Well, the problem is, while we now know of three strong use cases for permissioned Bitcoin-style blockchains (provenance, inter-company records and lightweight finance), we’re yet to find the equivalent for Ethereum-style smart contracts.

It’s not that people lack ideas of what they want smart contracts to do. Rather, it’s that so many of these ideas are simply impossible. You see, when smart people hear the term “smart contracts”, their imaginations tend to run wild. They conjure up dreams of autonomous intelligent software, going off into the world, taking its data along for the ride.

Unfortunately the reality of smart contracts is far more mundane than all that:

A smart contract is a piece of code which is stored on an blockchain, triggered by blockchain transactions, and which reads and writes data in that blockchain’s database.

That’s it. Really. A smart contract is just a fancy name for code which runs on a blockchain, and interacts with that blockchain’s state. And what is code? It’s Pascal, it’s Python, it’s PHP. It’s Java, it’s Fortran, it’s C++. If we’re talking databases, it’s stored procedures written in an extension of SQL. All of these languages are fundamentally equivalent, solving the same sorts of problems in the same sorts of ways. Of course, each has its strengths and weaknesses – you’d be crazy to build a website in C or compress HD video in Ruby. But in principle at least, you could if you wanted to. You’d just pay a heavy price in terms of convenience, performance, and quite probably your hair.

The problem with smart contracts isn’t just that people’s expectations are overblown. It’s that these expectations are leading many to spend time and money on ideas that cannot possibly be implemented. It seems large companies have sufficient resources to travel a lengthy path – from the moment when senior management encounters a new technology, to when that technology’s advantages and limitations are truly understood. Perhaps our own experience can help shorten this time.

Over the past nine months, we’ve been pitched many smart contract use cases, and have found ourselves responding, time and again, that they simply cannot be done. As a result, we’ve identified the three smart contract misconceptions that are most commonly held. These ideas aren’t wrong because the technology is immature, or the tools are not yet available. Rather, they misunderstand the fundamental properties of code which lives in a database and runs in a decentralized way.

Contacting external services

Often, the first use case proposed is a smart contract which changes its behavior in response to some external event. For example, an agricultural insurance policy which pays out conditionally based on the quantity of rainfall in a given month. The imagined process goes something like this: the smart contract waits until the predetermined time, retrieves the weather report from an external service, and behaves appropriately based on the data received.

This all sounds simple enough, but it’s also impossible. Why? Because a blockchain is a consensus-based system, meaning that it only works if every node reaches an identical state after processing every transaction and block. Everything that takes place on a blockchain must be completely deterministic, with no possible way for differences to creep in. The moment that two honest nodes disagree about the chain’s state, the entire system becomes worthless.

Now recall that smart contracts are executed independently by every node on a chain. Therefore, if a smart contract retrieves some information from an external source, this retrieval is performed repeatedly and separately by each node. But because this source is outside of the blockchain, there is no guarantee that every node will receive the same answer. Perhaps the source will change its response in the time between requests from different nodes, or perhaps it will become temporarily unavailable. Either way, consensus is broken and the entire blockchain dies.

So what’s the workaround? Actually, it’s rather simple. Instead of a smart contract initiating the retrieval of external data, one or more trusted parties (“oracles”) creates a transaction which embeds that data in the chain. Every node will have an identical copy of this data, so it can be safely used in a smart contract computation. In other words, an oracle pushes the data onto the blockchain rather than a smart contract pulling it in.

When it comes to smart contracts causing events in the outside world, a similar problem appears. For example, many like the idea of a smart contract which calls a bank’s API in order to transfer money. But if every node is independently executing the code in the chain, who is responsible for calling this API? If the answer is just one node, what happens if that particular node malfunctions, deliberately or not? And if the answer is every node, can we trust every node with that API’s password? And do we really want the API called hundreds of times? Even worse, if the smart contract needs to know whether the API call was successful, we’re right back to the problem of depending on external data.

As before, a simple workaround is available. Instead of the smart contract calling an external API, we use a trusted service which monitors the blockchain’s state and performs certain actions in response. For example, a bank could proactively watch a blockchain, and perform money transfers which mirror the on-chain transactions. This presents no risk to the blockchain’s consensus because the chain plays an entirely passive role.

Looking at these two workarounds, we can make some observations. First, they both require a trusted entity to manage the interactions between the blockchain and the outside world. While this is technically possible, it undermines the goal of a decentralized system. Second, the mechanisms used in these workarounds are straightforward examples of reading and writing a database. An oracle which provides external information is simply writing that information into the chain. And a service which mirrors the blockchain’s state in the real world is doing nothing more than reading from that chain. In other words, any interaction between a blockchain and the outside world is restricted to regular database operations. We’ll talk more about this fact later on.

Enforcing on-chain payments

Here’s another proposal that we tend to hear a lot: using a smart contract to automate the payment of coupons for a so-called “smart bond”. The idea is for the smart contract code to automatically initiate the payments at the appropriate times, avoiding manual processes and guaranteeing that the issuer cannot default.

Of course, in order for this to work, the funds used to make the payments must live inside the blockchain as well, otherwise a smart contract could not possibly guarantee their payment. Now recall that a blockchain is just a database, in this case a financial ledger containing the issued bond and some cash. So when we talk about coupon payments, what we’re actually talking about are database operations which take place automatically at an agreed time.

While this automation is technically feasible, it suffers from a financial difficulty. If the funds used for coupon payments are controlled by the bond’s smart contract, then those payments can indeed be guaranteed. But this also means those funds cannot be used by the bond issuer for anything else. And if those funds aren’t under the control of the smart contract, then there is no way in which payment can be guaranteed.

In other words, a smart bond is either pointless for the issuer, or pointless for the investor. And if you think about it, this is a completely obvious outcome. From an investor’s perspective, the whole point of a bond is its attractive rate of return, at the cost of some risk of default. And for the issuer, a bond’s purpose is to raise funds for a productive but somewhat risky activity, such as building a new factory. There is no way for the bond issuer to make use of the funds raised, while simultaneously guaranteeing that the investor will be repaid. It should not come as a surprise that the connection between risk and return is not a problem that blockchains can solve.

Hiding confidential data

As I’ve written about previously, the biggest challenge in deploying blockchains is the radical transparency which they provide. For example, if ten banks set up a blockchain together, and two conduct a bilateral transaction, this will be immediately visible to the other eight. While there are various strategies for mitigating this problem, none beat the simplicity and efficiency of a centralized database, in which a trusted administrator has full control over who can see what.

Some people think that smart contracts can solve this problem. They start with the fact that each smart contract contains its own miniature database, over which it has full control. All read and write operations on this database are mediated by the contract’s code, making it impossible for one contract to read another’s data directly. (This tight coupling between data and code is called encapsulation, and is the foundation of the popular object-oriented programming paradigm.)

So if one smart contract can’t access another’s data, have we solved the problem of blockchain confidentiality? Does it make sense to talk of hiding information in a smart contract? Unfortunately, the answer is no. Because even if one smart contract can’t read another’s data, that data is still stored on every single node in the chain. For each blockchain participant, it’s in the memory or disk of a system which that participant completely controls. And there’s nothing to stop them reading the information from their own system, if and when they choose to do so.

Hiding data in a smart contract is about as secure as hiding it in the HTML code of a web page. Sure, regular web users won’t see it, because it’s not displayed in their browser window. But all it takes is for a web browser to add a ‘View Source’ function (as they all have), and the hidden information becomes universally visible. Similarly, for data hidden in smart contracts, all it takes is for someone to modify their blockchain software to display the contract’s full state, and all semblance of secrecy is lost. A half-decent programmer could do that in an hour or so.

What smart contracts are for

With so many things that smart contracts cannot do, one might ask what they’re actually for. But in order to answer this question, we need to go back to the fundamentals of blockchains themselves. To recap, a blockchain enables a database to be directly and safely shared by entities who do not trust each other, without requiring a central administrator. Blockchains enable data disintermediation, and this can lead to significant savings in complexity and cost.

Any database is modified via “transactions”, which contain a set of changes to that database which must succeed or fail as a whole. For example, in a financial ledger, a payment from Alice to Bob is represented by a transaction that (a) checks if Alice has sufficient funds, (b) deducts a quantity from Alice’s account, and (c) adds the same quantity to Bob’s.

In a regular centralized database, these transactions are created by a single trusted authority. By contrast, in a blockchain-driven shared database, transactions can be created by any of that blockchain’s users. And since these users do not fully trust each other, the database has to contain rules which restrict the transactions performed. For example, in a peer-to-peer financial ledger, each transaction must preserve the total quantity of funds, otherwise participants could freely give themselves as much money as they liked.

One can imagine various ways of expressing these rules, but for now there are two dominant paradigms, inspired by Bitcoin and Ethereum respectively. The Bitcoin method, which we might call “transaction constraints”, evaluates each transaction in terms of: (a) the database entries deleted by that transaction, and (b) the entries created. In a financial ledger, the rule states that the total quantity of funds in the deleted entries has to match the total in those created. (We consider the modification of an existing entry to be equivalent to deleting that entry and creating a new one in its place.)

The second paradigm, which comes from Ethereum, is smart contracts. This states that all modifications to a contract’s data must be performed by its code. (In the context of traditional databases, we can think of this as an enforced stored procedure.) To modify a contract’s data, blockchain users send requests to its code, which determines whether and how to fulfill those requests. As in this example, the smart contract for a financial ledger performs the same three tasks as the administrator of a centralized database: checking for sufficient funds, deducting from one account, and adding to another.

Both of these paradigms are effective, and each has its advantages and disadvantages, as I’ve discussed in depth previously. To summarize, Bitcoin-style transaction constraints provide superior concurrency and performance, while Ethereum-style smart contracts offer greater flexibility. So to return to the question of what smart contracts are for:

Smart contracts are for blockchain use cases which can’t be implemented with transaction constraints.

Given this criterion for using smart contracts, I’m yet to see a strong use case for permissioned blockchains which qualifies. All the compelling blockchain applications I know can be implemented with Bitcoin-style transactions, which can handle permissioning and general data storage, as well as asset creation, transfer, escrow, exchange and destruction. Nonetheless, new use cases are still appearing, and I wouldn’t be surprised if some do require the power of smart contracts. Or, at the very least, an extension of the Bitcoin paradigm.

Whatever the answer turns out to be, the key to remember is that smart contracts are simply one method for restricting the transactions performed in a database. This is undoubtedly a useful thing, and is essential to making that database safe for sharing. But smart contracts cannot do anything else, and they certainly cannot escape the boundaries of the database in which they reside.

Please post any comments at LinkedIn.

If you’ve been reading my previous posts, you will know by now that blockchains are simply a new type of database. That is, a database which can be directly shared, in a write sense, by a group of non-trusting parties, without requiring a central administrator. This contrasts with traditional (SQL or NoSQL) databases that are controlled by a single entity, even if some kind of distributed architecture is used within its walls.

I recently gave a talk about blockchains from the perspective of information security, in which I concluded that blockchains are more secure than regular databases in some ways, and less secure in others. Considering the leading role that centralized databases play in today’s technology stack, this got me thinking more broadly about the trade-offs between these two technologies. Indeed, whenever someone asks me if MultiChain can be used for a particular purpose, my first response is always: “Could you do that with a regular database?” In more cases than you might think, the answer is yes, for the following simple reason:

If trust and robustness aren’t an issue, there’s nothing a blockchain can do that a regular database cannot.

This is a key point on which there is so much misunderstanding. In terms of the types of data that can be stored, and the transactions that can be performed on that data, blockchains don’t do anything new. And just to be clear, this observation extends to “smart contracts” as well, despite their sexy name and image. A smart contract is nothing more than a piece of computer code which runs on every node in a blockchain – a decades-old technology called stored procedures does the same for centralized databases. (You also cannot use a blockchain if this code needs to initiate interactions with the outside world.)

The truth about blockchains is that, while they have some advantages, they also have their downsides. In other words, like most technology decisions, the choice between a blockchain and a regular database comes down to a series of trade-offs. If you’re blinded by the hype and deafened by the noise, you’re unlikely to make that choice objectively. So I hope the following guide might help.

Disintermediation: advantage blockchains

The core value of a blockchain is enabling a database to be directly shared across boundaries of trust, without requiring a central administrator. This is possible because blockchain transactions contain their own proof of validity and their own proof of authorization, instead of requiring some centralized application logic to enforce those constraints. Transactions can therefore be verified and processed independently by multiple “nodes”, with the blockchain acting as a consensus mechanism to ensure those nodes stay in sync.

Why is there value in this disintermediation? Because even though a database is just bits and bytes, it is also a tangible thing. The contents of a database are stored in the memory and disk of a particular computer system, and anybody with sufficient access to that system can destroy or corrupt the data within. As a result, the moment you entrust your data to a regular database, you also become dependent on the human organization in which that database resides.

Now, the world is filled with organizations which have earned this trust – governments and banks (mostly), universities, trade associations, and even private companies like Google and Facebook. In most cases, especially in the developed world, these work extremely well. I believe my vote has always been counted, no bank has ever stolen my money, and I’m yet to find a way to pay for better grades. So what’s the problem? If an organization controls an important database, it also needs a bunch of people and processes in place to prevent that database being tampered with. People need hiring, processes need to be designed, and all this takes a great deal of time and money.

So blockchains offer a way to replace these organizations with a distributed database, locked down by clever cryptography. Like so much that has come before, they leverage the ever-increasing capacity of computer systems to provide a new way of replacing humans with code. And once it’s been written and debugged, code tends to be an awful lot cheaper.

Confidentiality: advantage centralized databases

As I mentioned, every node in a blockchain independently verifies and processes every transaction. A node can do this because it has full visibility into: (a) the database’s current state, (b) the modification requested by a transaction, and (c) a digital signature which proves the transaction’s origin. This is undoubtedly a clever new way to architect a database, and it really works. So where’s the catch? For many applications, especially financial, the full transparency enjoyed by every node is an absolute deal-killer.

How do systems built on regular databases avoid this problem? Just like blockchains, they restrict the transactions that particular users can perform, but these restrictions are imposed in one central location. As a result, the full database contents need only be visible at that location, rather than in multiple nodes. Requests to read data also go through this central authority, which can accept or reject those requests as it sees fit. In other words, if a regular database is read-controlled and write-controlled, a blockchain can be write-controlled only.

To be fair, many strategies are available for mitigating this problem. These range from simple ideas like transacting under multiple blockchain addresses, to advanced cryptographic techniques such as confidential transactions and zero-knowledge proofs (now being developed). Nonetheless, the more information you want to hide on a blockchain, the heavier a computational burden you pay to generate and verify transactions. And no matter how these techniques develop, they will never beat the simple and straightforward method of hiding data completely.

Robustness: advantage blockchains

A second benefit of blockchain-powered databases is extreme fault tolerance, which stems from their built-in redundancy. Every node processes every transaction, so no individual node is crucial to the database as a whole. Similarly, nodes connect to each other in a dense peer-to-peer fashion, so many communication links can fail before things grind to a halt. The blockchain ensures that nodes which went down can always catch up on transactions they missed.

So while it’s true that regular databases offer many techniques for replication, blockchains take this to a whole new level. For a start, no configuration is required – simply connect some blockchain nodes together, and they automatically keep themselves in sync. In addition, nodes can be freely added or removed from a network, without any preparation or consequences. Lastly, external users can send their transactions to any node, or to multiple nodes simultaneously, and these transactions propagate automatically and seamlessly to everyone else.

This robustness transforms the economics of database availability. With regular databases, high availability is achieved through a combination of expensive infrastructure and disaster recovery. A primary database runs on high-end hardware which is monitored closely for problems, with transactions replicated to a backup system in a different physical location. If the primary database fails (e.g. due to a power cut or catastrophic hardware failure), activity is automatically moved over to the backup, which becomes the new primary. Once the failed system is fixed, it’s lined up to act as the new backup if and when necessary. While all this is doable, it’s expensive and notoriously difficult to get right.

Instead, what if we had 10 blockchain nodes running in different parts of the world, all on commodity hardware? These nodes would be densely connected to each other, sharing transactions on a peer-to-peer basis and using a blockchain to ensure consensus. End users generating the transactions connect to (say) 5 of these nodes, so it doesn’t matter if a few communication links go down. And if one or two nodes fail completely on any given day, nobody feels a thing, because there are still more than enough copies to go round. As it happens, this combination of low cost systems and high redundancy is exactly how Google built its search engine so cheaply. Blockchains can do the same thing for databases.

Performance: advantage centralized databases

Blockchains will always be slower than centralized databases. It’s not just that today’s blockchains are slow because the technology is new and unoptimized, but it’s a result of the nature of blockchains themselves. You see, when processing transactions, a blockchain has to do all the same things as a regular database, but it carries three additional burdens:

1. Signature verification. Every blockchain transaction must be digitally signed using a public-private cryptography scheme such as ECDSA. This is necessary because transactions propagate between nodes in a peer-to-peer fashion, so their source cannot otherwise be proven. The generation and verification of these signatures is computationally complex, and constitutes the primary bottleneck in products like ours. By contrast, in centralized databases, once a connection has been established, there is no need to individually verify every request that comes over it.
2. Consensus mechanisms. In a distributed database such as a blockchain, effort must be expended in ensuring that nodes in the network reach consensus. Depending on the consensus mechanism used, this might involve significant back-and-forth communication and/or dealing with forks and their consequent rollbacks. While it’s true that centralized databases must also contend with conflicting and aborted transactions, these are far less likely where transactions are queued and processed in a single location.
3. Redundancy. This isn’t about the performance of an individual node, but the total amount of computation that a blockchain requires. Whereas centralized databases process transactions once (or twice), in a blockchain they must be processed independently by every node in the network. So lots more work is being done for the same end result.

The bottom line

Naturally there are other ways in which blockchains and regular databases can be compared. We could talk about codebase maturity, developer attractiveness, ecosystem breadth and more. But none of these issues are inherent to the technology itself. So when it comes to a long-term decision on using a blockchain, the question to ask is this: What’s more important for my use case? Disintermediation and robustness? Or confidentiality and performance?

When examined in this simple light, many of the use cases currently under discussion do not make sense. The biggest problem tends to be confidentiality. The participants in a fiercely competitive marketplace will naturally prefer the privacy of a centralized database, rather than reveal their activities to each other. This is especially true if a trusted central party already exists and can provide the neutral territory in which that database can reside. Even though there may be some cost associated with this central provider, this is more than justified by the value of the privacy retained. The only motivation for a shift to blockchains would be aggressive new regulation.

Nonetheless blockchains do have strong use cases, where disintermediation and robustness are more important than confidentiality and performance. I’ll write more about these in a subsequent post, but the most promising areas we’ve seen so far are: (a) inter-company audit trails, (b) provenance tracking, and (c) lightweight financial systems. In all three cases, we’ve found people building on MultiChain with a clear view to deployment, rather than just curiosity and experimentation. So if you’re looking for ways in which blockchains can add genuine value to your business, they might be a good place to start.

Please post any comments on LinkedIn.

As a change from blog posts about blockchains in general, I’d like to provide an update on MultiChain, both in terms of recent enhancements and our roadmap for 2016.

First, I’d like to thank the many thousands of you who have downloaded and built on MultiChain, asked questions and sent us feedback. In the eight months since the first public release, our stats have shown consistent organic growth in traffic and downloads, and I hope this means we’re hitting the spot. Indeed, without naming names, we know that MultiChain has been successfully used for long-running blockchain pilots in some of the largest banks, consulting firms, financial technology and IT companies on the planet.

One question we’re often asked is why MultiChain has been in “alpha” for so long. The simple answer is that we’ve been bombarded with feature requests, most of which made sense to us, so we’ve been focused on adding these enhancements rather than bringing the product to beta. Having said that, you should find MultiChain to be very stable for alpha software, and we’ve tested it thoroughly under extreme loads.

I also want to explain how we’re positioning MultiChain in the broadening space of blockchain platforms. In the last six months, many competing products have been announced, (quasi-)consortia have been formed, companies have raised tens of millions of dollars, and just occasionally we’ve seen some real software releases. Of course, competition is natural and inevitable and we look forward to watching these other platforms develop. No doubt we’ll be borrowing their best ideas, and we assume they’ll return the compliment.

So where does MultiChain fit in with all this noise? In a nutshell, it’s focused on product and practicality:

• Stability. By forking from Bitcoin Core, the reference implementation for the bitcoin network, MultiChain builds on the years of hard-earned stability and security which come from stewarding billions of dollars in cryptocurrency value on the open Internet. To be clear, the Bitcoin Core codebase does have architectural limitations, and we may eventually have to move away from it. Nonetheless, for current user requirements, the cost of doing so would significantly outweigh the benefits.
• Ease of use. Many MultiChain users have told us that it’s far easier to use than competing blockchain platforms. I can’t even recall how many times I’ve told someone they can go from zero to their own private blockchain in minutes, and they just haven’t believed me. But it’s really true – just follow the instructions on the download and getting started pages and see for yourself. No dependencies, no compilation, no messing with Docker. Just three self-contained executables and a README file.
• Features. When MultiChain was first released, it had far fewer features than today. No per-address control of assets, no atomic exchange transactions, no easy transaction metadata. So how do we decide what to add? Simple – we listen to our users. Sometimes they know exactly what they want, like follow-on asset issuance, and we’re happy to oblige. Other times they know what they want to achieve, but don’t know how to express that as a feature, and it’s our job to work it out. Either way, MultiChain’s roadmap is driven relentlessly by user feedback, and so it will continue.
• Bitcoin compatibility. If you’re building a blockchain solution, you’ll find the node is just a small part of the picture. You may need mobile or web wallets, key management solutions and a library in some obscure language for decoding, signing and encoding transactions. MultiChain is designed to make all this as simple and fast as possible, by maintaining maximal compatibility with bitcoin, for which a huge amount of information, tools and code is freely available. To prove the point, MultiChain can even be configured as a node on the bitcoin network.

Basically, we aim to delight our users, and firmly believe this is the surest path to commercial success. On that note, I’d like to describe some of the new features added in the last few months.

Follow-on asset issuance (alpha 17)

This request has been around for a while, and is the most upvoted question on the Developer Q&A. So why did it take so long? You can blame us for being purists. You see, in terms of security, there’s no difference between (a) issuing a gazillion units of an asset the first time round and keeping most of them out of circulation, and (b) allowing follow-on issuances of more units of the same asset.

But it turns out that from our users’ perspective, there is quite a difference between the two cases, because it’s not so easy to differentiate units in active circulation from those sitting on the sidelines. So we’re pleased to announce that, in the version released today, when you issue an asset, you can decide whether that asset is open or closed. If it’s open, the original issuing party can create more units as many times as they like.

On the flip side, MultiChain also now provides a canonical ‘burn address’ for every chain. This address is full of X’s and so was obviously created without a corresponding private key (doing so would take an interminable amount of time). Any asset units sent to this address can therefore never be spent and are destroyed in a transparent fashion. Note that for your safety, the burn address has to be explicitly granted receive permissions before it can be used.

API calls: issue, issuefrom, issuemore, issuemorefrom, listassets, getinfo response’s burnaddress field.

MultiChain Explorer

Together with alpha 17, we’re releasing the first beta of the free and open source MultiChain Explorer. This provides an intuitive web-based view of the global state of a MultiChain blockchain, similar to the blockchain explorers that bitcoin users know and love. It lets you quickly and comfortably view transactions, blocks, assets and addresses, as well as the connections between them, all from the comfort of your favorite web browser.

The MultiChain Explorer was forked from the popular Abe project, written in Python and powered by SQLite. It connects to the API of a local MultiChain node and includes a self-contained web server so there are no additional dependencies. We hope you enjoy this tool and welcome your feedback to help us make it even better.

Interactive command mode (alpha 16)

As a fork of Bitcoin Core, MultiChain inherited the bitcoin-cli tool, which we appropriately renamed to multichain-cli of course. This tool provides a convenient command-line interface for MultiChain’s JSON-RPC API, allowing API calls to be sent from the system command line, with their responses displayed in the terminal. Behind the scenes it reads the API credentials from the appropriate chain’s configuration file, builds the JSON-RPC request and decodes its response.

As users of MultiChain ourselves, one frustration we had was that multichain-cli had to be run separately for every API request. Apart from the system overhead, this prevents the sort of fluid interaction that SQL databases provide. And so we fixed it. As of alpha 16, if you run multichain-cli [chain-name] with no command, you’re dropped into an interactive mode that lets you repeatedly type commands and see their response. Interactive mode supports standard editing features such as history (up and down arrows), jumping to the start (Ctrl A) or end (Ctrl E) of the line, and moving to the next (Ctrl →) and previous (Ctrl ←) word.

Faster signature verification (alpha 15)

When it comes to performance in bitcoin or MultiChain, the most crucial bottleneck is the verification of the ECDSA signatures on which the blockchain’s security model is built. The original Bitcoin Core software relied on an open source library called OpenSSL for signature generation and verification, which did the job, although it had some issues with malleability, meaning that more than one signature was valid for a given private key and payload.

Recent versions of Bitcoin Core introduced a new library for ECDSA signing and verification, called libsecp256k1. This library, written from scratch by world-class blockchain developers, removes the dependency on OpenSSL, resolves issues with malleability, and performs several times faster. One of the benefits of being derived from Bitcoin Core is that MultiChain can take advantage of these sorts of enhancements, which are extensively peer-reviewed and tested before being deployed in the bitcoin network. And so alpha 15 does exactly that with libsecp256k1.

Activate permission (alpha 14)

When developing the first version of MultiChain, we faced a dilemma in terms of permissioning. On the one hand, we’d have no problem concocting and implementing an extremely powerful permissions model, with multiple layers of administrators, per-asset permissions, and weighted voting schemes. On the other hand, we knew that these would add complexity from a user perspective, and wouldn’t necessarily match user needs. So we decided to start with a simple model, containing just six permission types (connect, send, receive, issue, mine, admin) and some straightforward consensus-based voting for the most important privilege changes. We expected this model to get more complex over time, but driven by user requirements rather than our own theories.

It turns out that, in this case, simple is actually pretty good. But one serious partner we’re working with needed something more. You see, a MultiChain address with admin privileges has the power to control all types of permissions on a blockchain, subject in some cases to consensus with other administrators. But this partner wanted to give an address the power to control others’ connect, send and receive permissions only, for the purposes of onboarding, and have no influence on more crucial processes such as mining and asset issuance. So we added a new ‘activate’ permission which does exactly this. This was also the first example of a partner paying to implement a feature they needed in the product, a win–win if ever there was one.

Wallet transaction APIs (alpha 13)

As a fork of Bitcoin Core, MultiChain inherited some of the bad along with the good. One of the weak spots in Bitcoin Core is the API for retrieving information about the transactions in the local node’s wallet. It offers two choices: (a) the getrawtransaction call which decodes the binary content of transactions, but doesn’t explain how they affected the local wallet, and (b) the gettransaction and listtransactions calls which aim to describe transactions from the wallet’s perspective, but do so in a confusing way, with multiple response elements per transaction. Making things worse, the output from these calls couldn’t be easily extended to work with MultiChain’s implementation of blockchain-issued assets.

So this release introduced a bunch of new APIs for querying a node’s transactions. The output from these calls retains all the useful fields from the ones they supersede. But they also add a bunch of new fields describing how each transaction affected the local wallet’s balance, which addresses it involved, how it modified permissions, and any metadata contained. Following the introduction (in alpha 8) of the ability to isolate the activity of each address in a wallet, the calls come in two versions – one pair which describes transactions from the perspective of the wallet as a whole, and another which describes them from the perspective of an individual wallet address.

API calls: listwallettransactions, getwallettransaction, listaddresstransactions, getaddresstransaction.

Looking ahead to 2016

Those are some of the major enhancements introduced in MultiChain during the last few months. Of course, many smaller features have been added as well, and they’re listed in full in the download’s README file. And our first priority will always be to fix bugs as soon as they appear. Thankfully the issues we’ve seen have never been of a serious architectural nature – the happy result of using Bitcoin Core as a starting point.

In terms of MultiChain itself, after a breakneck release schedule, we’re going to slow down a little. This is because we’re working on something big that will take a few months to finish. I’ll describe this feature in detail in a future blog post, but the basic idea is to provide a simple and efficient immutable recording and timestamping mechanism for any type of information, a kind of digital ‘tape’. Although transaction metadata in MultiChain can already be used for this purpose (in up to 8MB chunks), it’s not particularly convenient for storage or retrieval, and there are scalability issues when dealing with large pieces of data.

What’s motivating this feature? Your feedback, of course, which has taught us that general purpose immutable storage is a very common use case for blockchains. And if we ever see significant demand for “smart contracts” (i.e. on-blockchain computation) in MultiChain, this system can serve as the underlying storage layer, with computations performed per node, when required. As I’ve explained previously, there’s little value in requiring every node in a private blockchain to perform on-chain computations in real time.

And after that? Well no doubt there will be more enhancements to the free product, but we’re also going to start working on a premium version of MultiChain. As luck would have it, over the past 8 months we’ve seen a bunch of common feature requests which share the following characteristics:

• They’re important for real world deployments, but not for initial experimentation.
• They can be implemented on a per-node basis, without affecting a chain’s consensus.
• Real companies doing real projects seem more than happy to pay for them.

These features are related to performance, security, logging and analytics, and we’ll describe them in detail in the fullness of time. For now, I want to emphasize two key things about this premium version. First, it will be a drop-in replacement for the free version, so any code or applications you build on MultiChain today will continue to work unmodified. Second, every node in a blockchain will be able to independently decide whether to upgrade or not, because none of the premium features affect the blockchain’s consensus. This isn’t just us being kind-hearted – it’s crucial if we want MultiChain to continue to grow organically. A new entity will be able to connect and interact with an existing MultiChain network full of premium nodes, without spending a dime.

If you’re interested in discussing the premium version of MultiChain, please email premium@multichain.com or use this form. We’ll be happy to learn about your requirements and see how we can meet them.

One thing I’ve learnt in the past couple of years is that nobody takes software seriously until they can actually see and use it. A month before the first release of MultiChain, I was telling people about the product, and I noticed them politely nodding while obviously thinking “Oh save me, here’s another fast talker with a white paper and no working code.” But as soon as you make a product available, the response changes completely. So if you’re reading about this future premium version with a dose of skepticism, I understand and won’t hold it against you. All I can say is that, so far, MultiChain has a very solid record of delivering on its promises, and we look forward to continuing.

I also want to take this opportunity to thank our team for their outstanding work. Although I’m a serious coder by profession, these days I spend all my time writing content, managing product and talking to customers. I’m incredibly fortunate to know I can trust our developers to craft solid and efficient code, day after day, and I don’t take it for granted for a moment.

And finally, thanks to you for reading, and for being an early user of the MultiChain platform.

When it comes to using blockchains for inter-enterprise coordination, there’s an elephant-sized problem in the room. In my view, nobody’s talking about this issue enough, whether due to denial or the need to keep the hype going. The problem, in a nutshell, is confidentiality.

To recap what I’ve explained previously, a blockchain allows a database to be shared between entities who do not fully trust each other, without requiring a central administrator. Instead, a blockchain-based database is based on a set of “nodes” which are owned by the participating entities. Nodes send transactions to each other in a peer-to-peer fashion, with each node independently verifying each transaction. Groups of transactions are then confirmed in “blocks” created by special nodes called “miners”. These blocks link to form a “blockchain” which acts as a unified transaction log, ensuring that all nodes reach consensus on the database’s state.

At this point, blockchains are a proven technology, both in public cryptocurrencies like bitcoin and their private equivalents. But they still suffer from a fundamental problem. Putting aside advanced cryptography (for now), blockchains reveal the content of every transaction to every participant. Why? Because in order to verify a transaction, every node has to see that transaction. This makes blockchains fundamentally different from centralized databases, in which transactions are only visible to their creators and the database administrator.

So if you’re considering a blockchain for a project, you should bear this simple principle in mind:

Blockchains are for shared databases in which everyone sees what everyone else is doing.

To be clear, seeing what someone is doing doesn’t necessarily mean that you know who is doing it. Blockchains represent identity using meaningless alphanumeric “addresses”, and most participants need not know to whom these belong. Nevertheless, a lot can be learned by analyzing the behavior of an address, and especially from how it transacts with other addresses. In formal terms, this means that blockchains provide pseudonymity rather than anonymity, because identities persist over time. In the case of bitcoin, several companies are already selling services which mine the “transaction graph” to reveal information about the owners of bitcoin addresses.

The bottom line is that blockchains are best suited for shared databases which are write-controlled but read-uncontrolled. Or, to put it more poetically, blockchains are transparency machines.

The economics of mining

Blockchains began with bitcoin – a digital, decentralized and censorship-proof form of money. One of bitcoin’s key design goals was allowing anybody to “mine” a block which confirms transactions, to prevent governments or banks controlling who can pay whom. In theory, open mining sounds democratic, but on its own it leads to dictatorship by stealth. Why? Because on the Internet it’s possible for one entity to use many different identities, a problem known as the Sybil attack. This means that someone could seize control of block mining, deciding unilaterally which transactions get confirmed, without anyone else even knowing it happened.

Bitcoin cleverly resolves this problem through proof of work. Bitcoin mining may be open, but it is also extremely difficult. In order to create a block, a miner must win a global race to solve a pointless and tricky computational problem, which consumes a lot of electricity (and therefore money). These days mining is performed by specially optimized hardware, but this doesn’t make it any cheaper, because the network regularly adjusts the problem’s difficulty to maintain a steady rate of 1 block every 10 minutes. This makes it hard for any single actor to seize control of the chain and, so far at least, the scheme has worked.

In exchange for the hard work and expense, the winning miner receives a reward, currently 25 newly-minted bitcoins per block (to halve during 2016). Miners also receive a little extra from the fees attached to transactions, although for now these play a minor role. And here are some shocking numbers: During 2015, bitcoin miners raked in $375 million in rewards and fees, in exchange for confirming 45 million transactions. That comes out to over $8 per transaction, even ignoring the fact that many of these weren’t genuine transfers of funds.

Who on earth is paying for all this? The answer is: bitcoin investors. For the most part, miners exchange their new bitcoins for regular currencies like dollars and yuan, because they need this money to pay for mining hardware and electricity. And what will happen if the investors stop coming? Well, the bitcoin price will crash, as miners are forced to dump their bitcoins at a significant loss. Indeed, looking at bitcoin’s price history, there have been several periods during which the price drifted gradually and undramatically downwards, because of the constant supply of bitcoins to be sold.

In the meantime, as the first and most prominent blockchain, bitcoin continues to attract an impressive level of incoming investment. Clearly there’s room for just a handful of such high-profile public blockchains, because the economics of open mining leads inevitably to consolidation. Any new blockchain secured by a low quantity of mining power will not be attractive to end users, because of its inherent insecurity. This will keep its currency value low, which will prevent it attracting additional miners. In other words, the virtuous circle underlying bitcoin’s explosive growth will be hard to repeat. In my view, the only likely exceptions will be newcomers such as Ethereum and Dash which offer a step change in terms of functionality. (I’m ignoring so-called merged mining as well as ideas like proof of stake, because they have not yet been proven to work at scale.)

As luck would have it, private blockchains avoid all of this trouble. Instead of open mining, private chains rely on a whitelist of permitted miners, with all blocks signed digitally by their miner of origin. This is combined with some form of distributed consensus scheme which prevents a small group of these miners from monopolizing the process. If you like, it’s democracy for the privileged, rather than democracy for all. Since private blockchains have no need for proof of work to enforce diversity, they also don’t have to incentivize miners with a financial reward. Instead, a private blockchain costs no more to run than a regular replicated database. The reward is simply the immediate and sufficient benefit of being able to make use of the chain.

With the economics of open mining out of the way, a universe of possibilities opens up. One organization can participate in thousands of blockchains, just like it accesses thousands of (internal or external) databases today. And globally there can be millions (or billions) of blockchains, all serving different purposes and sets of users. But if the world will be filled by so many blockchains, it’s safe to assume that each of them is going to be small.

From monolithic to small blockchains

What do I specifically mean by a “small blockchain”? I mean a blockchain whose scope is restricted to a narrow and specific purpose. This is the polar opposite of catch-all public blockchains like bitcoin and Ethereum, or even the permissioned global bank blockchain that some think is in the offing. It is, in fact, rather more like a regular database, but with a different model of sharing and trust.

Of course, there are many ways in which a blockchain’s scope can be restricted, so I’ll focus here on three simple examples: (a) per-order blockchains, (b) bilateral blockchains, and (c) notarization by hash.

Per-order blockchains

Let’s imagine a blockchain designed to manage the lifecycle of a single container of branded goods, manufactured in China and sold in the US. There can be a bewildering number of parties involved in this process, such as a retailer, agent, distributor, importer, shipping company, manufacturer, licensor and designer, as well as multiple subcontractors, shipping ports, banks, customs agencies and tax authorities. A large amount of information has to flow back and forth between these parties, leading to bureaucratic delays, errors and expenses. In theory, all this could be streamlined using a centralized database, but the question is: who will run it? Considering the gap in geography, culture and legal systems, it may not be easy to find someone that all the parties can trust.

Now, much has already been said about how blockchains can simplify coordination in supply chains. A blockchain can be used to record important documentation, digitally signed as appropriate, as well as enable the transfer of digital equivalents of key assets such as a bill of lading or letter of credit. However, putting all of this data on a monolithic blockchain can leak confidential information. For example, if two competing manufacturers use the same shipping company and bank, they could learn a lot about each other’s activities from transactions which involve those counterparties but are not their own.

One solution is to keep all the information relating to a single order in a blockchain which is dedicated just to that order. In this case, the confidentiality problem is much diminished. For example, two competing manufacturers will never participate in the same chain. At the beginning of the process, a new private blockchain can be set up, and connected to by all the participants. This blockchain makes the state of the order visible to all users in real time. And once it is safely delivered and paid for, the order’s blockchain can be decommissioned and archived away, only to be reopened in case of a dispute.

One issue with per-order blockchains is identity management. When using a blockchain for inter-enterprise coordination, each participant needs to know the real-world identity behind many of the other addresses used on the chain. Obtaining this mapping securely is a potentially inconvenient process, involving either direct exchange of information (by fax?) or a trusted administrator who provides it. But the good news is that there’s no need for this process to take place every time a new blockchain is set up. Instead, participants can have the same address on all the chains that they use. Alternatively, a separate long-running blockchain could be used purely for identity management, allowing each entity to securely distribute its address for each new chain.

Bilateral blockchains

Now let’s consider a blockchain which is used for the rapid settlement of exchanges of financial assets, such as government-backed currencies. This chain would involve at least three types of participants: (a) the trading parties which are performing the transactions, (b) the custodial bank which holds the currencies and issues on-chain tokens to represent them, (c) regulators and/or auditors who receive a read-only view of the activity taking place.

This is a perfectly natural application of blockchains, and already supported in full by off-the-shelf platforms such as MultiChain (our own). But again, the problem of confidentiality rears it head. If the trading parties are locked in intense competition, they can watch each other in order to infer:

• How much of each currency is held by each trader.
• Which currencies they actively trade in, with what frequency and quantity.
• Who else they trade with on the blockchain, and at what prices.

Even if we assume that the parties are not told who is using which address (or multiple addresses), it won’t take them long to work it out. Fierce competitors in a marketplace tend to know a lot about each other, and this prior knowledge can be correlated with patterns of blockchain transactions in order to learn more. For many financial use cases, the risk of this leakage is simply a deal-killer, because the efficiency gained is outweighed by the confidentiality lost.

Nonetheless blockchains can still provide some assistance in this scenario – namely, to record the flow of transactions and messages across each bilateral communications channel between trader and custodian. By combining signed transactions with signed commitments, the blockchain provides realtime reconciliation across this channel, ensuring there is no way in which the parties can differ over what was done and when. In addition, regulators and/or auditors could be granted read-only access to many or all of these pairwise blockchains, giving them a comprehensive view of the activity in a particular marketplace, without needing to explicitly request data from its participants.

Notarization by hash

As I hope is now clear, blockchains can be used to digitally sign, store and timestamp any important data, including text, documents, images and database entries. So long as the blockchain’s miners do not collude maliciously, the chain becomes an irreversible and incontrovertible audit trail for all of the information within. For example, all of the emails sent between the members of a group could be recorded on a blockchain, with each message signed by both the sender and receiver.

But once again we come up against the problem of confidentiality. In many cases, the two parties to a correspondence will not want its content to be visible to anyone else. Their sole purpose in using the blockchain is to prevent future disputes, so that they cannot disagree over what was said, by whom and when.

In this case the solution is simple. Instead of storing the full text of the messages within the blockchain, a “hash” (or digital fingerprint) of their content is embedded instead. A hash is based on a one-way function, which means a function whose output is easy to compute for a given input, but which is practically impossible to reverse. By collaboratively embedding and signing the hash of a message’s content in a blockchain, the parties are able to “lock down” that content in an auditable way, without revealing it to the other participants.

In parallel to embedding this hash, both correspondents store the full message content on their own systems. If a dispute arises in future, either party can reveal this content to an independent party, who can calculate its hash and confirm that this matches the hash on the chain. If so, there is no denying the correspondence that took place. Indeed, this same principle is already applied by many services to notarize documents on the public bitcoin blockchain. Doing so on private blockchains gives greater scalability, lower transaction costs, and hides the entire process from the outside world.

Zero knowledge proofs

So there we have it – three examples of how blockchains can be used, given the limitations posed by radical transparency. But before I finish, it’s important to mention some emerging cryptographic techniques. Sporting names like homomorphic encryption and zero-knowledge proofs, these promise to untie the gordian confidentiality knot. In the context of a blockchain, they offer a seemingly impossible separation of visibility and verification. A partially encrypted transaction can be embedded in a blockchain, along with a proof of its validity, without revealing the transaction’s contents. Every participant can then verify the proof, while still only seeing the transaction in encrypted form. And the unencrypted version is revealed on a need-to-know basis, presumably only to the transaction’s recipient.

Although there has been some real progress in this space, these technologies are yet to mature. It’s still not computationally feasible to generate and verify a proof regarding the validity of a blockchain transaction while keeping its contents fully private. Nonetheless let’s assume that, at some point in the future, this technical problem is solved. I still think we might have a psychological one. You see, in the current way of doing things, a CIO knows that her employer’s confidential data is protected by physical and organizational barriers. Data can only escape if someone is grossly negligent or deliberately commits a crime. But when it comes to advanced cryptography, the picture is rather different, with the CIO relying on advanced mathematics and the soundness of random number generators.

So even when the technology problem is solved, I think it could still take a long time to overcome the emotional barrier. In the meantime, where does this leave us? With the stark assumption that every participant in a blockchain sees everything else that is going on. While this assumption might restrict the sphere of feasible applications, it will also prevent time being wasted on projects that will never be moved to production. And as others have said before me, 2016 is the year to transition from thinking and talking about blockchains, into building some real applications.

Please post any comments on LinkedIn.

Blockchains are overhyped. There, I said it. From Sibos to Money20/20 to cover stories of The Economist and Euromoney, everyone seems to be climbing aboard the blockchain wagon. And no doubt like others in the space, we’re seeing a rapidly increasing number of companies building proofs of concept on our platform and/or asking for our help.

As a young startup, you’d think we’d be over the moon. Surely now is the time to raise a ton of money and build that high performance next generation blockchain platform we’ve already designed. What on earth are we waiting for?

I’ll tell you what. We’re waiting to gain a clearer understanding of where blockchains genuinely add value in enterprise IT. You see, a large proportion of these incoming projects have nothing to do with blockchains at all. Here’s how it plays out. Big company hears that blockchains are the next big thing. Big company finds some people internally who are interested in the subject. Big company gives them a budget and tells them to go do something blockchainy. Soon enough they come knocking on our door, waving dollar bills, asking us to help them think up a use case. Say what now?

As for those who do have a project in mind, what’s the problem? In many cases, the project can be implemented perfectly well using a regular relational database. You know, big iron behemoths like Oracle and SQL Server, or for the more open-minded, MySQL and Postgres. So let me start by setting things straight:

If your requirements are fulfilled by today’s relational databases, you’d be insane to use a blockchain.

Why? Because products like Oracle and MySQL have decades of development behind them. They’ve been deployed on millions of servers running trillions of queries. They contain some of the most thoroughly tested, debugged and optimized code on the planet, processing thousands of transactions per second without breaking a sweat.

And what about blockchains? Well, our product was one of the first to market, and has been available for exactly 5 months, with a few thousand downloads. Actually it’s extremely stable, because we built it off Bitcoin Core, the software which powers bitcoin. But even so, this entire product category is still in its diapers.

So am I saying that blockchains are useless? Absolutely not. But before you embark on that shiny blockchain project, you need to have a very clear idea of why you are using a blockchain. There are a bunch of conditions that need to be fulfilled. And if they’re not, you should go back to the drawing board. Maybe you can define the project better. Or maybe you can save everyone a load of time and money, because you don’t need a blockchain at all.

1. The database

Here’s the first rule. Blockchains are a technology for shared databases. So you need to start by knowing why you are using a database, by which I mean a structured repository of information. This can be a traditional relational database, which contains one or more spreadsheet-like tables. Or it can be the trendier NoSQL variety, which works more like a file system or dictionary. (On a theoretical level, NoSQL databases are just a subset of relational databases anyway.)

A ledger for financial assets can be naturally expressed as a database table in which each row represents one asset type owned by one particular entity. Each row has three columns containing: (a) the owner’s identifier such as an account number, (b) an identifier for the asset type such as “USD” or “AAPL”, and (c) the quantity of that asset held by that owner.

Databases are modified via “transactions” which represent a set of changes to the database which must be accepted or rejected as a whole. For example, in the case of an asset ledger, a payment from one user to another is represented by a transaction that deducts the appropriate quantity from one row, and adds it to another.

2. Multiple writers

This one’s easy. Blockchains are a technology for databases with multiple writers. In other words, there needs to be more than one entity which is generating the transactions that modify the database. Do you know who these writers are?

In most cases the writers will also run “nodes” which hold a copy of the database and relay transactions to other nodes in a peer-to-peer fashion. However transactions might also be created by users who are not running a node themselves. Consider for example a payments system which is collectively maintained by a small group of banks but has millions of end users on mobile devices, communicating only with their own bank’s systems.

3. Absence of trust

And now for the third rule. If multiple entities are writing to the database, there also needs to be some degree of mistrust between those entities. In other words, blockchains are a technology for databases with multiple non-trusting writers.

You might think that mistrust only arises between separate organizations, such as the banks trading in a marketplace or the companies involved in a supply chain. But it can also exist within a single large organization, for example between departments or the operations in different countries.

What do I specifically mean by mistrust? I mean that one user is not willing to let another modify database entries which it “owns”. Similarly, when it comes to reading the database’s contents, one user will not accept as gospel the “truth” as reported by another user, because each has different economic or political incentives.

4. Disintermediation

So the problem, as defined so far, is enabling a database with multiple non-trusting writers. And there’s already a well-known solution to this problem: the trusted intermediary. That is, someone who all the writers trust, even if they don’t fully trust each other. Indeed, the world is filled with databases of this nature, such as the ledger of accounts in a bank. Your bank controls the database and ensures that every transaction is valid and authorized by the customer whose funds it moves. No matter how politely you ask, your bank will never let you modify their database directly.

Blockchains remove the need for trusted intermediaries by enabling databases with multiple non-trusting writers to be modified directly. No central gatekeeper is required to verify transactions and authenticate their source. Instead, the definition of a transaction is extended to include a proof of authorization and a proof of validity. Transactions can therefore be independently verified and processed by every node which maintains a copy of the database.

But the question you need to ask is: Do you want or need this disintermediation? Given your use case, is there anything wrong with having a central party who maintains an authoritative database and acts as the transaction gatekeeper? Good reasons to prefer a blockchain-based database over a trusted intermediary might include lower costs, faster transactions, automatic reconciliation, new regulation or a simple inability to find a suitable intermediary.

5. Transaction interaction

So blockchains make sense for databases that are shared by multiple writers who don’t entirely trust each other, and who modify that database directly. But that’s still not enough. Blockchains truly shine where there is some interaction between the transactions created by these writers.

What do I mean by interaction? In the fullest sense, this means that transactions created by different writers often depend on one other. For example, let’s say Alice sends some funds to Bob and then Bob sends some on to Charlie. In this case, Bob’s transaction is dependent on Alice’s one, and there’s no way to verify Bob’s transaction without checking Alice’s first. Because of this dependency, the transactions naturally belong together in a single shared database.

Taking this further, one nice feature of blockchains is that transactions can be created collaboratively by multiple writers, without either party exposing themselves to risk. This is what allows delivery versus payment settlement to be performed safely over a blockchain, without requiring a trusted intermediary.

A good case can also be made for situations where transactions from different writers are cross-correlated with each other, even if they remain independent. One example might be a shared identity database in which multiple entities validate different aspects of consumers’ identities. Although each such certification stands alone, the blockchain provides a useful way to bring everything together in a unified way.

6. Set the rules

This isn’t really a condition, but rather an inevitable consequence of the previous points. If we have a database modified directly by multiple writers, and those writers don’t fully trust each other, then the database must contain embedded rules restricting the transactions performed.

These rules are fundamentally different from the constraints that appear in traditional databases, because they relate to the legitimacy of transformations rather than the state of the database at a particular point in time. Every transaction is checked against these rules by every node in the network, and those that fail are rejected and not relayed on.

Asset ledgers contain a simple example of this type of rule, to prevent transactions creating assets out of thin air. The rule states that the total quantity of each asset in the ledger must be the same before and after every transaction.

7. Pick your validators

So far we’ve described a distributed database in which transactions can originate in many places, propagate between nodes in a peer-to-peer fashion, and are verified by every node independently. So where does a “blockchain” come in? Well, a blockchain’s job is to be the authoritative final transaction log, on whose contents all nodes provably agree.

Why do we need this log? First, it enables newly added nodes to calculate the database’s contents from scratch, without needing to trust another node. Second, it addresses the possibility that some nodes might miss some transactions, due to system downtime or a communications glitch. Without a transaction log, this would cause one node’s database to diverge from that of the others, undermining the goal of a shared database.

Third, it’s possible for two transactions to be in conflict, so that only one can be accepted. A classic example is a double spend in which the same asset is sent to two different recipients. In a peer-to-peer database with no central authority, nodes might have different opinions regarding which transaction to accept, because there is no objective right answer. By requiring transactions to be “confirmed” in a blockchain, we ensure that all nodes converge on the same decision.

Finally, in Ethereum-style blockchains, the precise ordering of transactions plays a crucial role, because every transaction can affect what happens in every subsequent one. In this case the blockchain acts to define the authoritative chronology, without which transactions cannot be processed at all.

A blockchain is literally a chain of blocks, in which each block contains a set of transactions that are confirmed as a group. But who is responsible for choosing the transactions that go into each block? In the kind of “private blockchain” which is suitable for enterprise applications, the answer is a closed group of validators (“miners”) who digitally sign the blocks they create. This whitelisting is combined with some form of distributed consensus scheme to prevent a minority of validators from seizing control of the chain. For example, MultiChain uses a scheme called mining diversity, in which the permitted miners work in a round-robin fashion, with some degree of leniency to allow for non-functioning nodes.

No matter which consensus scheme is used, the validating nodes have far less power than the owner of a traditional centralized database. Validators cannot fake transactions or modify the database in violation of its rules. In an asset ledger, that means they cannot spend other people’s money, nor change the total quantity of assets represented. Nonetheless there are still two ways in which validators can unduly influence a database’s contents:

• Transaction censorship. If enough of the validators collude maliciously, they can prevent a particular transaction from being confirmed in the blockchain, leaving it permanently in limbo.
• Biased conflict resolution. If two transactions conflict, the validator who creates the next block decides which transaction is confirmed on the blockchain, causing the other to be rejected. The fair choice would be the transaction that was seen first, but validators can choose based on other factors without revealing this.

Because of these problems, when deploying a blockchain-based database, you need to have a clear idea of who your validators are and why you trust them, collectively if not alone. Depending on the use case, the validators might be chosen as: (a) one or more nodes controlled by a single organization, (b) a core group of organizations that maintain the chain, or (c) every node on the network.

8. Back your assets

If you’ve got this far, you may have noticed that I tend to refer to blockchains as shared databases, rather than the more common “shared ledgers”. Why? Because as a technology, blockchains can be applied to problems far beyond the tracking of asset ownership. Any database which has multiple non-trusting writers can be implemented over a blockchain, without requiring a central intermediary. Examples include shared calendars, wiki-style collaboration and discussion forums.

Having said that, for now it seems that blockchains are mainly of interest to those who track the movement and exchange of financial assets. I can think of two reasons for this: (a) the finance sector is responding to the (in retrospect, minuscule) threat of cryptocurrencies like bitcoin, and (b) an asset ledger is the most simple and natural example of a shared database with interdependent transactions created by multiple non-trusting entities.

If you do want to use a blockchain as an asset ledger, you need to answer one additional crucial question: What is the nature of the assets being moved around? By this I don’t just mean cash or bonds or bills of lading, though of course that’s important as well. The question is rather: Who stands behind the assets represented on the blockchain? If the database says that I own 10 units of something, who will allow me to claim those 10 units in the real world? Who do I sue if I can’t convert what’s written in the blockchain into traditional physical assets? (See this asset agreement for an example.)

The answer, of course, will vary by the use case. For monetary assets, one can imagine custodial banks accepting cash in traditional form, and then crediting the accounts of depositors in a blockchain-powered distributed ledger. In trade finance, letters of credit and bills of lading would be backed by the importer’s bank and the shipping company respectively. And further in the future, we can imagine a time when the primary issuance of corporate bonds takes place directly on a blockchain by the company seeking to raise funds.

Conclusion

As I mentioned in the introduction, if your project does not fulfill every single one of these conditions, you should not be using a blockchain. In the absence of any of the first five, you should consider one of: (a) regular file storage, (b) a centralized database, (c) master–slave database replication, or (d) multiple databases to which users can subscribe.

And if you do fulfill the first five, there’s still work to do. You need to be able to express the rules of your application in terms of the transactions which a database allows. You need to be confident about who you can trust as validators and how you’ll define distributed consensus. And finally, if you’re looking at creating a shared ledger, you need to know who will be backing the assets which that ledger represents.

Got all the answers? Congratulations, you have a real blockchain use case. And we’d love to hear from you.

Please post any comments on LinkedIn. See also this follow up: Four genuine blockchain use cases.

I’m not a fan of the term “smart contracts”. For a start, it has been used by so many people for so many different things, that we should probably just ban it completely. For example, the first known reference is from 1997, when Nick Szabo used it to describe physical objects that change their behavior based on some data. More recently, the term has been used for the exact opposite: to describe computation on a blockchain which is influenced by external events such as the weather. For now let’s put both of these meanings aside.

I want to focus here on “smart contracts” in the sense of general purpose computation that takes place on a blockchain. This meaning was popularized by Ethereum, whose white paper is subtitled “A Next-Generation Smart Contract and Decentralized Application Platform”. As a result of the attention that Ethereum has received, this meaning has become the dominant one, with banks (and others) working away on smart contract proofs-of-concept. Of course, since we’re talking about regulated financial institutions, this is mostly in the context of private or permissioned blockchains, which have a limited set of identified participants. For reasons that are now well understood, public blockchains, for all of their genius, are not yet suited for enterprise purposes.

So is the future bright for smart contracts in private blockchains? Well, kind of, but not really. You see, the problem is:

In private blockchains, smart contracts combine four good ideas with one bad one.

So what are the good ideas? (a) expressing business logic as a computer program, (b) representing the events which trigger that logic as messages to the program, (c) using digital signatures to prove who sent the messages, and (d) putting all of the above on a blockchain.

And the bad one? Executing every program for every message on every blockchain node. In other words, making the execution of all programs the job of the blockchain, instead of just using it as storage for the programs and messages. And yet this global execution is the entire reason why Ethereum was developed.

If you’re aware of the deterministic nature of computation, know about the halting problem, and understand how data dependencies prevent concurrency then you may already be convinced. But if not, make yourself a coffee, take a deep breath, and follow me down the rabbit hole…

Understanding Ethereum

In order to understand Ethereum-style smart contracts, we need to start with bitcoin, the first (and still most popular) public blockchain. The bitcoin blockchain was originally designed for one thing only: moving the bitcoin currency from one owner to another. But once it was up and running, people started embedding “metadata” in transactions to serve other purposes, such as digital assets and document notarization. While some bitcoiners fought these applications, an official mechanism for metadata was introduced in March 2014, with usage growing exponentially ever since.

As well as projects built on the bitcoin blockchain, many next-generation public blockchains were developed and launched, such as Nxt, Bitshares, Ripple and Stellar. These were designed from the ground up to support a broader range of activities, such as user-created assets, decentralized exchange and collateralized borrowing. Each of these blockchains has a different set of features, as decided upon by its developers, and each must be upgraded by all of its users when a new feature is added. Things started to get rather messy.

Having been involved in some of these projects, Vitalik Buterin posed a simple but brilliant question: Instead of lots of application-specific blockchains, why not have a single public blockchain that can be programmed to do whatever we might want? This über-blockchain would be infinitely extendible, limited only by the imagination of those using it. The world of crypto-enthusiasts was almost unanimously convinced by this powerful idea. And so, with $18 million in crowd funding and to great excitement, Ethereum was born.

Ethereum is a new public blockchain with an associated cryptocurrency called “ether”, like hundreds which came before it. But unlike other blockchains, Ethereum enables anybody to create a “contract” inside the blockchain. A contract is a computer program with an associated miniature database, which can only be modified by the program that owns it. If a blockchain user wants to change a database, they must send a digitally signed message to its contract. The code in the contract examines this message to decide whether and how to react. (This “encapsulation” of code and data is also a foundation of object-oriented programming.)

Ethereum contracts can be written in one of several new programming languages, such as Solidity and Serpent. Like most programming languages, these are Turing complete, meaning that they can express any general purpose computation. A key feature of Turing complete languages is the loop structure, which performs an operation repeatedly until some condition is fulfilled. For example, a loop might be used to print the numbers from one to a million, without requiring a million lines of code. For the sake of efficiency, programs written for Ethereum are compiled (i.e. converted) into more compact bytecode before being stored on the chain. Ethereum nodes then execute this bytecode within a virtual machine, which is essentially a simulated computer running inside a real one.

When an Ethereum contract is created on the blockchain, it sets up the initial state of its database. Then it stops, waiting politely until it’s called upon. When a user of the blockchain (or another contract) sends it a message in a transaction, the contract leaps into action. Depending on the code within, it can identify the source of the message, trigger other contracts, modify its database and/or send back a response to the caller. All of these steps are performed independently on every node in the network, with identical results.

To give an example, a simple Ethereum subcurrency contract maintains a database of user balances for a particular asset. If it receives a message to transfer funds from Alice to Bob, it will (a) check the message was signed by Alice, (b) check that Alice has sufficient funds, (c) transfer funds from Alice’s to Bob’s account in the database and (d) respond that the operation was successful. Of course, we don’t need Ethereum for that, because a simple bitcoin-style blockchain with native asset support can do the same thing. Ethereum really comes into its own for complex multi-stage business logic, such as crowdfunding, decentralized exchanges, and hierarchical governance structures. Or so, at least, the promise goes.

Breaking it down

Now that we know how Ethereum smart contracts work, we can break them down into five constituent parts:

1. Expressing business logic as computer programs.
2. Representing the events which trigger that logic as messages to the programs.
3. Using digital signatures to prove who sent the messages.
4. Putting the programs, messages and signatures on a blockchain.
5. Executing every program for every message on every node.

To repeat what I said at the start, I believe that parts 1 through 4 are very good ideas. Let’s start with the first two (which, by the way, are not new). Unlike legal contracts which can have differences of interpretation, computer programs are unambiguous. For any given program in a well-defined programming language, the same input always leads to the same output. So if some business logic is expressed as a computer program, and events are represented as messages to that program, then the business outcome is set in stone. Indeed, this deterministic property of computation makes randomness a sticky problem in computer science, and even the geeks at Google can get it wrong.

What about digital signatures and blockchains? These avoid the need for a central authority to determine which messages were sent, in what order, and by whom. Instead, each participant creates a pair of private and public keys, and distributes its public key once to the other participants. Following that, they sign every message with their private key before distributing that message across the network. The other participants can then verify the message’s source using the sender’s public key only. It’s clever cryptographic stuff. Finally, by putting the program and signed messages on a blockchain, we can ensure that every participant has an identical view of who did what and when. Combined with deterministic computation, this means participants cannot disagree over the final business outcome.

But what about the last idea, of every node executing every program for every message? Here we come to the contentious part. Because while this global execution might be nice to have, it’s also not necessary. Because computation is deterministic, it makes no difference whether a program is executed by one node, every node, or some external process. It also doesn’t matter whether this happens in real time, on demand, or 10 years later. The computation’s result will always be the same. And if for some reason this isn’t the case, this can only be due to a problem in the blockchain software or network.

The trouble with computation

If it doesn’t matter where a computation takes place, why not do it everywhere? Well, it turns out that computer programs are unpredictable. However innocent they may look, they can take a long time to run. And sometimes, they go on running forever. Consider the following classic example (known as an LCG):

1. Set x to a single-digit number of your choice
2. Set y to 123*x+567
3. Set x to the last two digits of y, i.e. y modulo 100
4. If x is more than 2 then go back to step 2
5. Otherwise stop and output the value of x

Simple enough, right? So here’s a question for you: Will this program ever finish? Or will it get stuck in an infinite loop? Not so sure? Well let me put you out of your misery: It depends on the initial value of x.

If x is 0, 1, 2, 5, 6, 7 or 8, the program stops fairly quickly. But if x is 3, 4 or 9, it continues indefinitely. Don’t believe me? Open up Excel and try for yourself (you’ll need the “MOD” function).

If you couldn’t predict that just by looking at the code, don’t feel too bad. Because not only is this hard for people, it’s impossible for computers. The problem of determining whether a given program will finish executing is called the halting problem. In 1936, Alan Turing, of “Turing complete” and The Imitation Game fame, proved that it cannot be solved for the general case. Barring trivial exceptions, the only way to find out if a program will finish running is to run it for as long as it takes, and that could be forever.

For those of us who’d prefer to live without blue screens of death and spinning beach balls, it’s all rather inconvenient. But live with it we do and, remarkably, most software works smoothly most of the time. And if not, modern operating systems like Windows protect us against runaway code by letting us terminate programs manually. However the same thing can’t be done on a blockchain like Ethereum. If we allowed individual nodes to terminate computations at will, different nodes would have different opinions about the outcome of those computations. In other words, the network consensus would break down. So what’s a blockchain to do?

Ethereum’s answer is based on transaction fees, also known as gas. The sender of each transaction pays for the computations it triggers, and this payment is collected by the miner who confirms it in a block. To be more precise, every Ethereum transaction states up front how much of the sender’s “ether” can be spent on processing it. The fee is gradually spent as the contract executes, step-by-step, within the Ethereum Virtual Machine. If a transaction runs out of fees before it finishes executing, any database changes are reverted and the fee is not returned. If a transaction completes successfully, any remaining fee is returned to its sender. In this way, transactions can only burden the network to the extent that they’re willing to pay for it. It’s undoubtedly a neat economic solution, but it requires a native blockchain currency in order to work.

Smart contracts vs concurrency

If gas can prevent runaway computation, do smart contracts get the green light? Well, not so fast, because there’s another problem with smart contracts that we need to talk about:

Smart contracts work poorly for high transaction throughput.

Concurrency is one of the most fundamental issues in computer architecture. A system has good concurrency if it allows several processes to happen simultaneously and in any order. Concurrent systems reduce delays and enable much higher throughput overall, by making optimal use of technologies such as process scheduling, parallel processing and data partitioning. That’s how Google searches 30 trillion web pages almost 100,000 times per second.

In any computer system, a set of transactions can only be processed simultaneously if they don’t depend on, or interfere with, each other. Otherwise, different processing orders might lead to completely different outcomes. Now recall that a smart contract has an associated database, and that it performs general-purpose computation including loops. This means that, in response to a particular message, a smart contract might read or write every single piece of information in its database. For example, if it is managing a sub-currency, it might decide to pay some interest to every holder of that currency. Of course, this won’t always be the case. But the problem is: before running the contract’s program for a particular message, a blockchain node cannot predict which subset of the contract’s database it’s going to use. Nor can it tell whether this subset might have been different under different circumstances. And if one contract can trigger any other, this problem extends to the entire content of every database of every contract. So every transaction must be treated as if it could interfere with every other. In database terms, each transaction requires a global lock.

Now think about the world a blockchain node lives in. Transactions comes in from different peers, in no particular order, since there is no centrally managed queue. In addition, at average intervals of between 12 seconds (Ethereum) and 10 minutes (bitcoin), a new block comes in, confirming a set of transactions in a specific order. A node will probably have seen most of a block’s transactions already, but some may be new. Either way, the order of the transactions in the block is unlikely to reflect the order in which they arrived individually. And since the order of transactions might affect the outcome, this means transactions cannot be processed until their order in the blockchain is confirmed.

Now, it’s true that an unconfirmed bitcoin transaction might need to be reversed because of a double spend. But an unconfirmed Ethereum transaction has no predictable outcome at all. Indeed, current implementations of Ethereum don’t even process unconfimed transactions. But if an Ethereum node was to process transactions immediately, it would still need to rewind and replay them in the correct order when a block comes in. This reprocessing is a huge waste of effort, and prevents external processes from concurrently reading the Ethereum database while it goes on. (To be fair, it should be noted that bitcoin’s reference implementation also rewinds and replays transactions when a block comes in, but this is due only to a lack of optimization.)

So what is it about bitcoin’s transaction model that makes out-of-order execution possible? In bitcoin, each transaction explicitly states its relationship to other transactions. It has a set of inputs and outputs, in which each input is connected to the output of a previous transaction which it “spends”. There are no other dependencies to worry about. So long as (a) two bitcoin transactions don’t attempt to spend the same output, and (b) the output of one doesn’t lead to the input of another, a bitcoin node can be sure that the transactions are independent, and it can process them in any order. Their final positions in the blockchain don’t matter at all.

To use formal computer science terminology, Ethereum transactions must be strictly totally ordered, meaning that the relative order between every pair of transactions must be defined. By contrast, bitcoin transactions form a directed acyclic graph which is only partially ordered, meaning that some ambiguity in transaction ordering is allowed. When it comes to concurrency, this makes all the difference in the world.

To look at it in practical terms, there’s been a lot of talk about private blockchains in the enterprise. But a private blockchain is just a distributed database with some additional features. And if you tried selling an enterprise-class database today that does not support concurrency, you’d be laughed out of the room. Equally ludicrous would be the suggestion that an individual node has to wait 12 seconds before seeing the result of its own transactions. As Vitalik himself recently tweeted:

Key pt about dapp dev that I’ve underappreciated: main prob is not tx cost; ppl can handle $0.001. Prob is latency; ppl want 500ms, not 17s.

— Vitalik Buterin (@VitalikButerin) September 27, 2015

For us at Coin Sciences this is not just an academic issue, because we need to decide whether and how to incorporate smart contracts into MultiChain. Strangely enough, despite the hundreds of feature requests and questions we’ve received so far, only two have been related to smart contracts, and even then in a weaker form than Ethereum provides. So while we’re keeping an open mind, it may turn out that smart contracts don’t solve any real problems for our users.

In favor of Ethereum

If you’re interested in only one side of the argument, you can stop reading here. But you may be wondering: Are the creators of Ethereum stupid? Why on earth would they require global execution in a public distributed database, if each node could simply choose which programs it cared about running? Are there any good reasons for the Ethereum way?

Actually, if we’re talking about public blockchains, I believe there are. But in order to understand these reasons, we need to think about the dynamics of the Ethereum network itself.

Preventing transaction spam

A blockchain is maintained by a peer-to-peer network, in which each node is connected to a random subset of the other nodes. When a new transaction is created on one node, it spreads rapidly and haphazardly to the others through a process called “relaying”. In an open public network, anyone can create transactions, so we need a way to protect ourselves against transaction spam which could overwhelm the system. Since the network is decentralized, this can only be achieved by individual nodes assessing new transactions as they come in, and deciding whether or not to relay them. While this mechanism can’t prevent a spammer from overwhelming an individual node, it does protect the network as a whole.

In a public network, when a node decides whether to relay a new transaction, one key criterion is the ratio between its fee and its cost to the network. In the case of bitcoin, this cost is based mainly on the transaction’s raw size in bytes. In Ethereum, a more complex formula is used, based on the computational effort the transaction will consume. Either way, fees act as a market-based mechanism for the prevention of transaction spam.

But how does a node know if the sender has sufficient funds to cover the fee they’re offering? In the case of Ethereum, each user’s balance of “ether” is affected by the outcome of previous transactions, since contracts can both spend and pay out ether. So without actually executing all programs for all previous messages, an Ethereum node has no way of knowing a user’s up-to-date balance. Therefore, it cannot assess whether a transaction should be relayed to other nodes. And without that, an open network could be trivially destroyed.

Compact data proofs

In a blockchain, blocks are filled mostly by the transactions that they confirm. However each block also has a compact “header”, which contains important information such as a timestamp and a link to the previous block. For public blockchains based on proof of work hashing, the input for the hashing algorithm is this block header alone. This means that the authority of a chain can be assessed by a “lightweight client” without downloading most of its content. For example, as of November 2015, the complete set of bitcoin headers is 30 MB in size, compared to 45 GB for the full chain. That’s a ratio of 1500:1, making a crucial difference to mobile devices with limited bandwidth and storage.

The header of each Ethereum block contains a “state root”, which fingerprints the state of the chain after processing the transactions in that block. Among other things, this state covers the content of every contract’s database, with the fingerprint calculated efficiently using a tree of one-way hash functions. The slightest change to any contract’s database would lead to a completely different state root, so the root “ties down” the database’s contents. (An equivalent notion of “UTXO commitments” for bitcoin has been discussed but not yet implemented.)

The tree-like method of calculating state roots has an important property: Given a known state root, the value of a particular entry in a contract database can be proven efficiently. The size of this proof is proportional to the depth of a binary tree whose leaves are the individual database entries, i.e. log₂ the total database size. That means that, for an individual entry, the proof only doubles in length when the database size is squared – the kind of scalability that computer scientists kill for. Now recall that the state root of each block is in its header, which a lightweight client can verify. As a result, lightweight clients can safely and efficiently query any full node in the network for individual database entries, and full nodes cannot lie.

But if our blockchain headers include a state root, and the state root depends on the contents of the database, then every node must keep the blockchain’s database up to date. In turn this means running every contract for every message it has received so far. Without this, a mining node wouldn’t know the state root to place in a block header, nor could other nodes verify the blocks that they receive. The bottom line is: if we want lightweight clients to safely retrieve compact data proofs from the network, full nodes must perform all the computations described by the data in the chain.

The verdict for private blockchains

Let’s revisit these two arguments in the context of private blockchains. The first thing to note about private chains is that they tend not to have a native token or cryptocurrency. This is for several reasons:

• The sort of entities interested in private chains don’t want to deal with a new asset class.
• The consensus model for private chains is based on agreement between a set of closed miners, rather than proof of work. So the cost of mining is minimal and miners don’t need much reward.
• Since all the participants in a private chain are vetted, there is less concern over spam and abuse.

Recall that the first argument for global execution was to enable each Ethereum node to decide whether to relay an incoming transaction, based on the fee it offers. Well, the lack of a native token renders this reason irrelevant because, if a blockchain has no native token, transactions cannot pay fees. If for some reason spam remains an issue, it has to be controlled another way, e.g. by revoking the sender’s permissions.

Now let’s consider the second argument, to enable compact data proofs. A public blockchain is likely to have end users on mobile or other lightweight wallets. But this is less likely with private chains, whose primary function is sharing a database between larger companies. And if the blockchain is accessed from a mobile device, the mobile user is likely to be a customer of one of these companies, and can trust what that company tells them.

Instead, in private blockchains, the problems of global execution are especially acute. If a private blockchain has no native token, we have no gas-like market mechanism for preventing runaway code. Instead we would need to introduce some kind of fixed limit in terms of computational steps per transaction. But in order to allow transactions to intentionally perform a lot of processing, this limit would need to be high. As a result, the network could still end up wasting a lot of energy on unintended loops before finally shutting them down.

As for concurrency, private blockchains are far more likely to see the sort of transaction volumes that make concurrency essential. The capacity of public blockchains is limited by the fact that, in order to be meaningfully decentralized, they need thousands of nodes run by enthusiasts with limited budgets. By contrast private chains are far more likely to connect just a few dozen enterprises together, in which case capacity and speed are essential.

Double decker blockchains

So if everything about smart contracts makes sense in private chains, apart from global execution, where does this leave us? What type of blockchain will give us the performance and flexibility we need? To be honest, I’m still thinking about it. But one answer might be: a blockchain with two tiers.

The lower tier would be built on bitcoin-style transactions which are processed instantly and concurrently, and don’t need to wait for block confirmations. These transactions could perform simple movements of assets, including safe atomic exchanges, without resorting to smart contracts. But this lower tier would also be used as a blind storage layer for the programs and messages that represent more complex business processes, embedded as transaction metadata.

As for the upper tier, each network participant would choose which programs they want to run. Some might choose to run none at all, because they are only interested in simple asset movements. Others might execute a small group of programs that are relevant to their internal processes (with the knowledge that this group exchanges no messages with programs outside). A few might even opt for global execution, processing every message for every program, just like Ethereum. But the key thing would be that every node runs only the code it needs to. In computer science this technique is called lazy evaluation, because it entails doing as little work as possible, without omitting anything crucial. With lazy evaluation, if a blockchain computation goes awry, only those nodes which actually execute that program will notice. The network itself won’t feel a thing.

As for MultiChain, if we do end up supporting Turing complete computation, I doubt we’ll implement global execution. Perhaps we’ll go for this kind of lazy two-tiered approach, or perhaps we’ll think of something better.

Smart contracts in public blockchains

As I argued earlier, in public Turing complete blockchains like Ethereum, there are good reasons for global execution. But here’s another question: what is the enterprise use case for these chains? Let’s imagine some future time when enterprises have sufficient confidence in public blockchains to use them for real business processes. If a group of companies wants to embed some computational logic in a public blockchain, they have two choices: (1) using an Ethereum-style blockchain with global execution, or (2) using any blockchain as a simple storage layer and executing the code themselves. And given these options, why would they choose (1)?

The rational choice would be the public blockchain with (a) the cheapest price per storage byte, and/or (b) the highest level of security based on total mining power. Since computation is deterministic, the companies need only pay the network to store their contract and messages, not to actually process them. In addition, by using a blockchain for storage only, they can use any programming language they want. This might be Ethereum bytecode, JavaScript/ECMAScript for readability or even machine code for high performance. In fact, Ethereum contracts can already be stored using metadata on the bitcoin blockchain. This is exactly the two-tier approach I suggested.

This discussion is related to the notion of abstraction layers, made famous by the OSI networking model. For optimal reliability and flexibility, each layer of a system should be as abstracted (i.e. independent) from the other layers as possible. For example, we wouldn’t want our hard disk controllers to contain code for rendering JPEG images. So why would we want a blockchain to execute the programs that it stores? For the majority of use cases, we derive no benefit from this, and it comes at a significant cost.

Epilogue

If global execution doesn’t make sense in private blockchains, why is everyone working on this stuff? I think this can be explained, at least in part, by a misunderstanding over what blockchains can do in the real world. You see, public blockchains like bitcoin can directly move a real asset (namely, their native currency), because the blockchain defines the ownership of that currency. This conflates two aspects of assets which are usually distinct: (a) a ledger which records who owns the asset, and (b) its actual physical location. This makes cryptocurrencies the ultimate bearer instrument, creating a brave new world or a money launderers’ paradise, depending on who you ask.

But for other assets which exist independently of a blockchain, the only thing a chain can do is hold a record of who they should belong to. This will remain the case until we see the primary issuance of assets onto a blockchain, with legal ownership of that asset defined in terms of the chain’s database. For the institutional finance sector, I believe this day is still a long way off, not least because of the regulatory changes required. Until then, there will always be an extra step, contractual and procedural, between what the blockchain says and what happens in the real world. This step might as well include some Turing complete code, lazily executed at the last possible moment.

This problem is highlighted by the case of “smart bonds” that we’ve heard so much about. A smart bond is directly issued onto a blockchain, with the blockchain ensuring that coupon payments are made to the bond holders at the appropriate times. All well and good. But what happens if the bond issuer has insufficient funds in their blockchain account to cover a payment which is due? The blockchain can certainly set a flag to say that something is amiss, but it can’t do anything else. We still need an army of lawyers and accountants to sort the whole mess out, whether by a haircut, debt restructuring, forfeiture or outright bankruptcy. In short:

If smart contracts can’t deliver their promise, why are we paying their price?

Thank you for reading.

And so it goes on. From popular posts to contemptuous tweets to predictions about the future, the world and its mother are lining up to throw tomatoes at private blockchains, before even understanding what they are.

Saying that a private blockchain is just a shared database is like saying that HTML and HTTP are “just” distributed hypertext. It’s wrong in two ways. First, the semantic one: private blockchains are a technology that enables shared databases, like pens enable writing and HTML/HTTP enable distributed hypertext. The bitcoin blockchain and its primary application cannot be meaningfully separated, because one could not exist without the other. But this equivalence does not apply to private blockchains at all.

The second mistake is the use of the word “just”. Just? Were HTML and HTTP just a way to do distributed hypertext? Hypertext was invented decades earlier, so are these technologies a minor footnote in computer history? Oh but let me count the ways in which they earned their place: (a) a simple markup language that any layperson could learn, (b) a hierarchical addressing scheme that works both with TCP/IP and our conceptual model of place, (c) a simple protocol for the state-free retrieval of content, and (d) both client and server software that brought the whole thing to life. We might as well say that Newton was just a scientist and Dostoyevsky just a writer.

So let’s make this perfectly clear: Yes, private blockchains are just a way to share a database. But they enable a new type of shared database, with huge implications for the financial world and beyond. And if you’re willing to read on, I’m going to tell you exactly why.

What is a database?

A database is a repository of structured information, organized into tables. You can think of it as a collection of one or more Excel spreadsheets, which can optionally be linked together. Each table contains information about a set of entities of a particular type, with one entity per row. Each table also has one or more columns, which describe different aspects of those entities. For example, the table for WidgetCo’s internal staff directory might have columns for employee ID, first name, last name, department, internal phone number and room number.

One of the important ways in which databases go beyond spreadsheets is that they contain rules about the data stored within. These rules help ensure that the information remains sane and consistent for the benefit of the entire organization. In today’s most popular databases, the rules take a number of common forms:

• The database schema defines what kind of information is permitted in each column. For example, the phone number must contain 4 digits and cannot be left blank (“null”).
• Unique keys which state that a particular column (e.g. employee ID) must have a different value in every row.
• Check constraints which enforce relationships between the column values in each row. For example, if the department is “Procurement” then the room number must start with a 3 or 4.
• Foreign keys which enforce relationships between tables. For example, if the database contains another table used for payroll, there might be a rule that every employee ID in the payroll table must also exist in the staff directory.

A transaction is a collection of changes to a database that is accepted or rejected as a whole. Every time a transaction modifies the database, the software ensures that the database’s rules are followed. If any part of a transaction violates one of these rules, the entire transaction will be rejected with a corresponding error.

There are other more esoteric rule types I could list, but they all have one thing in common. They answer the question: Is the database in a valid state? In other words, they act as a constraint on the database’s contents when viewed at a single point in time. And this works just fine for a database which sits inside a single organization, because the main job of the constraints is to prevent programmer error. If one of WidgetCo’s internal applications tried to insert a 3-digit phone number into the directory, this wouldn’t be due to malice, but rather a bug in the developer’s thinking or code. The ability of a database to catch these mistakes is undoubtedly handy, and helps prevent bad information propagating within an organization, but it doesn’t fix problems of trust. (Constraints can also help simplify application logic, for example via foreign key cascading or on-duplicate clauses, but these are still just ways to help developers.)

Database sharing

Now let’s think about how WidgetCo’s internal staff directory might be shared with the outside world. In many cases, there is no problem providing shared read access. The directory can be exported to a text file and emailed to customers and suppliers. It can be posted on the Internet, just like this one. It can even be given an API to allow searching by external code. Shared read is a technical doddle, a question of deciding who can see what.

But things start getting stickier when we think about shared write. What if WidgetCo wants an external entity to modify its database? Perhaps the phones are being replaced by PhoneCo, who will then update the phone numbers in the staff directory. In this case, WidgetCo would create a new “account” for PhoneCo to use. Unlike accounts for WidgetCo’s internal use, PhoneCo’s account is only permitted to change the phone number column, and never add or delete rows. All of PhoneCo’s transactions are processed by WidgetCo’s database system, which now applies two types of restriction:

• Global rules which apply to all database users. For example, the phone company can’t change a number to contain only 3 digits, and neither can anybody else.
• Per-account rules restricting what PhoneCo is permitted to do, in this case only modifying the phone number column of existing rows.

So far, so good. We have a shared write database. It works because WidgetCo is in charge of the database and the phone company gains access by virtue of WidgetCo’s good grace. If PhoneCo started setting phone numbers randomly, WidgetCo can shut down their access, terminate their contract, and restore some old data from a backup. And if WidgetCo started misbehaving, say by reversing the new phone numbers entered by PhoneCo, well that would be entirely pointless, since it would only harm WidgetCo themselves. The phone company would consider WidgetCo to be a peculiar customer but not particularly care, so long as they paid their bill on time.

But now let’s see what happens if two or more parties want to share a database which (a) none of the parties controls, (b) can be written to by any party, and (c) can be relied upon by everyone. To make things worse, let’s say that these parties have different incentives, don’t trust each other and may even be fierce competitors. In this case, the solution has always been the same: introduce a trusted intermediary. This intermediary manages a database centrally, provides accounts to all of the parties, and ensures that all operations are permitted according to a known set of rules. In many cases, especially financial, every party still maintains its own copy of the data, so everyone spends a lot of time checking that their databases agree.

It all gets rather messy and cumbersome. But if we’re talking about a shared write database in an environment of limited trust, there is currently no alternative. That’s one of the main reasons why financial transactions go through central clearing houses, why you use Google Calendar even in a small workgroup, and why the crowd-sourced wonder that is Wikipedia spends millions of dollars on hosting. Even as the user interface of the web moves to the client side, centralized servers continue to store the data on which those interfaces rely.

Real shared write

So let’s say that we wanted to allow a database to be shared, in a write sense, without a central authority. For example, several competing companies want to maintain a joint staff directory for the benefit of their mutual customers. What might that actually look like? Well, it would need a number of things:

• A robust peer-to-peer network that allows transactions to be created by any party and propagated quickly to all connected nodes.
• A way to identify conflicts between transactions and resolve them automatically.
• A synchronization technology that ensures all peers converge on an identical copy of the database.
• A method for tagging different pieces of information as belonging to different participants, and enforcing this form of data ownership without a central authority.
• A paradigm for expressing restrictions on which operations are permitted, e.g. to prevent one company from inflating the directory with fictitious entries.

Whew. That’s a tough list right there, and it’s simply not supported by today’s off-the-shelf databases. Current peer-to-peer replication technology is clumsy and has a complex approach to conflict resolution. Those databases that do support row-based security still require a central authority to enforce it. And standard database-level restrictions like unique keys and check constraints cannot protect a database against malicious modifications. The bottom line is this:

We need a whole bunch of new stuff for shared write databases to work, and it just so happens that blockchains provide them.

I won’t go into too much detail about how blockchains do these things, because I’ve covered much of it before. Some key elements include regular peer-to-peer techniques, grouping transactions into blocks, one-way cryptographic hash functions, a multi-party consensus algorithm, distributed multiversion concurrency control and per-row permissions based on public key cryptography. A long list of old ideas combined in a new way. HTML/HTTP, if you like.

In addition to all of these, shared write databases require an entirely new type of rule, to restrict the transformations that a transaction can perform. This is an absolutely key innovation, and makes all the difference if we’re sharing a database between non-trusting entities. These types of rules can be expressed as bitcoin-style transaction constraints or Ethereum-style enforced stored procedures (“smart contracts”), each of which has advantages and disadvantages. Perhaps there are other better ways waiting to be discovered. But they all share the property of tying together the database’s state before and after a transaction takes place. In other words, they answer the question: Was that a valid transaction? This is fundamentally different from asking whether the database is valid at a single point in time.

If you’re wondering if this type of database has useful real-world applications, well that’s a fair question. But you might note the intense interest in private blockchains from one sector at least, because of their potential for simplifying processes and reducing costs and delays. Financial institutions are heavy users of today’s database platforms, and those platforms do not enable a shared write scenario. This is what banks are looking for.

This problem and its solution have absolutely nothing to do with bitcoin and the idea of censorship-free money. In fact, the only connection to bitcoin is the technical similarity between the bitcoin blockchain and how some of these private blockchains are implemented today. One key difference is that private blockchains don’t need proof of work mining, since blocks are created by a closed set of identified participants. Over time the two worlds may well diverge further, because their requirements are completely different. Whether you like financial regulation or not, the simple fact is that private blockchains are potentially useful in a regulated world, whereas for now at least, public blockchains are not.

If I may finish with an analogy, the UN Declaration on the Principles of International Law does not tell countries that they can hold any territory they want, so long as it’s surrounded by a clearly-marked fence. Rather, it states that “No territorial acquisition resulting from the threat or use of force shall be recognized as legal”. In other words, it’s a rule regarding the legitimacy of changes, not just of situations. And the UN declaration, which seems so obvious to us now, was a complete revolution in international law. It meant a world no longer based on unilateral power and authority, but one where differences can be resolved by mutual consensus.

When it comes to shared databases, private blockchains do exactly the same thing.

Trading between people is as old as humanity itself. It began at the moment when caveman Ogg said to caveman Ugg: “me give you rock, you give me berries”. But trading carries with it a fundamental problem: it requires trust. What stops Ogg from using the rock to bash Ugg, then grabbing both rock and berries before running away? How do we translate a verbal exchange agreement into an enforcement mechanism that ensures both sides keep their word?

To take a modern example, a few years ago I sold a car on the second-hand market. I found a buyer over the Internet, we met in person, he had the car tested and we agreed on a price. So he went to his bank to get a cashier’s check, which is effectively cash in a more compact form. We walked together to a post office, where I can sign and submit an official government form that transfers legal ownership of the car.

So there we are, standing at the post office window, and we reach an awkward impasse. The check is still in his pocket, and I’m holding the signed form. We met a few hours ago and have no reason to trust each other. Do I hand in the form first then hope he gives me the check, rather than run away? Or does he hand me the check then hope I give in the form? Either way, someone is exposing themselves to the risk of betrayal.

And then it dawned on me that I should stop worrying and just hand in the form. Why? Because one of two things could happen next. Either the buyer hands me the check, in which case everyone’s happy and the exchange is complete. But what if he runs off instead? In that case, the post office clerk will see, and tear up the form I just gave him. Bingo, we have ourselves a safe exchange.

Did you see what happened there? Our dilemma was solved through the use of an intermediary, in this case the post office clerk. The clerk ensures that either a fair transaction takes place, or no transaction at all. And not just any intermediary can provide this service. It has to be someone trusted by both parties. In the case of an employee of a government-owned post office, this stems from our trust in the government itself. If post office clerks could be bribed, either I or the buyer could engineer a situation where we end up with both cash and car. Indeed, in many countries, corruption like this can be a huge drain on prosperity.

Cavemen and cars are one thing, but let’s shift our focus to the financial world, in which trading plays a central role. Of course, banks don’t pay their employees to run off with someone else’s shares. But the safe exchange of financial assets remains an important problem, because there are less cartoonish ways in which participants in a transaction can fail to uphold their promise. For example, one party might become insolvent, or a sudden change in market conditions might prevent them from securing an asset. They can suffer from clerical errors or from the knock-on effects of an accounting fraud at another counterparty.

As a result of these “settlement risks”, most financial transactions are settled using delivery versus payment (DvP). This is just a fancy term for the post office process described above. DvP ensures that, if one party to a transaction doesn’t deliver what was promised, the other party can keep the asset they offered in exchange.

And how is delivery versus payment implemented in the world of finance? You guessed it, via trusted intermediaries. These could be other banks, clearing houses or central securities depositories. Since most of today’s trades occur digitally, this isn’t a matter of managing the transfer of physical certificates or cash. Rather, DvP is achieved by the intermediary simultaneously updating a number of records in their database and/or transmitting instructions to other institutions.

Delivery versus payment by blockchain

Talking about databases brings us neatly to the subject of blockchains. A blockchain allows a ledger or database to be shared and synchronized between a number of parties. However, unlike regular databases, blockchain databases can be safely and directly modified by multiple users even if they are in fierce competition with each other. If you work in corporate IT, you might want to give the implications of that sentence some thought.

To understand how delivery versus payment works on a blockchain, we need to start by understanding bitcoin’s transactional model. It should be noted here that other blockchain designs use a different model for transactions, and we’ll talk more about these differences later on.

A bitcoin transaction has a set of inputs and outputs. Each input is connected to one output of a previous transaction, with all the bitcoin from the previous output flowing in. The bitcoin in a transaction’s inputs are then redistributed across its outputs according to the quantities written within. In addition, each transaction output contains the public identifier of its new owner, for which the owner holds a corresponding private key. A bitcoin transaction is only valid if:

• The total quantity of bitcoin in the transaction’s inputs is greater or equal to the quantity written in its outputs. Any difference is collected as a fee by the “miner” who confirms the transaction in a block, creating a market mechanism by which transactions can bid for confirmation.
• The transaction is approved by the owners of every prior output which that transaction “spends”. This approval is expressed via a cryptographic signature of the new transaction’s content. The signature for a prior output can only be created using the private key which matches its public identifier.

Both of these rules are crucial in a financial ledger which is shared between non-trusting parties. Without the first, anybody could create bitcoins out of thin air. And without the second, everybody could spend everybody else’s bitcoins. But we also need a third rule, which is enforced globally rather than within individual transactions:

• Each transaction output can only be used by one subsequent transaction. This prevents an attack known as double-spending in which the same bitcoins are sent to more than one recipient.

To enforce this rule, the blockchain contains a chronological log of valid transactions which do not conflict with each other, and this log is independently verified by every node in the network.

The bitcoin transactional model can be easily extended to represent any financial asset. Instead of a transaction output containing bitcoins, it can hold an asset identifier and quantity. All of the rules covering bitcoin transactions still apply, preventing participants from (a) creating assets out of thin air, (b) spending other people’s assets, and (c) spending the same asset twice. For non-cryptocurrency assets, we tend to insist that input and output quantities balance exactly, rather than allowing miners to collect the difference.

So how do we create a safe delivery versus payment transaction using this model? Let’s say that Alice and Bob have agreed to exchange Alice’s £10 for Bob’s $15. For the sake of convenience we’ll assume that Alice already has exactly £10 sitting neatly in a single transaction output, and Bob likewise has $15. (If this is not the case, they can easily shift their funds around to make it so.)

To begin with, either party builds a transaction with two inputs and two outputs. The two inputs spend the prior outputs containing Alice’s £10 and Bob’s $15 respectively. As for the outputs, the first contains Alice’s identifier and $15, and the second goes to Bob containing £10. Since the input and output quantities in both currencies balance, our transaction fulfils the first condition above. To fulfil the second, both Alice and Bob must now sign the transaction, since it spends prior outputs belonging to each of them.

The transaction can now be finalized by including it on the blockchain, but we still need to consider the problem of double-spends. What if Alice had created a conflicting transaction exchanging the same £10 with a different counterparty who offered her a better deal? Here the third rule comes into play, in which the blockchain ensures that each output can only be spent once. If the competing transaction is transmitted after Alice’s exchange with Bob is on the blockchain, then it simply won’t get confirmed. And if the competing transaction was confirmed first, Alice’s exchange with Bob will fail instead. Either way, the blockchain ensures delivery versus payment for Alice and Bob’s exchange, as well as any other. If Bob doesn’t get Alice’s £10, then Alice doesn’t get his $15.

The power of partial transactions

So blockchains give us a way for two parties to come together, build and sign an exchange transaction, and ensure that it succeeds or fails as a whole. This enables delivery versus payment on a shared ledger, without needing a trusted intermediary to manage the process. The miners who confirm transactions in blocks still have some power, but it’s much less than a traditional intermediary. The worst they can do is refuse to confirm a particular transaction in its entirety, and this does not violate DvP. Furthermore, if mining is shared between the parties actually creating the transactions, this risk falls away completely, since everyone will get a chance to confirm their own.

So far, so good. But bitcoin-style blockchains have more tricks up their sleeve. Recall that a transaction must be signed by the owner of each prior output which that transaction spends. By default, this signature locks down the full list of inputs and outputs within the transaction. The cryptography ensures that the slightest modification to an input or output would render the signature invalid. To follow the example above, if Bob was substituted for Carol after Alice signed the transaction, then the transaction would completely fail.

But what if Alice doesn’t care who she performs the exchange with? For most purposes, why should she care? Unless Alice is determined to work specifically with Bob, there are only two parts of the transaction that truly concern her. First, the fact that her £10 output will be spent, rather than a different quantity or asset. Second, that she receives $15 in a new output in return. So long as all the money in the system is clean, Alice doesn’t really mind where that $15 comes from, or what else might happen to facilitate her exchange.

Perhaps a single party will come along with $15 and perform a straight swap for Alice’s £10. But maybe Bob and Carol only want to exchange $7.50 each. In this case, they would add two inputs to the transaction, along with two outputs collecting £5 each. Or maybe Carol actually wants to exchange $15 for 950 rubles, while Sasha in Moscow has 950 rubles and is looking for £10. In this case a 3-way exchange can take place, in which each party still only cares about their own piece of the puzzle. The transaction that Alice started can be completed in an infinite number of different ways. But from Alice’s perspective, these all achieve the same purpose of giving her $15 in exchange for £10, and they all make her equally happy.

How does a blockchain facilitate this? Through partial transactions and partial signatures. Alice starts a transaction with a single input (her £10) and a single output ($15 to her). She locks down these parts of the transaction with a digital signature which states that any number of other inputs or outputs can be added. She hands this partial transaction to Bob and says “see what you can do”. Maybe she hands it to Carol as well, and to any number of other potential counterparties or syndicate-builders. Each of these can add on their own pairs of inputs and outputs, either to balance the exchange, or to create a larger partial transaction that can be handed on again. No matter what anyone does, the transaction can only be executed (i.e. settled through confirmation on the blockchain) once the input and output assets are balanced.

A blockchain transaction is just a chunk of digital data, so these partial transactions can be sent over email or any other communications medium. They can even be posted publicly, because the participants in the potential transaction know that the blockchain will take care of them. Alice’s signature ensures that she will only spend £10 if someone gives her $15 in exchange.

Finally, if Alice chooses to disable the offer, all she has to do is spend that same £10 in another transaction, most simply by sending it back to herself. Because the blockchain won’t allow the same output to be spent twice, this makes her existing partial transaction worthless. All the other participants on the blockchain will see this, and stop wasting their time trying to complete the exchange.

From DvP to smart contracts

As I have argued previously, a bitcoin-style blockchain can be viewed as a way to manage synchronization and security in a shared relational database. Both bitcoin and database transactions are treated atomically, meaning that they succeed or fail as a whole. The key to the analogy is the equivalence between a transaction output in a blockchain, and a row in the database. A blockchain transaction which spends some outputs and creates some others is the same as a database transaction which deletes some rows and creates some others instead. (A database operation that modifies an existing row is equivalent to deleting that row and creating a new updated one in its place. This equivalence underlies the popular MVCC method of concurrency control in databases, of which bitcoin-style blockchains can be seen as a distributed form.)

So let’s imagine that our financial data is held in a database, in which each row contains three pieces of information: its owner’s identifier, an asset identifier and an asset quantity. A blockchain enables this ledger to be safely shared between its participants, even if they do not trust each other at all. In the language of databases, it ensures that:

• The asset quantities in the rows deleted by a transaction match those in the rows it creates.
• For every row deleted (or modified) by a transaction, the transaction must be signed by the owner of that row.
• If a database row was deleted by one transaction, this prevents another transaction from deleting it again.

Let’s look at the first of these rules, namely that transactions must preserve asset quantities. We can broaden this into the general notion of a “transaction constraint”. A transaction constraint takes the form of a black box which sees two sets of rows for each transaction: (a) the rows deleted by the transaction, (b) the rows that it creates. The black box’s job is to look at these two sets and answer ‘yes’ or ‘no’ as to whether the transaction is valid. In our specific case, it will only answer yes if the total asset quantities in both sets match exactly.

Once we have the ability to apply transaction constraints, they can be extended to contain any set of rules. Some examples might be “a unit of this asset can only be created if these three other assets are simultaneously locked in escrow” or “this asset can only be transferred if there’s a corresponding row reporting insufficient rain”. From the perspective of a blockchain’s distributed architecture, the logic inside the box make no difference, so long as it can give us a definite and consistent evaluation of every transaction that it sees.

As a result, transaction constraints can serve as a general method for restricting the data transformations that blockchain participants can perform. This approach to “smart contracts” provides an alternative to the stored procedures used in Ethereum and its Eris derivative. In a future piece we’ll dive deeper into the advantages and disadvantages of these two paradigms, in terms of simplicity, scalability and concurrency.

You can follow me on Twitter here. See also: Ending the bitcoin vs blockchain debate.

Technical addendum

To build partial DvP transactions, use a signature type of SINGLE|ANYONECANPAY. If you’re using MultiChain, the preparelockunspent, createrawexchange and appendrawexchange API calls take care of the details for you. See the Getting Started page for a simple example of how they can be used.

Please post any comments on LinkedIn.

The debate has been running for a while but the past month has seen a serious uptick. The question being asked is:

Is there any value in a blockchain without a cryptocurrency? And can these “tokenless shared ledgers” be called blockchains at all?

So I’ve read Bailey’s article, watched Tim’s video, read this Nasdaq post, followed Richard’s every word, and even had my own good-spirited debate (see comments) with the Counterparty foundation’s Chris DeRose. So much hot air.

One thing Chris does well is boil it down to the question: is the blockchain an economic or a computer science innovation? The implication is that if blockchains are a purely economic innovation, there is no point to blockchains without cryptocurrencies. So let me state my position at the start:

The bitcoin blockchain was both an economic and a computer science innovation.

I’m allowing “innovation” here to include a new combination of existing techniques, rather than something which has no precedent whatsoever. This definition allows the world wide web to be considered as an innovation, even though it did little more than combine hypertext with a twist on some existing Internet protocols. If you want to adopt a stricter definition of innovation, be my guest, but you’ll be surprised at how few true “innovations” remain. To paraphrase The Teacher, there is little new under the sun.

To be precise, I am making the claim that blockchains without a token do serve a purpose, but it’s a different purpose compared to the original bitcoin blockchain. Crypto-heads laugh at token-free blockchains because they can’t provide censorship resistance and decentralized security through proof-of-work. Fintech-heads laugh at public blockchains because they are slow, expensive and unsuitable for traditional finance. Well, keep laughing everybody, because I believe you are both right.

I’m going to argue that token-free blockchains are useful for keeping decentralized databases in sync, even in a single organization in which there is perfect trust. And then we’ll see what other features blockchains offer, which make them suitable for creating consensus for specific types of transactions between organizations, where there is only limited and imperfect trust.

Unfortunately, to follow the argument, you’re going to have to geek out with me on the bitcoin transactional model, database multiversion concurrency control (MVCC), and the problem of conflict resolution in multi-master database replication. I’ll do my best to stick to English but still, this is technical stuff, and there’s no avoiding it.

Bitcoin’s transactional model

The bitcoin transactional model is simple but powerful. Every bitcoin transaction has a set of inputs and a set of outputs. Each input “spends” one output of a previous transaction. All of the bitcoin in a transaction’s inputs flow into that transaction and are distributed across its outputs according to the quantities written within. In this way, transactions form a multi-way connected chain which terminates at the “coinbase” transactions in which new bitcoins are created.

Bitcoin has a bunch of additional rules which are enforced by every node in the network:

• Every input in a transaction must prove that it has the right to spend the prior output to which it is connected. That right is restricted by conditions encoded within the prior output.
• A transaction must have sufficient total bitcoin in its inputs to cover the total written in its outputs. The only exceptions are coinbase transactions which create new units of the currency.
• Each output can only be spent once, in other words, it can only be connected to one input in one subsequent transaction.

Because of this last rule, the network requires a mechanism for reaching consensus about which transactions are valid, and this is what the blockchain does. Specifically:

If two transactions attempt to spend the same output, then only one of those transactions will ultimately be accepted. A blockchain acts as a unified mechanism to detect and prevent these conflicts across the network.

The blockchain is structured as a series of linked blocks, in which each block contains a set of transactions that don’t conflict with each other or with previous blocks, starting from the first block created in 2009. In theory, the chain could contain a series of individual transactions, but by grouping transactions into blocks, we gain a number of efficiencies that make the scheme more practical.

So what is the purpose of a cryptocurrency in all this? It comes down to the question of who decides on the blocks which form the chain. Bitcoin is decentralized and has no authority that can make this decision, so it needs to find some other way of reaching consensus.

We might like to use a democratic approach, in which nodes in the network vote on blocks, and the majority wins. Unfortunately, as any Internet poll can demonstrate, representative democracy is not possible online, because of the problem of impersonation (also known as a Sybil attack). One person can take over a million computers and decide how they vote, thus seizing control of the network consensus. Nobody else will even know this has happened.

To solve this, bitcoin makes it deliberately difficult to add a block to the chain, via a process called “mining”. To create a block, you must solve a difficult but pointless mathematical problem that demands a lot of computation (and therefore electricity and money). You also need some luck, since you are in competition with many other block miners around the world. You can’t get ahead for long by buying a more powerful mining computer, because the network regularly adjusts the problem’s difficulty to keep a steady global rate of one block per 10 minutes.

If it’s so difficult and costly to create a block, why would anyone bother? The answer is in the block reward. The successful miner of a block controls the coinbase transaction that awards them 25 bitcoins (this sum halves every four years). They can sell these bitcoins on the open market for $7,000 (at today’s rate), pay off their electricity bill and hopefully pocket some profit. Miners also collect a little extra from fees that are attached to transactions, although for now these fees play a minor role.

So bitcoin generates consensus via proof-of-work and the crux of the bitcoin-heads’ argument is this: Without a cryptocurrency, there is no way to incentivize decentralized mining of blocks. Therefore there is no way to secure an open blockchain against impersonation attacks. Therefore anybody can monopolize the network consensus and render the whole thing useless. I won’t argue with any of that.

Multiversion concurrency control

In the meantime I want to talk about something that may seem completely unrelated.

A database is a repository of structured information, grouped into spreadsheet-like entities called tables. A simple example of such a table is a list of bank accounts, in which each row contains an account number along with the balance of that account. Let’s say your account starts the day with a balance of $900. Today an automatic mortgage payment of $750 is scheduled and you also need to withdraw $400 from an ATM. Unfortunately you do not have an overdraft facility so one of these operations is set up to fail.

The processes for mortgage payments and ATM withdrawals run on separate systems, both of which access this single account database. Let’s say that each process works by reading your account’s balance, checking it is sufficient for the operation, initiating that operation, verifying the operation completes, calculating the new balance and then finally writing it into the database.

So long as your mortgage payment and ATM withdrawal don’t overlap, this logic will work fine. The first operation will execute successfully, and the second will abort because your account has insufficient funds. Depending on the order, you’ll get an angry phone call from the bank or a rude message on the ATM screen.

But what happens if the two processes happen to start at the same time? In this case, each will read your account’s balance and deem it sufficient to proceed. When the mortgage payment completes, your new balance will be calculated as $150 and written to the database. When the ATM withdrawal completes, your new balance of $500 will similarly be written. One of these write operations is going to override the other and, depending on your luck, you’ll receive a $750 or $400 bonus from your bank. No doubt you’ll soon learn to time your ATM visits for mortgage day.

Of course, this doesn’t happen in reality, because of a database technology called concurrency control. Concurrency control keeps our data (especially financial) sane and secure, and it comes in many forms. But all share the principle that database operations are grouped into “transactions”, which are treated atomically, meaning that they succeed or fail as a whole. Concurrency preserves consistency by locking or freezing parts of a database while they are in use by one transaction, to prevent other transactions from operating on the same information in a conflicting way.

If we didn’t need to run transactions in parallel, we could lock the entire database for the entire duration of every single transaction. However this is not practical in most real-world applications. A good concurrency control scheme permits parallel operations by locking as little data as possible for as short a time as possible. In the example above, only the database row corresponding to your account would be locked, and only for the split second in which a final check and deduction took place. A conflicting transaction operating in parallel would simply have to wait until this lock is released.

One popular concurrency control technique is called multiversion concurrency control, or MVCC for short. In MVCC, each transaction sees a consistent snapshot of the data at a certain point in time, even if part of that data is in the process of being updated by a second simultaneous transaction. This snapshot isolation property ensures, for example, that a statement showing our total balance across several accounts will always be correct, even if some funds are in the process of moving from one account to another. One transaction will only affect the data seen by a second transaction if the second begins after all of the first’s changes have been successfully applied.

Behind the scenes, MVCC works by allowing multiple versions of a row to be maintained simultaneously, alongside a timestamp that represents each version’s date of last modification. Modifying a database row in MVCC marks the current version of that row for deletion, while applying the modification to a copy of that row with an updated timestamp. From the perspective of the database’s storage layer, there is no such thing as modifying a row in place. Each transaction knows exactly when it started, and only sees versions of rows whose timestamp predates that time. Old versions of rows can be removed from storage once there are no ongoing transactions which might need to access them.

Crucially for our purposes here, MVCC prevents conflicts between write operations. Specifically:

If two transactions attempt to delete the same row version, then only one of these transactions will ultimately be accepted. Multiversion concurrency control acts as a unified mechanism to detect and prevent these conflicts within a database.

Ring any bells? There’s one more piece of background we need to discuss.

Multi-master database replication

Now let’s talk about database replication, in which a database exists in multiple copies. There are a number of good reasons to replicate a database, such as:

• To increase reliability, so that if one copy of the database is lost (e.g. due to a disk failure), we can instantly switch over to a second copy.
• To increase throughput, if the volume of operations goes beyond the capacity of a single database server.
• To reduce latency, so that processes running in the Singapore office need not wait for responses from a database sitting in Toronto.

When it comes to reading data from databases, replication is an ideal technique, because all of the replicas contain the same information. However things get stickier when it comes to write operations, because we need to decide where those write operations are performed, and how they get transferred to other copies of the database.

The most common answer is to use master-slave replication, in which a single database (the “master”) is considered authoritative. Any changes to the data are performed exclusively on the master and then trickle down to all of the other “slave” databases via a transaction log. This keeps all the database copies (more or less) instantly in sync.

Unfortunately, if write operations are frequent, master-slave replication brings us right back to the problem that replication was designed to solve. The master database becomes a bottleneck in terms of reliability, throughput and latency, since every write operation is performed on it alone.

A more complex strategy is called multi-master replication, in which writes can be performed on any of the database copies, rather than on a single master. In this case, the copies share updates with each other in a peer-to-peer fashion in order to remain in sync.

This sounds simple in theory, but multi-master replication introduces a new problem because conflicts can arise. What if two copies of a database update the same row at the same time, then attempt to exchange these updates with each other? Both databases will notice that a conflicting update has taken place, and have to apply some agreed strategy for resolving these conflicts. And here things get pretty complex – see the docs for MySQL, SQL Server or Oracle for some examples of conflict resolution strategies. (I’m ignoring synchronous or so-called “eager” multi-master replication, in which all replicas must commit to a write operation before it can take place, because that turns every copy of the database into a bottleneck.)

So here’s where all this background is leading:

Wouldn’t it be nice if we could have distributed multiversion concurrency control, to prevent conflicts occurring in multi-master replication?

Well, yes, I imagine that would be very nice indeed. And I believe that this is precisely what blockchains do.

Blockchains as distributed MVCC

Let’s copy down a couple of sentences that I wrote in bold above:

If two transactions attempt to spend the same output, then only one of those transactions will ultimately be accepted. A blockchain acts as a unified mechanism to detect and prevent these conflicts across the network.

If two transactions attempt to delete the same row version, then only one of these transactions will ultimately be accepted. Multiversion concurrency control acts as a unified mechanism to detect and prevent these conflicts within a database.

These sentences are identical except for the bold terms. So here’s what I’m going to claim:

A blockchain provides distributed MVCC (with a few extra bells and whistles).

Let’s flesh out the comparison a little further. From the perspective of a blockchain node, the current set of unspent bitcoin transaction outputs forms a database, in which each row is a single unspent output. This is similar to the database of bank accounts we described earlier, with the minor difference that the balance of each account can be split across multiple rows, each of which is marked with the same account number.

A bitcoin transaction spends one or more of these outputs and creates one or more new outputs as a result. This is exactly like a database transaction which deletes one or more row versions, and creates one or more new rows as a result (recall that in MVCC that there is no such thing as modifying a row in place). The bitcoin blockchain ensures that a single output cannot be spent by more than one transaction. This is equivalent to ensuring that a single row version cannot be deleted by more than one database transaction.

Now before we get carried away, I’m not claiming that blockchains are a great general purpose technology for distributed database synchronization in a fully trusted environment. There are plenty of other technologies such as Paxos, Raft and Two-phase commit which perform the job very nicely. But I do believe that blockchains have a sweet spot, which can be characterized as applications where:

• We can accept a short delay between when a transaction is probably accepted and when it is definitely accepted. (This delay can be a matter of seconds rather than 10 minutes as in bitcoin.)
• Conflicting transactions should never happen if everyone is being honest and their systems are functioning properly.
• Each transaction modifies just a few rows simultaneously (otherwise our blockchain transactions will have an unwieldy number of inputs).
• The size of each database row is fairly small (again, to prevent our blockchain transactions ballooning in size).

All of these criteria are fulfilled by financial applications. The financial world is already used to delays (of up to 3 days!) between conducting a transaction and its final settlement. In terms of preventing conflicts, it has contracts and regulations in place to detect fraud, and the consequences can be severe. And the amount of data involved in each transaction is pretty small – think of the bank account example above.

So far, all I’ve demonstrated is that blockchains are yet another synchronization mechanism for distributed databases. Big wow. Things only get truly interesting when we consider the additional features that blockchains provide.

Blockchains beyond MVCC

A bitcoin transaction does much more than just point to some previous transaction outputs and create some new ones in their place. Even the simplest bitcoin transaction serves two additional purposes.

First, the rules regarding valid transactions contain some of the application logic for our account database. Recall that the total quantity of bitcoin in a transaction’s inputs must cover the total quantity in the outputs. Translated into database application logic, this is a rule which states that database transactions (with the exception of coinbases) are not permitted to increase the total quantity of bitcoin in the database. This kind of constraint goes beyond regular database stored procedures because it cannot be circumvented under any circumstances.

Second, recall that each bitcoin transaction output encodes the conditions under which it can be spent. For regular bitcoin outputs, this condition is based on public key cryptography. A public address is embedded inside the output “script” so that it can only be spent using the private key corresponding to that public address. If we consider this output to be a database row, what we have is a database with per-row permissions which are based on public key cryptography. Furthermore, every transaction presents a publicly auditable proof that its creator(s) had the right to delete/modify its prior rows. This (I believe) is a genuine novelty in database technology.

And again, it just so happens that both of these features are incredibly useful for financial applications. We like the fact that our database ensures, at the lowest possible level, that money cannot be created out of thin air. And we like having an incontrovertible audit trail demonstrating that every transaction was authorized by the holder of the funds which it moved. As discussed in detail here, we may also like performing safe atomic peer-to-peer exchange transactions (delivery-versus-payment in finance-talk), without even knowing the identity of our counterparty.

So where’s the token?

Of course, none of this is a coincidence, because bitcoin itself is a beautiful peer-to-peer financial application. Still, none of the above characteristics of a blockchain are dependent on the token at all. If we modify our “database” schema so that each row can represent multiple assets, rather than the blockchain’s native currency, then we can rid ourselves of that currency entirely. This leaves us with a blockchain as a way to achieve consensus and security in a peer-to-peer financial application for any class of asset.

Only one little question though: Who does the mining to generate this consensus? In bitcoin anonymous miners must perform expensive useless computations, and are incentivized to do so by the block rewards (and transaction fees) denominated in the blockchain’s native currency or token. Do we have any other options?

It turns out that we do. We can have a closed list of permitted miners, who identify themselves by signing the blocks that they create. Rules about distributed consensus (or “mining diversity” as we call it in MultiChain) provide a different way of preventing minority control of the blockchain, so long as you can accept that miners are pre-approved. Of course for bitcoin this is not acceptable, because part of the point is to permit anonymous mining, so there is no way to censor transactions centrally. But if, say, we had a highly regulated financial system, in which bitcoin’s model was inapplicable, perhaps we could accept a pre-approved list of miners after all? If we had enough of them, and spread them well enough between institutions, and had legal contracts with all of them, are they really likely to gang up and undermine the network they depend on, when doing so will land them in jail?

Epilogue

I hope I have demonstrated that blockchains without tokens do have some useful applications, even if these are very different from the bitcoin blockchain. Nonetheless one question remains:

Are these permissioned, token-free shared ledger systems really worthy of the name “blockchain”?

The short answer is: who cares? It’s rarely worth arguing about the meaning of words, because there is no right answer.

But to go a little deeper, let’s say I accept the premise that the bitcoin blockchain is the archetypal blockchain. In that case, what we should really be asking is:

Are these shared ledgers similar enough to bitcoin to merit the name “blockchain”?

My own personal view here is yes. Because they share a huge number of technical similarities, even while they differ in the permissions model and economic incentives. And most importantly, because they both generate consensus in a distributed database via a chain of blocks.

Thank you for reading.

You can follow me on Twitter here. See also: Delivery versus payment on a blockchain.

Here are a couple of other pieces worth reading on this subject by Piotr Piasecki and Dug Campbell.