Rethinking applications in the age of edge computing

The European Union’s GDPR directive represents an important turning point in the information society. Put very briefly, it states that data property is an inalienable right of the data’s producer, whereas before the GDPR data property was something tradable.

Today in Europe, in order to use other people’s data, precise authorisations must always be obtained, and those authorisations can be revoked at any moment. Beyond that, the IoT increasingly requires local processing of data, driving the edge-computing paradigm.

Projects such as SOLID and IPFS promise radical but practical solutions for moving towards a true paradigm of data distribution, seeking to restore the web’s original goal: the sharing of knowledge.

This vision, in which every person or machine has full control of their own data, contrasts with the centralised management of data used by most applications.

Many signals suggest that the demand to return to a genuinely decentralised web is gaining ground, both in the political and in the social world; yet today, even when applications declare themselves to be distributed (Wikipedia, for example), they in fact still adopt a centralised architecture for data management.

According to Tim Berners-Lee, “The future is still so much bigger than the past”. To be ready for it, we need to rethink data architectures, allowing applications to use information produced and managed by someone else — whether people or machines — and therefore outside our control.

Eric Brewer’s theorem (also known as the CAP theorem) states that it is impossible for a distributed data store to provide simultaneously more than two of the following three guarantees:

• Consistency: every read receives the most recent write or an error
• Availability: every request receives a response (not an error) — without the guarantee that it contains the most recent write
• Partition tolerance: the system continues to operate despite an arbitrary number of messages being dropped or delayed by the network between nodes

The CAP theorem is often misunderstood, as if one of the three guarantees always had to be abandoned. In fact the choice is really between consistency and availability only when a network partition or a failure occurs; on all other occasions no trade-off has to be made at all.

But in a genuinely distributed data model, in which datasets are not under direct control, partition of the data and network failure are ALWAYS an option, so the CAP theorem applies in full.

Dynamic caching is probably the only practical solution for addressing the problem of data distribution effectively, but as soon as data is replicated a trade-off between consistency and latency is created.

In 2010 Daniel J. Abadi of Yale University found that even when the system runs normally in the absence of network failures, one has to choose between latency and consistency. This is also known as the PACELC theorem.

What does all this imply? An easy corollary of these two theorems is that we must forget the illusion that functions return the same results when we give them the same inputs. In fact, the determinism on which much of today’s computing is founded is being called into question, and we need to start thinking in terms of probability.

Figure: from determinism to probability in distributed applications.

This already happens with search engines (the same query does not give the same result), or with social networks (you cannot see the same list of posts on your wall), or in most edge-computing applications. It is not a “feature”: it comes from technical constraints, but Facebook, Google and many other companies have intelligently turned this problem into an opportunity, for example by giving priority to paid advertisements.

If the edge-computing paradigm enjoys the success expected of it, all applications — business applications included — will have to face similar problems. The customer and supplier archive, for example, might be — or might be obliged to be — distributed.

Technologies and solutions such as IPFS, linked data and graph databases offer practical solutions for the caching and querying of distributed datasets, helping to resolve inconsistencies and performance problems.

Even though they cannot be considered a silver bullet for every need, they are indispensable tools for designing a new generation of applications able to survive data that is increasingly scattered across the network.

For a deeper look at the technologies available for building a distributed data architecture, see this link:

Smart Data as a Service