What is a smart data management platform

Managing information in a data-driven company with DMPs

Data is the oil of the future, but like oil it has to be refined before it can be used. To increase the competitiveness of a company, an information system is needed that clearly separates the world of data from the world of applications.

Data is everywhere, and there is no single way to treat it: some of it must be analysed in real time, while other parts can be analysed more calmly. Think, for example, of an alarm system or of Google’s web-page indexing.

Often data has to be cleaned, transformed and loaded into intermediate storage before it can be analysed. A data warehouse alone is no longer sufficient. Today it is necessary to use data lakes in order to handle structured and unstructured data arriving from different sources and to integrate open data. When the data is very large it must be aggregated using big-data technologies such as those made available by Apache Hadoop.

Even more often, thanks to new machine-learning algorithms, the relationships between data become so complicated and profound that they need to be organised into a knowledge base — for example by using a knowledge graph. The knowledge obtained in this way can also be navigated by artificial-intelligence algorithms based on neural networks or on logical rules (Neuro-symbolic A.I.).

A data management platform (DMP) is precisely the set of tools that allow data coming from a great many sources, both inside and outside the company, to be cleaned, federated, integrated and analysed. Thanks to DMPs, applications reach all available data through a unified approach, overcoming the limits imposed by architectures in which data sits in silos that cannot communicate with each other.

The platform becomes smart at the moment when it is able to understand the meaning of the data as well. Only then is it possible to connect heterogeneous data while guaranteeing the orderly growth of the information that supports decisions. In the figure below, the typical components of a Smart Data Management Platform are highlighted in orange:

Figure: typical architecture of a Smart Data Management Platform.

The meaning of data also changes over time; fortunately the semantic technologies developed over the last decade in the field of the Semantic Web have proved capable of supporting even the most radical changes of business model.

Data with a formal meaning has earned in the field the qualification of smart, and is today mature for use in business, as Google, Facebook, IBM, governments and very many start-ups already do.

The major cloud platforms (Amazon AWS, Google Cloud, Microsoft Azure, etc.) make available a rich set of tools “as a Service” for managing big data, data lakes and many other components of the data management platform. LinkedData.Center specialises in tools for the ingestion of meanings and for interfacing with smart data, with its product Smart Data as a Service (SDaaS).

An example of a completely “as a Service” implementation based on tools available on the Amazon AWS and LinkedData.Center platforms is described in the figure below:

Figure: an example of a Smart Data Management Platform “as a Service”.

With a smart data management platform it is possible to analyse the tastes and behaviour of one’s customers in order to offer them the best product, to identify new business areas and new trends, to optimise costs, and anything else the entrepreneur’s imagination suggests.

The only indispensable element is the data, whether it is company data (first-party data), data coming from partners (second-party data) or data from sources external to the company (third-party data), such as the open data supplied free of charge by governments or the paid data supplied by data providers.

A good smart data management platform is able to use them all, even in the presence of inconsistencies and differing quality among the data being processed.