Data-Driven Organizations

A new class of organization that uses data and analytics to find business solutions is emerging in more and more areas of economic activity

The information and knowledge revolution is creating a new form of economy right before our eyes. Within just the space of a single generation, we will switch from an economy grounded in an industrial model to one based on information and knowledge and out of this will emerge a new form of enterprise. This new form of enterprise will be based on services, it will be managed through information platforms and coordinated by advanced analytics, with the function of people in this new enterprise primarily moving into the domains of design, innovation and all kinds of knowledge activities. The rise of this innovation economy is set to change the nature of work in fundamental ways in the coming decades. The advent of smart systems and cognitive computing raises many fundamental questions about the relationship between computers and humans and challenges us to reinvent organizations based on information and knowledge production.

In this section, we are going to be talking about data-driven organizations. Which is a term we are using to implied organizations that base themselves on data and analytics as the foundation to how they operate and make decisions? The next generation of organization won’t just do analytics, they will be analytics, whether we are talking about those operating in the realm of the smart grid, personal health data, IoT, insurance or real estate; analytics will be first and foremost, to what they are and do. In research by the consulting firm Bain & Company they note that “big data and advanced analytics are creating profound new opportunities for businesses, yet we found that only 4% of companies are able to combine the right people, tools, data and organizational focus to take advantage.”

With the rapid developments of advanced analytics and big data over the past years, we are starting to see the importance of the analytics revolution that is underway. But to fully appreciate the scale and depth of this transformation that organizations will go through it is important to link it to broader processes of change. That broader process of change is what we call the information revolution, that transition that advanced economies are going through as they move away from the centuries-old industrial economic model to the new form of information, services and innovation economy that is emerging. All organizations operate within this broader economic social context and when the context change it is required that they too change their structure and function, the data-driven organization is this new form of organization that is relevant for the information economy.

New Organizations

Complex analytics will enable us to create new forms of organization. With the advent of globalization, the internet and cloud computing we are in the process of developing ever larger networked systems of organization that span across whole industries and around the planet. We find that our systems of organization are being converted into large automated networks that process information to coordinate themselves, our capacity to manage those networks will depend upon our capacity to master complex analytics. Technology is driving the natural scaling of the activity beyond the institutional boundaries within which we have been used to thinking about it. The basic story here is that what used to be vertically integrated, oligopolistic competition among essentially similar kinds of competitors is evolving, by one means or another, from a vertical structure to a horizontal one. It’s happening because transaction costs are falling drastically. The dropping of transaction costs weakens the glue that holds value chains together and allows them to separate. This allows for scalable networked communities to replace conventional corporate production. Big data and advanced analytics are a key component in the development of these ever larger networks as they require a much more complex fluid and dynamic form of coordination.

JD, one of the largest e-commerce platforms, uses data science and machine learning to manage its huge supply chain network

The problem that we are having the level at which we can process information. With our existing institutional structures that are based around closed centralized and hierarchical organization, we are constrained by the amount of information that the central members can process, which is limited it doesn’t really scale to very large complex systems. Information technology enables massive amounts of people to collaborate within open networks like never before.  Unlike physical products knowledge is nonrivalrous, new discoveries differ from other inputs because they are nonrivalrous and fuel further innovation and this can be the foundations for collaboration instead of competition within an innovation economy. It is increasingly becoming recognized that organizations can thrive and serve their users better as participants within open networks. Unlike products that are largely zero-sum rival goods – meaning they can only belong and be consumed by one person before they have to be produced again – knowledge and information are positive-sum meaning they can be accessed and utilized by many people at the same time. This means the locus of production can shift to much greater investment and collaboration around producing the initial item and then it can be duplicated and exchanged many times. For example, instead of having thousands of lectures around the world giving the same introductory course on economics, that course can be produced once with the best teacher and then distributed via the internet.

Learning Organizations

The organizations that win this game will not be those that are simply most technically competent it will be those that are able to integrate people and information systems effectively. Those that are able to build a full stack of not just big data and advanced analytical capacities but or able to use that information in context and are able to integrate it with human knowledge and insight. An information economy implies an innovation economy, when we commoditize physical processes and automate information processing the function of people has to move up to the level of ideas and innovation. The move into the information age also engenders a changing dynamic in knowledge and knowledge production as knowledge becomes no longer confined to books and academia but begins to flow in all directions, it becomes increasingly recognized as the critical asset and resource that flow throughout the economy and society. A learning organization is the business term given to a company that facilitates the learning of its members so as to be capable of continuously transforming itself. The key shift from a traditional organization to an innovative one is the move from a fixed structure designed to maximize efficiencies to one that is designed to respond to new ideas. A knowledge organization works with ideas, creates new ones and transfers them around the organization, it keeps them alive but it is also able to act on those ideas. In a knowledge economy ideas are something real and the organization is a means for executing on them, not a fixed structure. The ability to work autonomously and be an active component of a network becomes paramount in the new economy.

This kind of worker requires higher order thinking skills. Just as previous technology revolutions drove the need for members to have greater thinking skills and knowledge so too does the innovation economy. In an agrarian economy, most people needed a ‘know-how’ type of knowledge. The required knowledge was very practical and they learned this knowledge by participating in the everyday life of their community without the need for formal education. In the Industrial Age economy, people needed a more abstract ‘know-what’ kind of knowledge, they needed a basic stock of general knowledge about the world and basic logic with which to reason in an analytical fashion. In a post-industrial economy, members need more than a stock of “know-what” knowledge they also need to be able to generate new knowledge, they need to be able to think.1 Thinking requires developing a systematic set of intellectual capabilities. It requires a more abstract set of conceptual skills, the intellectual infrastructure of critical thinking, systems thinking and creative thinking to synthesize information and process it into new knowledge. Just as innovation is a combination of abstract knowledge and its application, so also an innovative organization needs not only these workers with abstract thinking skills but also it has to have in place the processes for the whole organization to execute on new knowledge, receive feedback and iterate through an evolutionary process.

1. (2018). Shifting to 21st Century Thinking » The Knowledge Age. [online] Available at: [Accessed 13 Feb. 2018].