Home > Big Data

Disruptive Technologies

Big Data

Creating insight from the connected world

Download PDF

Updated on: 11 July 2017

The Disruptive Technology Executive Briefs are produced in collaboration with PA Consulting Group, combining cross sector technology, innovation and business design expertise. The briefs are intended as an easy to digest introduction to disruptive technologies, to help organisations understand how they could advance the Sustainable Development Goals and business performance. These overviews explore key features, examples of applications, potential positive and negative impacts, and how they may enable the new business models.

Overview

One of the major trends of our time is the exponential increase in data. The internet, social media and connected devices are just some of the sources of immense amounts of data which can be used to answer questions about almost every aspect of our lives and industries. Big Data is the ability to capture and analyse this data to create meaningful insights or actionable information. It can be used to answer questions, identify trends, spot anomalies or explore the information in more detail through processing both structured and unstructured data such as text, imagery and video.

The technology

Big Data relies on computers which can spread the storage and analysis of data across many low cost servers, often hosted in the cloud. There are many ways to analyse this data, ranging from traditional statistical techniques for testing hypotheses to novel approaches identifying objects in images or video data. Often Big Data solutions bring sets of data together in innovative ways to create unexpected answers and insights. These techniques are now more readily available and can be used in ways that make the most of new computing technology.

As well as the computational techniques required to manage this data there have been many advances in analysis and visualisation of the data. These have created new ways of showing information on standard web browsers opening up analysis and understanding to anyone with a personal computer or tablet

The potential

As more aspects of our lives become monitored and connected, it is difficult to imagine an area where Big Data will not make an impact. For example, utility companies are using their measurement and finance data to better predict maintenance routines. Financial institutions can predict risk and monitor fraud more accurately and retailers have used it very successfully to predict customer behaviour.  We are all only beginning to fully understand the applications of Big Data techniques and its ability to improve our wellbeing, communities and planet.

The barriers

The major risk with Big Data is the power it gives organisations to monitor multiple aspects of our lives and potentially try to control them. This ranges from the mild irritant of targeted advertising to mass surveillance programmes.

Some example applications...

Condition based monitoring

Aviation is one of the most safety-conscious industries, where companies are consistently striving to improve the performance of expensive assets. Both Rolls Royce and General Electric’s aviation business closely monitor the performance of their engines on behalf of their customers, both in real time to check they are working properly and also using historic analysis to improve performance.  Each engine can generate terabytes of sensor data every day, so preparing and analysing these high volumes to generate insights is a Big Data challenge. By getting it right manufacturers and their service teams can identify faults before they occur, understand the root cause of problems and improve manufacturing techniques to avoid problems re-occurring in the future.

Predictive policing

The Los Angeles and Santa Cruz police decided to take a more data driven approach to reducing crime. In conjunction with a company called PredPol they collected crime statistics and applied a mathematical model previously utilised for earthquake detection. The resulting tool predicts likely locations for crimes enabling police forces to better deploy their officers. Several forces around the world claim to have seen a measurable improvement in crime prevention as a result of this or similar tools.

Logistics optimisation

UPS delivers 17m packages and documents every day, shipping over 4 billion items per year using tens of thousands of vehicles. It uses Big Data techniques in a number of ways, one of which is optimising its fleet.  On-truck measurements and advanced algorithms help with route planning, engine emissions and predictive maintenance. Since starting the programme, the company has saved almost 40 million gallons of fuel and avoided driving over 350 million miles. 

80%

Percentage of data which is unstructured

Source: IBM

40

The number of Petabytes of data Walmart processes in a day

Source: Datafloq

9 billion

The number of connected devices in 2016 (excluding computers

Source: IDC

Advancing the Sustainable Development Goals (SDGs)

Big Data has the potential to advance many of the SDGs. Below are some examples of areas of application across a wide variety of sectors.

Read more

SDG 1 No poverty

Analysis of data sources including open data sources such as social media feeds to understand the impact of policies on poverty.

SDG 2 Zero hunger

  • Provide insights to increase farm productivity and enable more sustainable farming
  • Improve supply chain management and reduce food

SDG 3 Good health and wellbein

  • Already being used by pharmaceutical companies to look for new insights in their significant stores of data
  • Enable other technologies such as gene editing.

SDG 4 Quality education

Already being used in the US to analyse teaching/education data, to understand where it is most effective and update approaches accordingly.  

SDG 5 Gender equality

Support more effective policy development, through using Big Data analytics to provide insights that were not previously possible.

SDG 6 Clean water and sanitation

Sensor data enables the preventative maintenance of water networks, reducing costly leakages.

SDG 7 Affordable and clean energy

Big Data has the potential to drive significant efficiencies through enabling preventative maintenance and better demand forecasting

SDG 9 Industry, innovation and Infrastructure

  • Big Data is already being used in most sectors to monitor performance, improve marketing and identify efficiencies
  • The value of data is being increasingly understood and monetised  creating significant sources of wealth
  • For infrastructure, Big Data can be an important part of condition based monitoring when combined with sensor data

Barriers to adoption

For a new technology to gain acceptance it must be safe and reliable but, just as important, it must be supported by the right legal and regulatory framework or there is a risk of a public backlash that will limit its use.

Data usage and privacy

Challenges of rapid change

Skill shortages

Big Data fatigue

Read more

Data usage and privacy

As more opportunities become available to bring data together, careful consideration needs to be given to the legal, ethical and reputational implications of doing so.  For example this could include government departments sharing data but also companies using social media to make risk or commercial decisions (e.g. increasing a person’s insurance premium due to their social media behaviour).  Data privacy laws will need to keep pace, both at a local and global level.

In some cases the statistical techniques common in Big Data have led to accusations of racial profiling or of disadvantaging poorer sections of the community.

Challenges of rapid change

The Big Data industry is fast growing and changing rapidly.  Many startups meet niche needs but launch and close every day. As a result, there are growing product differences and selecting a set of providers to build a system that meets particular requirements could cause problems if the software changes or the suppliers disappear. To address this a group of interested parties and vendors have created the https://www.odpi.org/Open Data Platform initiative to enable a common computer code and interchangeability for certain Big Data technologies.

Skill shortages

As demand for people with data science and Big Data skills and experience is growing dramatically, businesses will see a marked increase in salary demands for high performing staff.  This can be minimised by retraining existing staff or fixing prices with providers (e.g. outsourcing or agencies).  As with previous analytics and visualisation trends, some will opt to use user friendly self service tools for analytics tasks.  Whilst these are initially more expensive they simplify the use of Big Data techniques, reducing dependency on expensive resources.

Big Data fatigue

There has been much hype surrounding Big Data and whilst there have been significant investments many projects have not brought the benefits organisations wanted to see. The focus has often been on the technology rather than delivering business insight that can improve data-driven decision making, a big part of this is knowing what questions to ask.

Many companies are still developing their capabilities as the products mature, people develop and lessons are learned. This can lead to delay and frustrate many business who can then view Big Data’s promise as just hype.

This can be tackled by taking small steps to demonstrate the incremental value of Big Data. These can include short term cloud computing based pilots that allow for faster response times and a more agile approach. It is always important to ensure the team developing this capability is comprised of both Big Data scientists and subject matter experts who can interpret and tune the insight being generated.

Technical Considerations

Big Data is one of the more established of the breakthrough technologies and so many of the early technical challenges have been addressed – however, as it continues to gain popularity, additional areas will need to be addressed.

Read more

The rate of development is rapid and constant as new gaps are identified by niche players that then add to the products available. It can be hard for suppliers to keep their versions up to date and supportable - proprietary versions often significantly lag open source solutions.  

Many of the emerging  solutions, particularly with leading edge capabilities, currently rely heavily on very technical developers that can use a wide variety of computer languages and syntaxes.  This can lead to a shortage of resources and in turn higher salaries.

Some of the new technology, such as Hadoop, are ideally suited to cloud computing giving companies rapid access to relatively cheap and flexible new capacity.

Security is another important and legitimate concern for many businesses and it is important that Big Data uses well established guidelines for securing internet applications. Big Data applications are often hosted on platforms provided by the US companies such as Google and Amazon.  In many cases customers and the wider public have shown concerns that data hosted on these platforms can be accessed inappropriately by these companies. Whilst these fears are unfounded it is important to be aware that these perceptions exist. 

Enabling new business models

Big Data is transforming business

Personalisation

Closed-Loop

Asset sharing

Read more

In 2014, Wired magazine said ‘Data is the new oil of the digital economy.’ Since then the dramatic increase in the amount of data being captured has continued. This gives businesses deeper and finer insight into customers, sophisticated intelligence about the environment they operate in, and new ways to control their internal operations. For many organisations Big Data has improved both the quality and time taken to make business decisions. This enables them to respond in real-time to customer requests and to take more innovative approaches. For example, it is making loan applications quicker and easier or enabling retail companies to combine social media, weather, finance and logistics data to optimise stock distribution.

The potential of Big Data touches all industries, from supply to demand, and increasingly across value chains and industries. Many of the large technology companies that have emerged recently have Big Data at the heart of their business model. Google successfully monetised their search systems and massive infrastructure through sponsored adverts and the ability to match advertisers and consumers. Uber and Airbnb in the transport and accommodation sectors leverage high volumes of usage data to analyse employee and customer behaviour to improve customer satisfaction and increase sales.

Some of these companies have also been at the forefront of forging new business models providing Big Data services. These include Amazon and their Elastic Compute service, or Google’s BigQuery cloud data analytics platform. Other new business models have emerged whose purpose is to create value from very deep analysis of raw data. For example there are many firms able to analyse website viewing so that each site maximises its chance to make a sale. 

By providing easy access to the tools and infrastructure needed, the large technology companies have opened up the world of Big Data to start-ups and innovators. This has also disrupted established organisations who no longer need to invest large amounts of capital to build data centers filled with highly sophisticated machines, but instead can use cloud based solutions and flexible analysis tools to provide powerful capabilities.

Big Data will enable a number of the disruptive business model levers identified on the Project Breakthrough website, specifically:

Personalised product and service

Big Data is a key technology in enabling services which are individually tailored to the user or to their current activity. Big Data techniques can identify what data is relevant to a given user in their current context.

Closed loop process

Big Data allows consumption and supply patterns to be better understood and controlled, improving efficiency, and reducing wastage.

Asset sharing

Big Data allows sophisticated insight into demand and supply to better match and promote asset sharing.

More examples...

GE Expands into industrial analytics

Pirelli use Big Data to improve manufacturing & resource use

When Big Data and satelilites collide

Get involved in the conversation

Download PDF

How will your organisation use these technologies? What are the opportunities, and the risks, for advancing the SDGs? Share your ideas with us.

Other disruptive technologies

Explore how disruptive technologies can impact the Sustainable Development Goals. Developed in association with PA Consulting, this section will evolve over the next month to look across 12 technologies ranging from Robotics to Blockchain.

Unmanned Air Systems

Next Generation Robotics

Internet of Things

Artificial Intelligence

Digital Agriculture

Gene Editing

New Realities

Blockchain

The Microbiome

Additive Manufacturing

Autonomous Road Vehicles

Related Content