Big data is critical to the ongoing progression of financial services, but many businesses in the industry are failing to utilise its full potential of this industry.

Many financial businesses have harnessed significant volumes of new data as a result of adopting digital technology, but most have barely begun to gain the full value from the information available to them. According to a study by Seagate and the International Data Corporation, the average business collects and measures approximately 24% of the operational data available to it. There are several reasons for such a low figure, ranging from hesitancy regarding user privacy and regulatory compliance to challenges acquiring all the appropriate material in the necessary format for assessing data.

The first major challenge facing financial services businesses looking to make better use of their data comes back to their data ecosystem. The second challenge relates to the available talent, and the third is data management. The sheer capacity of the cloud is tackling the first challenge. Cloud-based technology has also created further possibilities for data and analytics teams, but industry experts believe that despite the advances in managing and governing data, there are still opportunities to improve the process. Most of the focus has moved from cross-functional platforms that allow businesses to make the most of their data.

In terms of the talent challenge, industry experts highlight how the sector is struggling to attract more people with the appropriate IT skills. Nick Broughton, the CIO at Novuna, believes that technology alone isn’t capable of generating all value; the industry also requires data-capable people with new and innovative ideas. Data science skills especially are critical to delivering real insights from the large volumes of data we have available. Attracting, retaining and growing internal talent around these important skills is another challenge when the demand in the market is rising.

A recent survey of financial businesses discovered that over 80% had struggled to hire data scientists, despite average annual salaries often exceeding £100,000. Over 25% of respondents suggested they didn’t have the necessary skills needed to achieve their commercial requirements. Considering these figures, the industry will need to become more flexible with employment policies and practices if it wants to attract and retain the data specialists it requires. Based on the high demand, these data professionals have the power to dictate the terms. Many people prefer to work remotely and flexible hours, so employers must be prepared to meet these demands.

Some businesses are going to greater lengths to create a reliable pipeline of potential talent, building networks with data science communities and creating special training programmes.

In terms of the other challenges, businesses are working hard to put more governance systems in place. The overall aim is to create a holistic view of their services and customers using data gathered and managed in real-time. It’s important to make tools accessible to everyone in a business, not just a handful of IT specialists. One opportunity emerging from this work is that it enables businesses to create new ways of meeting clients’ expectations. Ultimately, delivering the best customer experience is pivotal to any data-driven plan.

Creating a personalised service is one important area of development, but the potential of integrating augmented services, and combining data insights and human interactions is exciting for some businesses. For example, by using a range of data, companies can create investment signals and intelligence that managers can use to improve their relationships with clients. With the help of AI tech such as machine-learning systems, client-facing employees can determine trends that wouldn’t be possible. New tools can help customers understand their financial health and the risk profile of investment opportunities available to them.

The future is likely to involve clients using tools for experimentation. AI tools can show how investments could change over time but we find ourselves in a stage where regulation is failing to maintain pace with technological progression. Financial businesses must be cautious in their approach toward AI-enabled services, to maintain customer trust. Specialist positions such as data-ethics managers ensure an approach to AI is transparent, unbiased and capable of delivering the best outcomes for clients. Financial companies are already using chatbots and other virtual technology supported by natural language processing. Natural language processing also allows businesses to perform automated searches of various sources which can identify certain problems such as profit warnings or greenwash. Applying alternative data sources like smart sensors for climate-focused investments is likely to become more common in the future.

As the world moves toward data as a product, we need to start developing services with the data they serve. Making data services simple, secure and governable will be critical to the success of plans.