While businesses are recognising the value of data science and its ability to enhance business revenue, implementing and scaling data solutions across a business continue to be a challenge. A recent survey of data and analytics professionals suggests that developing a positive business culture with employees is a major factor that influences the success of data science. Led by DataIQ, a memberships-driven forum for the data and analytics community, the survey covered a panel of leading professionals across multiple sectors and companies in the UK.
The survey findings showed that 1 in 4 businesses believe data science will impact their top line revenue by over 10%. The results also indicated a continued challenge with company culture, suggesting a positive, model-focused culture is difficult to develop and still needs to be focused 0n. Approximately 40% of respondents want more clarity of the needs from stakeholders and a further 38% understand the necessity to train business users in data science solutions. Furthermore, another 32% believe there is a need for a more positive relationship with their stakeholders.
Nick Elprin, the CEO of Domino Data Lab believes that most businesses begin their work in data science by employing several data scientists, but ignore the importance of developing a model-driven culture that corresponds with their needs and the needs of business users. Mr Elprin believes the survey highlights the impact of not having a positive culture has on identifying proper use cases, creating expectations and generating quantifiable impacts on the business. Recognising these challenges is vital for businesses so they can create the right path and scale data science solutions successfully.
Additionally, 40% of respondents indicated that limited understanding or support for data science in business is regarded as a major challenge. The survey suggested that 1 out of 3 businesses stated that the conflicts between IT and data science remain another challenge. Even businesses that regard their adoption level of data science and analytics as advanced are not necessarily free of cultural conflict. Other findings from the survey included:
Over half of all organisations believe they will experience an uplift of under 5%, indicating that the failure to implement data science contributes to lower expectations.
1 out of 5 companies is experiencing a significant competitive advantage via applying data and analytic tools in their organisation.
A total of 67% have assigned their data scientists together to form a core function, rather than dispersing them throughout the business.
1 out of 3 organisations believes they require months to get their models into production. This needs to be considered because market changes are constantly changing and models that utilise outdated data will not generate valuable recommendations.
1 in 10 businesses has implemented an enhanced automated monitoring model that creates proactive alerts when models are deteriorating. Data Scientists can then examine potential issues before they have any major impact on the business.
David Reed, the Knowledge and Strategy Director at DataIQ believes that for data science to provide real value, a positive culture needs to be developed, enabling stakeholders and data science professionals to collaborate and share common goals. Mr Reed explains that the survey results suggest that this is easier said than done. 4 in 10 businesses identifying limited understanding or support for data science in their organisation as their business challenge. This presents a cycle that results in 1 in 8 businesses failing to generate a compelling use case for data science.
In the last few years, Business Intelligence has progressed into a driving force for business development, and with the impacts of Covid-19 accelerating further progress, we are likely to see BI transform into an essential part of an enterprise.
This year we are likely to see BI and AI collaborating to develop augmented analytics and augmented data systems. With augmentation, businesses will be capable of determining data types automatically and gathering information directly from the source. With AI and BI working together, jobs will be easier and quicker to complete. Users will be able to apply natural language techniques to ask questions and generate key insights in a matter of seconds. With the support of AI, industry experts believe we will see more users with less technical experience gaining more analytical insights by 2021.
Storytelling or Automated insights will enable support businesses further throughout 2021. Rather than spending time understanding insights which will impact their business, innovative storytelling will emphasise specific findings. The system will monitor changes and create a narrative that is easy for the user to understand and to detect key findings. Industry experts suggest that while storytelling will have a direct impact on businesses, it will need to be integrated in a manner that enables insights to be delivered from regular business tools. Enabling storytelling to be available in this format i.e. an email or a chat will be vital in driving the progression of storytelling BI.
Embedded and Self-Service BI are anticipated to grow considerably during 2021. Embedded analytics enables end users the potential to manipulate their data, allowing a business to produce advanced analytics, rather than just traditional static reports. The progression concerning the adoption of embedded BI has grown over the years and interactive filtering will become more commonplace in the future.
With embedded BI, users will be able to generate actionable insights and business applications with analytical tools will become more integrated. Self-service BI will provide businesses with AI solutions to automate search engine reporting, insights, unified analytics and more. It has become even more important for businesses to be able to connect to all data points and have a holistic view of their business insights.
Cloud and Mobile BI
With the pandemic and surge of remote working driving cloud adoption, the connected cloud will be a major influence on BI this year. Businesses will be seeking vendors that provide cloud-based BI and other vendors that offer self-managed cloud networks or private cloud servers. Mobile BI will likely become more prevalent in 2021, enabling access to insights to work alongside the ability to work from anywhere.
The demand for cloud and AI-enabled BI is anticipated to surge, as remote working continues to be the norm for many businesses this year. Regardless of the working conditions, the benefits of BI in supporting businesses remain ahead of the curve, to predict market changes and improve services will be hard to ignore. Senior leaders should focus on investing or enhancing their digital and market intelligence plans.
Augmented Analytics – combining artificial intelligence (AI) and analytics is one of the latest innovative developments in the data analytics industry. For businesses, data analysis has progressed beyond simply hiring data scientists, to incorporating smart technology that provides clear insights that directly influence decision making, thanks to AI.
Augmented Analytics or AI-driven analytics, supports organisations in determining hard-to-find patterns in large data sets and reveals patterns and actionable insights. It utilises a combination of analytics, machine learning and natural language generation to automate data management processes and help with more complicated elements of analytics.
A study by Gartner suggests that by the end of 2024, 75% of businesses will rely on AI and generate a forecasted 5x increase in streaming data and analytics services. The potential of AI will enable businesses to enhance their internal data-driven decision making while enabling all members to have easier access to the data. AI can save data scientists, analysts and other data professionals considerable amounts of time spent on repetitive manual tasks.
AI benefits to analytics
The progression in the AI industry plays an important role in making businesses more efficient and capable with the support of automation. With the support of ML algorithms, AI can automatically measure data and reveal hidden patterns and insights that can be applied to the decision-making process. AI automates the report generation process and enables data to be easier to understand by applying Natural Language Generation. Using Natural Language technology means AI enables all members of the business to discover the information and extract important insights from data efficiently.
While traditional BI utilised rule-based systems to generate static analytical reports augment analytics uses AI techniques to automate data analysis and visualisation. Machine Learning uses the data to determine trends, patterns and relationships between different data sets. It can apply past events to make the necessary changes.
Augmented analytics can apply user queries to create answers in the text and visual formats. This process of data generation is automated and allows non-technical users to understand data and detect insights.
Business Intelligence can support better business decisions and improve ROI by simply gathering and processing information. An efficient BI system collects important data from various sources and generates actionable insights. Augmented analytics will improve BI and support businesses in several ways:
Enhance Data Preparation
Data analysts generally spend a lot of time extracting and cleaning up data. Augmented analytics eliminates the time spent on these processes by automating time-consuming tasks and generating valuable insights that can be applied for analysis.
Automated Insight Generation
Once data is ready for processing, augmented analytics can automatically generate insights. Using ML algorithms, it can automate processes and generate insights that generally take much longer to be completed by data scientists.
Efficient interaction with data
Augmented analytics will make it simpler for users to make queries and communicate with data sources. With the support of NLG, it can convert natural language into machine language and then generate useful insights in a much simpler language. This allows businesses to ask questions regarding their data and get answers in real-time.
Enable an entire business to use analytics
The ability to query data makes data much more accessible for everyone in an organisation to use analytics products. Businesses no longer necessarily require data scientists or technical professionals to use BI tools and understand their data.
The level of complexity and scale of data now being generated and used by businesses has reached a level that is simply not manageable by humans. Organisations are embracing the development of AI in analytics to manage data and improve overall processes. Augmented analytics is enabling this movement and applying it with BI platforms is allowing businesses to interpret data quicker and as a result, enhance their operation and make their data teams more effective.
A coalition of health and technology businesses, including Microsoft, Salesforce and Oracle are working together on a project that intends to make it easier for people to access their Covid-19 vaccination records digitally. As more people begin to get vaccinated against Covid, they may need to have proof that they are vaccinated so they are able to return to work, school, or travel, and having easy access to a digital vaccination record could help with this. The program is referred to as the Vaccination Credential Initiative (VCI).
The vision of VCI is to empower individuals to obtain an encrypted digital copy of immunisation credentials to store this information in a digital wallet of their preference. VCI explains that they are working to make the credentials using SMART Health Cards criteria, which enables people to store immunisation or laboratory results in a digital wallet.
A recent press release from VIC doesn’t include a timeline as to when organisations administering the vaccines will be able to generate these records, so it remains a little unclear when one can be added to a digital wallet. Residents in the US are already receiving paper cards recording their vaccines and it’s a little unclear how these records can be transferred to a digital version.
Another potential challenge is encouraging health facilities to participate in the project, as some provider may have the ability to incorporate these services into the vaccination process, but others may not. There are other ethical concerns about whether a person that can prove they are vaccinated should be given more freedom than others that are not vaccinated.
VCI isn’t’ the first coalition to consider a digital Covid-19 vaccination record system. Other projects are being tested based on existing vaccine documents required by some countries for entry, such as yellow fever or polio.
The significant rise in remote working hasn’t only increased the opportunities for hackers to exploit but has also heightened the need for more cybersecurity specialists.
More people are beginning to appreciate the importance of cybersecurity in the business world. According to non-profit organisation ISC, a specialist in cybersecurity training, over 3 million people are needed in the cybersecurity industry worldwide, with over 20% of businesses showing a shortage of available cybersecurity talent.
According to cybersecurity experts, the most feasible way to close this talent gap is to employ and retain more women into the industry. At present women make up approximately a quarter of employees in the cybersecurity industry. The cyber industry is historically a very male-dominated market. Some organisations have focused on involving women at a young age into the cyber industry to improve this trend. For example, the UK government introduced the CyberFirst Girls Competition back in 2017. The main challenge, however facing recruiters is retaining female employees. Emily Staph, cybersecurity leader for the US at PwC believes there is a lot of available talent out there, but businesses need to create a culture and the opportunities to ensure they keep women in the industry.
Aside from closing the talent gap, hiring and retaining women could support the further progression of cybersecurity in various ways. According to Ms Stapf, many women approach and think differently about business and balancing tasks and more importantly identifying potential threats. Cybersecurity group Attivo Networks is promoting the development of creative projects which could temporarily provide subsidies for female graduates or women looking to change careers. The program would enable women to gain valuable skills and experience that could lead to a career in cybersecurity. Industry experts believe the industry could be more welcoming to women, especially those willing to learn and progress in the industry.
The lack of available cybersecurity professionals, regardless of gender, is partly a result of businesses focusing on specific areas that drive revenue and pushing aside cybersecurity processes. Cybersecurity evolved as an added discipline of IT teams but has evolved into something critical for risk management in corporations, yet many businesses are yet to recognise this. A US-based survey by IBM security stated that cyberattacks cost targeted businesses an average of just under $4 million.
While businesses may recognise the need for a cybersecurity team, many believe that finding the ideal candidate is difficult because of the technical skills required and the necessary qualifications. Cybersecurity roles tend to offer secure and higher than average earning careers for those with the knowledge, experience and technical skillsets.
Despite a cut back in employment during 2020 due to the uncertainties surrounding the pandemic, businesses are predicted to increase their spend on cyber teams this year. The last year has been pivotal in terms of the number of security incidents and the need to invest in new technology and the right staff. Industry experts predict a surge in hiring in cybersecurity throughout this year. The industry has evolved from its days of being a predominantly reactive technically-minded industry to an innovative and forward-thinking part of a business. As we experience a surge in cybersecurity, industry experts believe we will see more women move into the industry, facilitating the rising demand for new talent.
Aside from the extended coverage of the pandemic lies the continued threat of climate change and the challenge of implementing effective measures to reduce emissions and keep our global temperatures to the levels set in the Paris Agreement. Despite short term reductions in emissions due to the impact of Covid-19, reports in 2020 highlighted predicted rises in emissions and temperatures in the long term, unless we take drastic action. Some analysts believe that big data could be critical in rising to the challenge.
A connection between big data and climate change has been acknowledged for some time, but until recently, applying big data to climate science and measuring the impacts of pollutants and other gases has been relatively limited. There is increasing support for big data, and it’s potential supporting our transition towards a green economy.
There is a simple rationale for big data. For all markets to operate more efficiently requires information and data on their products. At present, commodity production businesses have huge amounts of data on their products and where these materials are created. Despite having all of this information, very little is passed on to investors and even less to the end-user. By making this data available, means both investors and customers would be capable of making better investing and purchasing decisions, which could gradually result in a switch to greener products.
For example, the petroleum extraction industry is very technologically advanced, and businesses gather significant amounts of data on each barrel of gas extracted and sold. At present, no information is available to investors in the stock market. No details are provided on where the oil is purchased and sold, which is crucial when some refinery procedures are far more detrimental to the environment. Investors and customers have little knowledge of these details, and consequently are unable to make an informed decision. Today, businesses, investors and consumers are more aware and conscious of our environment, purchasing choices, and the impact of products on our environment is critical.
In reality, we have all of this data available and the information to enable investors and people to make more informed decisions. Data has been collected and stored by extensive IoT networks spanning nearly every industry, but at the moment, this information remains in the networks and remains unused.
The future challenge
Improving this information connection and increasing the availability of data to customers and investors will be a challenge. Many production businesses consist of the necessary technology, but this may not be the case for everyone. Small-scale producers in developing nations would struggle to utilise this data, with limited data systems and lower access to the internet.
On a more positive note, businesses are working on leveraging big data to transition to greener markets. For example, carbon credit registries in North America are a fine example of this in action. Another example is the systems being created by Oxy Low Carbon Ventures, focusing on creating a world where carbon credits are traded just like commodities.
Data can make powerful changes and has proven already to be capable of transforming industries for the better. For example, the music business experienced a dramatic industry change in previous years, and this movement was predominantly down to customers having more information. In the late 1990s, CD sales had peaked, and in this year Napster launched downloads. This was the beginning of a digital music revolution, sparked by the fact that customers had more information about the music they were downloading. In terms of the oil industry, big data could equip investors of the future with detailed information on the oil extraction process and environmental impacts of this. For businesses and investors actively seeking to focus on their green credentials, this level of information could make a considerable difference.
It may not be in the interest of large manufacturing and mining businesses to make their data widely available to the public but climate policies and added pressure from investors and consumers is driving a big data revolution in industry. We are demanding more information and are aware the data is available for us to utilise. Businesses should be looking to investors and consumers and using this data to work together to improve and transform the impact of industry on climate change.
Finance transformation may not necessarily have to be a response to disruption. There are examples of finance teams implementing changes to their team, processes and technology to deliver a more effective approach towards business planning. They have focused their efforts on implementing data solutions and transitioning from a finance-based approach to planning and reports to a more innovative and intelligent process. These businesses were ahead of the game and had recognised the importance and value of implementing these systems before the disruptive events of the last year.
Disruption can, however, accelerate change and is why, as we move into 2021, we are experiencing a higher rate of businesses adopting transformation within the finance-based plans. Whether or not a business has recognised the benefits of transferring to an integrated multi-functional planning system, and possibly initiated some measures to drive this, the changes in the industry and continued uncertainties of the last year drove plans forward, and as a result, prioritised transformation for 2021.
What is needed to enable a long last and successful planning move?
Businesses adapting and moving towards a more holistic planning model recognise what needs to be done to grow and remain resilient to changes. They understand that innovative businesses utilise predictive and forecasting technology to monitor the latest trends in their organisation and the wider industry. They embrace analytical tools within their planning procedures, allowing them to measure what is going on, and more importantly, why. By combining their overall strategy, operational and financial plans, they can extend the impact of performance management and increase profitability by applying their focus on what works best.
The most effective planning transformation measures tend to have several things in common:
They are initiated with the best level of support
Transformation requires support and momentum to develop and expand plans further. This is why the most successful planning systems begin with a senior-level leader promoting the necessity for change, supporting this and ensuring the business remains focused on this goal. The success of business change depends on collaboration, ensuring a business is working together to enable strategic development.
A focused and strategic approach
Transformation is a challenge and time consuming, and so implementing a staggered approach to planning is optimum. Whether you are striving for an ongoing planning system or regular interval plan, it’s complicated to achieve this without a staged plan. Businesses need to consider where you intend to go, the measures you need to put in place to get there and then focus on developing an approach that will enable you to achieve these goals.
Adopt and utilise technology
The progression in technology means the skills and time required to transform planning systems have reduced. Intelligent data streams are widely available, and BI reporting tools have excelled enabling businesses to perform detailed and more insightful analysis. None of this will be possible, however, if business continues to rely on traditional legacy systems or spreadsheets. Predictive analytics, AI and Machine Learning can deliver vital insights and alleviate time spent on forecasting, but only when a business has the necessary foundation of technology and data in place.
Transitioning from finance-led planning to a more integrated approach that combines operational planning and reporting will drive further agility and resilience needed in a business to remain on track towards achieving your goals and being prepared for potential disruptions that lie ahead.
The growth of data has enabled businesses to understand large volumes of data and generate valuable insights. Companies that rely on this information to execute important decisions and continue to improve their products and services. The data industry continues to progress, and ‘dark data’ offers the ability to take it a step further.
Harnessing dark data is vital for businesses, particularly those in the fintech industry looking to grow and remain resilient. Dark data refers to information collected as a by-product of regular business activities but isn’t directly used to create additional income. The data gathered in these processes may not necessarily be the target, but additional information could still have some value.
Dark data represents the information that is pushed aside but could be vital in understanding customers and overall market conditions. This could include anything, from internal data such as email conversations and financial statements to external data sources, like customer profiles. Interpreting these data sources can enable businesses to gather new insights and intelligence that could greatly influence their decision-making process.
As the volume of data increases, so does the volume of dark data. Many businesses, however, are ignoring this source of information and the potential it could have on their overall performance.
Harnessing this data and transforming it into usefully structured information can be challenging. Automation and other emerging technologies have enabled businesses to manage this process more effectively. With further support, access to dark data sources can enable teams to gather more insights into their business and anticipate patterns and industry trends. As a result, the business will be more informed and prepared when making important business decisions. For example, a major hotel chain in Europe measured its internal dark data, which in their case was Wi-Fi usage data. The company used the data to identify and solve potential issues regarding waiting times to check-in and out and ways to improve staff allocation across the hotel. Dark data supported the business in optimising their resources, while at the same time improving the customer experience. Research by Accenture indicates that dark data analysis has assisted insurance companies in generating vital insights and resulted in a profit increase of up to 21%.
The potential benefits of dark data can be applied to the fintech industry too, enabling businesses to deliver stronger analytics and potential business opportunities. Dark data provides a means to remain competitive and profitable and this year all businesses will be focused on resilience. The difference between businesses that continue to progress and those that struggle will be closely related to their alignment with market conditions and customer expectations.
Dark data is a valuable tool in delivering resilience and provides a great opportunity for fintech to generate, informed, data-driven business decisions. Leveraging dark data within a business model can enable CFO’s to focus on generating new business opportunities and a considerable edge over other competitors.
AI and automation became essential for many business’s efforts to manage the impacts of the pandemic. How will these innovative services continue to transform business strategy this year?
At the beginning of 2020, the uptake of tools like AI and automation was increasing but at a cautious and steady rate. In just a few months, the rate of growth changed significantly. The global pandemic became a catalyst, accelerating the uptake of new services and changing business direction, including increased use of AI, Analytics and Automation.
The move to working remotely facilitated a major business change. This shift to remote work came at a time when there was a major increase in demand for additional customer service and support. As a result, AI and automation became a top priority in 2020 for many enterprises. 2020 witnessed a big increase in the use of automated chat bots, as businesses actively looked for ways to automate regular interactions or reduce the volume of manual tasks. This year we will inevitably continue to see businesses exploring ways to implement automation into their business and alleviate the pressures of customer demands and expectations. The surge of technology solutions has enabled most companies, big or small, to implement these types of services.
Another trend of 2020 that is likely to continue throughout this year is edge intelligence. Edge Intelligence refers to processes where data is collected, analysed and insights are generated close to where it is captured in a network. Today, smart edge solutions offer real-time insights by assessing data at the edge itself. Edge Intelligence is generating a lot of excitement in the industry, where AI/ML technologies come together with the cloud. Edge Intelligence enables systems to make decisions on locally generated data, instead of sending it to a centralised cloud or on-premises server. The ability to integrate AI and ML on the edge is a game-changer, according to many industry experts, because it can perform on the collected data and generate decisions before any data is moved to the cloud.
With a major shift of services to the cloud in 2020, a response that was driven more so by the impacts of the pandemic. Industry analysts believe edge intelligence will complete the shift to the cloud. Technologies such as 5G combined with artificial intelligence will only be capable of enhancing projects completed at the edge. After a year where remote working became the norm, this year seems a suitable time for edge intelligence to become more prominent.
Another trend expected throughout 2021 is the rising adoption of augmented analytics, a set of technologies that utilises machine learning to make data management and analysis simpler. For businesses, this means their workforce will be capable of effectively applying analytical tools without necessarily having to send analytics requests to data specialists. Data analysis is becoming more important in business and as a result the tools have become easier for employees to use. Technologies like this will be critical throughout 2021 to monitor and measure business performance and determine the best path and strategy to take this year.
Key trends for this year
AI Rate of Adoption
Gartner has forecasted that nearly 80% of enterprises will make a move from testing to operating AI by the end of 2024, creating a surge in streaming data and analytics infrastructure. 2021 will inevitably see many businesses make the move from testing the potential of AI, to integrating it into their performance plans.
Rise of Data Stories
Data stories will continue to become more prominent and are predicted to become the most popular way of consuming analytics by 2025. A large portion of these stories will be automated via augmented analytics. AI and ML processes are becoming more common in BI platforms. The traditional dashboard requires further manual work to determine the insights, but data stories provide the information without requiring the user to perform their analysis.
Increase in Decision Intelligence
Larger businesses will have dedicated analysts for decision intelligence, a practical service that incorporates several decision-making techniques. Decision intelligence combines conventional techniques with more advanced solutions like AI and Machine Learning.
Cloud will dominate
Public cloud services are forecast to be essential for 90% of data and analytics innovation by 2022. Cloud-based AI is forecast to reach a level five times higher by 2023 compared to 2019. This trend began before the pandemic, but the impact of Covid-19 has accelerated the rate of growth.
Integration of Data and Analytics
Gartner believes that non-analytical services to develop to incorporate analytics over the coming years. By 2023, 95% of Fortune 500 companies will include analytics governance into their wider data and analytics governance plans. By 2022, approximately 40% of machine learning development and measurement will be done with products that do not have machine learning as their primary goal. Gartner explains that analytics and BI providers are widening their data management capabilities and suggest that there will be more convergence soon.
A significantly challenging 2020 has left many questioning what will come next in the technology market. Many data and analytics experts from leading businesses such as Qlik, Cloudera and SAS have highlighted their own predictions for the year ahead.
One of the key areas and a popular discussion point is the prediction surrounding AI and Machine Learning. One area discusses how AI will continue to become more integrated with industry, more accessible, affordable and refined. Industry experts believe ML and AI will become more accessible to a wider range of businesses. While AI has largely been viewed as a tool generally applicable to data science experts, industry predictions suggest that this is changing and having these skills doesn’t necessarily mean you can’t utilise the advantages of AI. Other analysts expect the overall economics of AI to continue to improve, along with its accessibility.
Ryohei Fujimaki, founder of dotData reiterates this prediction, stating the automated machine learning will enhance AI accessibility for non-data scientists and enable AI to go beyond predictive analytics and generate valuable insights into other trends and events that may have previously been overlooked.
Analytics and the challenge of the pandemic
Many of the discussed developments for this year view Covid-19 as a significant influence on technology predictions. For example, analysts believe more organisations will transfer their infrastructure to the cloud due to Covid-19 investments in MIL will rise rapidly.
When the pandemic transformed the global economy, businesses were forced to invest quickly into BI tools and data software to try and gain an understanding of what was going on and to make standard business decisions. Many businesses are having to make significant cuts in budgets to alleviate the impacts of the pandemic and maintain their core business functions. Yet, this year’s predictions believe that many organisations will sustain or even increase their investment plans into data science to enable decisions that could be vital in the survival of their business.
The pandemic has spurred accelerated demand for AI solutions and has raised the focus on the ethical use of AI. Data and analytical experts believe the focus will shift towards measuring changes in customer behaviour in real-time to deliver vital actionable insights.
Analytics can continue to influence how we manage the impacts of the pandemic, rather than being the other way around. Buno Pati, the CEO of Inforworks refers to it as a battle against Covid-19 and gaining access to vital data about the health industry will ultimately enable a more efficient response to the pandemic. Analytics will not only play an important role in approvals for the vaccine development process but will also be vital for planning the expansion of tracking distribution, measuring side effects and the overall effectiveness.