In brief
- A clear and comprehensive data strategy is essential for organizations to effectively manage and leverage their data assets.
- Building this requires a thorough understanding of the data landscape and alignment with business goals and objectives.
- Effective data governance, including data quality management and data security, is critical for ensuring the accuracy, reliability, and compliance of organizational data.
The financial services industry has always relied on data and accurate record-keeping. In this article (based on my Data Management Summit keynote), I'm going to look at the latest data trends and how they're affecting (or about to affect) financial organizations like yours.
1. Data volume
A decade ago, former Google CEO, Eric Schmidt, commented that "every two days we produce as much content as was produced by all of mankind for the 20,000 years before 2003." Today, the rate of (primarily unstructured) data creation is far, far greater.
Accordingly, to manage the exabytes of big data created daily, we need better tools — particularly around automation and the cloud — because there's way too much data to cope with manually. Automation also involves artificial intelligence (AI) and machine learning, which help organizations automate intelligent decisions based on data.
2. A changing society
COVID-19 has been a significant contributor to societal change, a development that has impacted data too. Our response to the pandemic showed that remote working is possible on a scale that would have been inconceivable a decade ago. Now we have the networking, bandwidth, tools and capabilities to make the disparate workforce a force for good.
For many office-based workers, going virtual didn't change the basics of their working day too much. We live our lives digitally anyway, generating more and more data each day. And as we spend more of our lives online, data privacy becomes a priority. Privacy and security have to be built seamlessly into all of our data tools and technologies. Also, the increasing importance and visibility of data are encouraging the regulators to pay more attention.
3. Change acceleration
Along with the increase in data volume, escalating technological advances are driving new and more extensive transformations in organizations.
Here, we're not just dealing with vast and monolithic datasets; there's a hoard of smaller and more detailed sources of information in so-called "small and wide" datasets. Again, this drives the evolution of flexible tools and designs to cope with big monolithic datasets and small, wide ones.
To master accelerating change, we need to automate data management itself, deploying metadata tools that can help us manage data at scale (e.g., data cataloging, data lineage, etc.).
In fact, we need to automate as much as possible. Some elements are still quite hard to operationalize — AI, machine learning and model management in particular. Although there are many advanced tools, it's still not the easiest part of the process to automate fully. So, this situation drives a need for both standardization and flexibility.
Data should drive the tools, not the other way round. For example, in the early 2000s, the data industry dealt with highly structured SQL databases, which brought a certain rigor to the way we collected the data. Our approach was, "the data we ingest or create has to fulfill the following needs and requirements, as per this predefined schema."
Organizations don't dictate the structure of the data so much anymore — the data itself dictates the form. So now, we receive tons of largely unstructured data, which we need to figure out what to do with and determine how to extract the information and value from it. Consequently, data tools are becoming more flexible to help us achieve this.
4. Ubiquity
Data is everywhere. It seems like almost everything can generate the stuff — your doorbell, your bicycle, even your running shoes.
The processing of all this data can now occur just about anywhere. With smart devices, the IoT, edge networks, containers and APIs, increasingly, it's more practical to process the data wherever the data sits.
Therefore, we shouldn't tie data tools to a particular location. Some years ago, this resulted in the development of various container-related technologies so that users could process anything anywhere, relatively easily. Today, this drives us toward using fabrics — distributed and interoperable collections of tools and services — rather than one specific tool or cluster.
Drivers of data transformation
Organizations make the data management transformation journey for a variety of different reasons.
Increasingly, regulatory requirements such as GDPR are driving data system improvements, with substantial penalties for failure to manage data correctly.
Productivity — aiming to put data to work — is another prominent driver. More than 60% of enterprise data goes unused for analytics, creating a gap between potential and actual business insight. In many machine learning proofs-of-concept, far more time is spent interfacing with the correct data than doing valuable work. So, anything we can do to improve data productivity, whether by improving data lakes or enhancing AI-based data interaction, will help the bottom line.
Governance is an increasing area of interest. It's vital to make sure we've got a good grip on the data. For example, it's becoming critical for organizations to back-up data and be able to recreate the state of the data at any point in the past. These actions lead us to developments like the most recent ISO/ANSI SQL:2016 database standard or Amazon's Quantum Ledger Database (QLDB), which almost give you "data time travel" capabilities.
The financial services industry is placing greater emphasis on AI and machine learning governance, ensuring unbiased, non-discriminatory and fair AI. Regulators are following this trend (legislation is on the way).
Legacy technology and its replacement is the final driver of change. There's still an enormous amount of legacy tech out there, particularly in the financial services industry. Mainframes, for instance (not all are legacy, of course), often with tooling and architectures that haven't been updated for several years. Indeed, you need strict and coherent processes and strategies to work with legacy data, ensuring that QA is built into your processes. Any new fabrics and architectures must be engineered for future expansion.
Data strategy — the transformation journey
Let's look at the data strategies emerging from financial institutions and driving the evolution of financial data spaces. Some are generic; some are more specific to the financial industry — such as strategies for compliance with new data regulations.
Most organizations will follow a similar path toward data maturity, analytics and AI:
- Data management: Consolidation and curation
- Data democratization
- Data visualization: Self-service analytics
- Enterprise-wide AI, machine learning and decision support
During the initial data management phase, the organization should consolidate its data into one place. It's much cheaper to connect to and work with one location than multiple, diverse data sources. Teams can then curate data on an ongoing basis with automated tools.
In due course, there's a process of data democratization. Anyone in the organization who needs the data should be able to access and use it in their tool of choice, whether that's Excel, a visualization package or something else. Easy access also acts as an enabler for self-service data analytics and visualization with packages like Power BI, Cognos and Tableau.
The next step is self-service (visual) analytics. Success depends on the quality of the data model to which your visualization package is attached. It's vital to get the right data model (or data environment) for a successful roll-out of self-service visual analytics. In other words, if users are to create dashboards, they'll expect the underlying data model to be correct, easily understood and "to do what it says on the tin." If this is not the case (e.g., the data is incorrect or poorly labelled), user trust will be lost and regaining lost trust is a long process.
Lastly, the data foundation is essential for implementing AI and machine learning, as is self-service visual analytics. Put simply, if an organization cannot create a reliable self-service model with underlying data models that are correct and substantial, then the chances of building an enterprise-wide AI and machine learning capability are slim.
Weaving your data fabric
A data fabric is a single environment consisting of a unified architecture with services or technologies running on top of that architecture. Stacks from many different providers now describe themselves as data fabrics. But the basic idea is to try and centralize things so that they're easier to govern and manage, and you replicate fewer unnecessary services.
The goal is to maximize data value, reduce the knowledge gap as much as possible and accelerate ongoing digital transformation.
Defining your delivery approach
How do we deliver this data transformation? Organizations are outsourcing more and more of the data infrastructure and fabric. A decade ago, moving onto Azure, AWS or Google Cloud Platform was regarded as state-of-the-art innovation. Now, a cloud platform is just another service, and infrastructure and fabric can be operationalized, commoditized and outsourced, easily.
On the other hand, the amount of insight and intellectual property (IP) generated is also increasing. And firms are controlling this knowledge in-house, much tighter than in the past.
These are the two approaches to delivery. Organizations are keeping a tighter rein on data insights but are happy to outsource their data infrastructure.
Elements of data management delivery
The structure of the data management delivery has three key components:
- IT delivery
- Data and model delivery
- Regulatory and compliance delivery
IT delivery is an area with which most of us are pretty familiar. Increasingly, this is moving to agile models, combined with DevOps processes.
Data and model delivery mainly concerns your analytics model — an area that's maturing rapidly. The new issues are about how you manage and deliver data, AI and machine learning strategies. For example, regulators are increasing pressure on data versioning, so you can reproduce training results you had before machine learning and audit any data changes. Also, you need to be able to explain how you arrived at your machine learning models and if you tested them for things like discrimination bias.
Regulatory and compliance delivery is also evolving rapidly. There's a swathe of new regulations coming out in 2022, including new EU regulations. So, it's vital to manage data privacy and security in a compliant manner, and be auditable.
Linking it all together
We can tie all this together with a target operating model that considers:
- People
- Processes
- Technology
- Governance
People need a broader range of skills. It's no longer enough just to say you're a DevOps expert. You need to train for regulatory issues in different jurisdictions and conditions. And to know that authentication and authorization requirements are correct, and to be acutely aware of jurisdictional issues and so on.
Processes have to be considered holistically and not in isolation. So, if you have a data analytics delivery project, you have to look beyond models, accuracy and confusion matrices to the various IT, regulatory and compliance aspects. As I'm sure you're aware, you cannot deliver an analytics project involving consumer data without a significant number of compliance checks and issues around data governance. You should factor these aspects into the overall project.
Technology transformation should be focused on increasing commoditization. Organizations need to concentrate on technologies that add the most business value and outsource everything else. Explore how you can outsource elements cheaply and effectively, and make sure the interface between in-house and outsourced components is seamless and secure.
Governance is critical, and the penalties for failure are increasing. Key areas to focus on include data quality assessments, cataloging, management, lineage and so on.
Linking it all together
In a nutshell, successful modern data management relies on integration across a much broader range of disciplines.
And the constant theme is that there's going to be change — and lots of it.
Take the next step toward data transformation
To find out more about the latest data management approaches, get in touch with Zoreza Global to continue the discussion.
Paul Hewitt
Paul leads the Data and Analytics practice for Banking and Capital Markets Consulting EMEA. His role spans all advisory and consultancy in data and analytics, and delivery of all projects from PoCs to large multiyear projects. He’s worked with Zoreza Global for three years and has over 20 years’ industry experience.
