Despite data and data analytics being top of mind for many organizations – and generative AI making waves in the news – many executives still bristle at the idea that data can add new insights to existing intuition and experience. “What’s wrong with my intuition?” I can hear them saying. “I’ve built a successful business with my intuition. It’s served me just fine up until now.” It’s a fair point. Many businesses’ leadership have decades of battle scars that help them avoid big mistakes, filter good ideas from bad ideas, and make good decisions despite having very little objective data. This is the whole point of resumes and interviews – we seek experienced candidates because we hope their experience will foster clarity and better decision-making in critical moments.

But the reality is that data does not replace experience. Moreover, data is not just to help people make better decisions. Data is a digital representation of real-world events, and when combined with experience and automation, it can be absolute gold. Equally, investing in data is less about “building some reports to address questions I already know the answer to” and more about streamlining business operations to free up time and money that can be spent (or saved) on innovation. It’s about tracking revenue and communicating KPIs broadly instead of panicking when it looks like the organization won’t meet its goals. It’s about anticipating demand for materials or equipment, for example, and ordering it just in time instead of holding inventory forever. It’s about automating regulatory reporting processes so teams don’t need to work through the night to compile the right documents. More than anything, investing in data is about investing in efficiency.

Data Doesn't Replace, it Augments

In other words, organizations that have turned data into a competitive advantage – organizations that are “data-driven” – haven’t replaced experience, they’ve augmented experience. They have freed up breathing room for teams around the organization so they can be proactive instead of reactive, so they don’t need to rush to put out fire after fire. 

To be clear, data-driven organizations don’t robotically follow the data to wherever it leads like someone driving a car into a lake because the GPS told them to turn left. Instead, through data analytics and automation, they have given themselves the space to look around, think about where they should be headed, and determine how to get there. And thanks to automation, their teams – finally – have the bandwidth to get them where they want to go.

The Data on Data Maturity Today

Or at least that’s the dream. The reality is that most organizations never finish their journey toward becoming data-driven, never make it a superpower. Let’s look, of course, at the data. According to a recent New Vantage Partners (NVP) survey of executives at Fortune 1000 companies, “just 23.9% of companies characterize themselves as data-driven, and only 20.6% say that they have developed a data culture within their organization.” A 2021 survey from 2nd Watch found that “only 26% of survey respondents said they have any data strategy at all and 70% don’t have what they consider to be a mature data strategy.” 

Most organizations don’t make data a superpower for lack of trying: 87.8% of organizations in the aforementioned NVP study reported increases in data investments during 2022. In reality, it’s mostly because doing data well is really hard. In most organizations, there are a myriad of obstacles: data doesn’t become a priority; they make the wrong investments; they can’t properly demonstrate the value of data, or a hundred other reasons. In other words, they get stuck. And even the organizations that do become successful with data often take a long time to get there because, while there are many paths to becoming data-driven, some are much slower and harder than others. 

The Technology Isn't Everything

A disclaimer: My job is to build and sell data analytics technology. I can’t help but think about these things from the perspective of technology (and Altair’s technology in particular). But to be clear, I do not think technology is the solution to every problem. In fact, this journey is as much about changing mindsets and process as it about technology. Indeed, according to the NVP study, 79.8% of executives “cite cultural issues — organizational receptivity to change and business transformation, changes to organizational processes, people and skills, organizational alignment, and communications – as the greatest obstacles to realizing business value.” 

That said, there are some clear milestones, themes, and technologies that can help any organization forge an efficient path to data success. Join me as we dive deep into becoming data-driven. What does it take to make data a differentiator for your business? What are the most common pitfalls on the path? Answers to these questions and more below.

A metaphor can help illustrate what the journey to becoming data-driven looks like. In many ways, becoming data-driven is like scaling a mountain. Just like you wouldn’t scale a mountain without reserve food and water, warm layers, and a sturdy pair of boots, you wouldn’t want to undertake your data journey without the right partners and technology. So yes, there are going to be some technology recommendations in this, and yes, I am going to recommend Altair’s technology because we build it specifically to solve these types of problems.

Without further ado, let’s start our climb up Mount Data. We’ll start at our home base, Beginner’s Base Camp.

Looking Up: Beginner's Base Camp

Organizations at Beginner’s Base Camp and their executives are often skeptical that data will provide enough value to outweigh the cost and time it takes to do well. They think the business operates just fine without significant investments in data and can’t see why now is the time to make it a priority. 

And to be clear, these organizations do have and use data. Siloed groups at organizations in Beginner’s Base Camp use applications like Salesforce, HubSpot, NetSuite, OSISoft Pi, and Google Analytics to understand small slices of their universe. The issue isn’t whether they have data (every company has data), or whether they get value from data (most companies get some value from data). The issue is that they haven’t prioritized processes that amplify what data can do. They may create Salesforce reports to roll up billings by region or month, but these aren’t linked with marketing campaigns so no one knows how effective these campaigns are. 

And while some isolated individuals seem to know the secret to continued sales success, it has not been easy to align the whole sales force on priorities and strategy. At Beginner’s Base Camp, manufacturing equipment is carefully monitored and hums along day after day… until it doesn’t. Then these organizations eat unplanned downtime, fix it, and change the maintenance schedule so that maintenance is a bit more frequent (and hope it’s frequent enough). They get bill after bill for cloud resources, and despite saying how important it is to cut costs, each month the bills get bigger. Maybe the loudest person on the team keeps saying product price is too high, so they listen and lower the price. And then someone else on the team claims it’s too low, and they listen. But wait, somehow it’s too high again – should we change it again?

Data at companies at Beginner’s Base Camp is scattered and often trapped in business applications, PDFs, “final-final-final” Excel spreadsheets, manufacturing machines, and the heads of superstar salespeople. No KPIs are tracked, and even if they were, most people in the organization do not (and cannot) see them because they don’t have a license to the right business application. Information is siloed and hoarded by leadership; this isn’t on purpose, but rather because there is no systematic process to disseminate information. Sure, people get the job done and the company trudges forward, but it often takes long nights of repetitive tasks; and because people are doing tasks under duress, sometimes mistakes crop up that create even more work and more firefighting to remedy. It’s a self-perpetuating cycle.

Leaving Beginner's Base Camp

Leaving Beginner’s Base Camp starts with a mentality shift. Before all else, data must become a priority. The entire company – starting with the CEO and moving down from there – needs to believe that data can and will make an impact for the organization not as a novelty or a “maybe someday” thing, but as a core competency. If you want to move fast, you have to commit. Sure, you can take it slow and see how it goes, but you’ll be just that: slow. And if you’re slow, people might start getting fed up – and if people get fed up, you may never reach the “seeing how it goes” because you are forced to stop too soon.

Make Data a Priority

Unfortunately, the beginning is particularly hard. Early on, there will be plenty of opportunities to give up. You’ll quickly find out that data is expensive; talent is expensive; software and hardware is expensive; your team will have doubts; leadership will question whether now is the right time and will feel their experience will always outweigh anything on a dashboard, so “Why bother?” And, of course, there will always be other priorities: hitting revenue and EBITDA numbers; improving your cybersecurity score; onboarding a new channel partner; the list can go on and on.

So much gets in the way of data becoming a priority – but you have to believe it’s worth it, that doing the work now will pay off. Don’t get a consensus, don’t take a vote – just make it happen or it never will.

Establish Who Will “Own Data”

Next, you need to establish a core of data expertise in the company. You need someone to “own data” within the organization, ideally a chief data officer (CDO) or chief data analytics officer (CDAO). And ideally that person should have a team; a small team of three to five is ideal, but even one or two people go a long way. These are the people who will own your data efforts’ success, the people the company will turn to with data ideas and requests, and starting out, the people who will fulfill these requests. They will build dashboards, set up automated reports, and perform simple data science projects. They will be the vanguard, the ones grabbing quick wins and demonstrating immediate value. 

This core group will also be responsible for laying the foundation for your data architecture, so at least one of them needs to be an experienced data engineer or data architect. They will decide the right tools to invest in, devise scalable processes for the company’s success, and ensure costs don’t get too high. Expect them to inventory your data-producing machines and applications and devise a plan for creating a data architecture that allows your data team to reliably work with the data you produce. Also expect that they will put processes and safeguards in place that keep the data in your data architecture clean, secure, and accessible. You’re going to start hearing a lot about Python, SQL, Apache (insert tool here), extract-transform-load (ETL), data warehouses, data lakes, views, tables, schemas, and pipelines. When new data or views are needed, there will be a channel to request that. When a new dashboard is needed, there will be a channel for that too. They will turn the mindset change you started into something actionable and real.

Obviously, this is not everything that needs to be done to establish expertise in data, and these steps are somewhat vague – but that’s on purpose. Each decision you make will alter your path, and they are highly context-dependent; they will differ based on the size of your organization, your objectives, and your market. Startups will have different needs from mature manufacturing companies. Steel foundries will have different needs than a company that makes meditation apps. But many of the themes and major “landmarks” on the path to becoming data-driven will remain the same. If you prioritize data, establish a strong core of experts, give them the support they need to create the foundational architecture that fits your organization, and have the patience to see things through, things will start to click.

Reaching "Proficiency Point"

Organizations who reach Proficiency Point have made great progress. Within these organizations, data access is transparent and controlled. A small group of experts shepherds data architecture. A central data team creates and distributes reports when requested, ranging from revenue projections to customer segmentation to manufacturing process monitoring. Typically, these organizations have a CDO or at least a single senior leader whose sole responsibility is to own the data estate. Systems have been set up to keep data secure, clean, and available to the data team. And just as importantly, the organization sees and understands the value of data. Employees know to look for data in a decision and are eager for reports. Many menial tasks have been automated, and data entry is rarely needed. They have made investments to make it possible to get information from data, and it has started to pay off. In all, the organization’s first instinct is now to think about problems from the perspective of data – a data-first mindset exemplified.

More Questions to Answer

But while data has solved many problems at organizations that have reached Proficiency Point, new problems have popped up. Bills roll in month after month, year after year and, man, it is expensive. These bills raise eyebrows and leadership starts asking a ton of questions: 

• “Surely we should be focused on reducing scrap from our manufacturing line – why are we building compensation reports?”
• “How many projects have we done that contribute to the goals we set at the beginning of the year?”
• “Who on my team is working on what?”
• “Who is the right person to work on this next project?” 
• “Who are my star performers?”
• “What pieces of all this cost the most?”
• “Who’s consuming the most resources? Is it worth it? Do they need to be consuming that much?”
• “Are we having an impact on the business?”
• “Which business lines ask the most of us?” 
• “Are we a successful organization?”

Unfortunately, only rarely do good answers emerge. Even worse, the data team – even at organizations at Proficiency Point – never seems big enough for the number of requests they get. They are working tirelessly but it’s not clear what the best way to relieve their workload is: do we take on less or expand the team? And even if leadership decides the only option is to increase the data team, it still takes months to fill the new spot. Furthermore, with data talent, it often turns out salaries have increased 40% since the last person was hired – which is way over budget. Even more, cloud costs keep going up and up with no end in sight. Worse yet, sometimes it feels like the data team is working on things that never make it to production, almost like research projects. Everyone appreciates the need to learn and grow, but it looks strange when there’s also a huge backlog of things to do.

Further To Go: Moving Beyond Proficiency Point

A lot of organizations are content to stay in Proficiency Point. I don’t necessarily blame them: there’s hot coffee, a great view, and some pretty nice reports. Organizations here have made data a key to their business, sometimes to the point where they find it hard to identify how they could improve. Everyone seems to agree: “Our data adds value.” 

The numbers back this up. According to the NVP study, 91.9% of companies say they see value from their data analytics investments even though just 23.9% described themselves as “data-driven.” Much of the data maturity content from the early to mid-2010s effectively has our proverbial Proficiency Point as the journey’s final destination because the mere task of making data valuable is an enormous achievement by itself.

Add Accountability to Avoid the Abyss

But being content here is the whole reason why that 23.9% number is so low. The final stage of becoming data-driven shouldn’t just be that data adds value, you should know how much value. You should be able to prioritize projects based on their value and it should always be clear – both to the data team and to leadership – why the team is working on one project over others. This is commonplace in other areas of the business – salespeople have quotas, marketers have lead goals, developers have quality and velocity metrics to achieve, etc. – why should the data team be any different? They certainly aren’t less expensive or less important. Data teams should be able to demonstrate why a project was prioritized, and more importantly, they should be able to assess whether projects were successful based on objective, impact-driven KPIs. Organizations who do not do this fall into what I call the “Accountability Abyss.” They never quite know how much more to invest in their data team because they never quite know how much value their data team provides, they just know their team has to be providing some value.

Democratize Data

Another big part of moving beyond Proficiency Point – to the rarefied air of Superpower Summit – is being able to scale data analytics without incurring a costly increase in the data team. The data team can’t do everything – they are too expensive, talent is too hard to find, and they are too removed from the business. If you keep data analytics walled inside the confines of the data team, the data team will be constantly bogged down working on trivial data tasks: another report; a new column; “Can you change this line on the graph from burgundy to burnt orange?” Even worse, “Oh shoot, I meant fire engine red, not burnt orange – can you change it again?” or “Turns out the metric I asked you to track doesn’t tell me what I thought, can we try this one instead?” Sure, just give me three weeks to work through all these other requests first. 

Data analytics needs to be democratized throughout the organization if it’s going to scale. It’s a given that everyone at a company should be able to write an email, or give a presentation, or build a spreadsheet. The same should be true for building a dashboard, creating a new data view, or forecasting the future from historical results. No one expects the entire company to suddenly be doctorate-level data scientists – but at the same time, 90% of data analytics tasks don’t require a doctorate-level data scientist. Let your data team tackle the things that require a Ph.D. Let everyone else do the rest, and critically, make sure they have the tools they need to do it well.

A big part of making this democratization possible is upskilling. You may not need everyone to present a thesis, but you do need them to understand the basics of data. They should know what it is, how to work with it, and other basic concepts. If you don’t establish this baseline, then it isn’t realistic to expect a person or team to be able to make an impact with data.

To upskill well, you need to give the team technology accelerators that fit their skill level. Instead of asking everyone to learn to code, direct them to the great low- and no-code point-and-click tools that can give non-experts the ability to do almost everything an expert can do. Some examples include workflow-based development tools for building everything from simple pipelines to advanced machine learning models, autoML tools to quickly see the results of potential machine learning techniques, and of course, generative AI, which has lowered the bar even further by giving business users the ability to ask questions of data in their preferred language. You have a diverse team – you will need an equally diverse toolset to fit their skill sets.

Make Data Access Transparent But Governed

Transparent data access is also critical. You can’t say your data is democratized unless the whole organization can access it when, where, and how they need. The data team may know exactly what data is available and how to get to it, but the rest of the team probably doesn’t. They shouldn’t have to go to IT and wait weeks for approval to see sales data from last month. Instead, they should be able to search their data estate and, without knowing the ins and outs of various schemas, find the data with the information they are looking for; if they can’t find it, they should be able to quickly create a new view that does what they need. They should be able to answer questions like:

• Where is the source of truth for sales data?
• Do we have sales data connected to marketing data? 
• Do we have data on how our customer use our products?
• Can we follow an entire customer journey from first contact to sale? 
• Where is our manufacturing sensor data? What values are collected?
• Where is our quality data? Can we follow a product from sourcing to shipping?

That said, heed wise old Uncle Ben: “With great power comes great responsibility.” Transparent doesn’t mean unsecured or uncontrolled. The more people who need access to data, the more important it is to make sure only the right people have access to the right data. After all, most organizations don’t actually want everyone to have access to everything. People should know their organization has sales data for the whole world, but it may be best if they’re only able to see the data for their region. This is where a strong data fabric, catalog, and governance come in, to provide both flexibility and control. Once you have these three aspects – accountability, democratization, and transparency/governance – you’re very close to the summit.

Views from the Top: “Superpower Summit”

Now, after everything, we have reached Superpower Summit, the true peak of Mount Data. Those at the summit have turned data into a differentiator and unique competitive advantage – a superpower – and have achieved this by doing several key things.

First, they have broken down the walls surrounding the data team. They’ve enabled their entire organization to work with data in a scalable, transparent, secure way. That means everyone has access to the right data, the freedom to create their own views of the data, and their own dashboards. Even machine learning modeling and predictions are not out of reach for most people. 

Additionally, these organizations have taken undue pressure off the data team. Now, the data team can focus on maintaining the sanctity of the data architecture and tackling the organization’s most complex projects. They make sure data is governed, correct, and accessible, and have the bandwidth to tackle obstacles that require advanced techniques. But most importantly, at the summit, the data team is the source of diligence and process. Data projects are tied to results. They have real KPIs that help team members determine whether projects were successful. Each project is tied to clear impact metrics and each project is prioritized before work starts. Costs are monitored, managed, and compared with impact to make sure the investments the company makes in data are worth it. In this way, companies at the summit have figured out not what the magic bullet for success is – for there is no magic bullet, sadly – but rather the right process and mentality that will help them keep getting better. This is the key to staying at the summit. 

Organizations that make their data a superpower aren’t just winning bragging rights, they also see the largest impact on their business. McKinsey estimates that data-driven companies earn up to 20% higher revenue than competitors. Other studies show that data-driven companies see 2-3x improvements in time to market, customer satisfaction, operational efficiency, and productivity compared to their competitors. And for employees, it means no more long nights of data entry; no more last-minute panic about hitting numbers; no more confusion about maintenance schedules, about causes of scrap, about who the strongest customer segments are or about the right price for products; regulatory reporting and reconciliation are automated; risks and threats are identified before they become problems. Data-driven businesses run – and thrive – on data.

The Key Steps to Becoming Data-Driven, Summarized

The path will never be straight. You will run into a hundred issues along the way. As I mentioned, no two data journeys will be identical, but some basic steps remain true. In my experience, if you can achieve these things, you will have a much higher chance of making data a superpower for your organization.

1. Commit: Convince your leadership this is a priority. 
2. Establish the core: Get a CDO. Hire a data team. Build out the baseline data storage ecosystem.
3. Get your feet wet: Tackle the first couple of projects that got you interested in data. 
4. Democratize data work: Empower your team to work with data themselves through tools that fit them (so your data team doesn’t have to do everything).
5. Make data accessible: Everyone should be able to find the data they need (and not the data they don’t).
6. Make projects accountable: Tie measurable, actionable impact to projects and report on it so you can improve over time. 
7. Adapt, iterate, and always look to improve: Make sure you know what success looks like, and then take one step after another in that direction.

And of course, as I said earlier, I’m biased. I feel like I know the technology that will accelerate this path because my job is to build it. The Altair® RapidMiner® platform and Altair’s Center of Excellence (CoE) are specifically built to streamline this journey. Our CoE helps organizations make the organizational shift towards reaching Proficiency Point. And Altair RapidMiner, our powerful unified data platform, makes data accessible, provides expert tools for data teams, low-code tools for non-data teams, and helps organizations tie impact to projects so they can accelerate their path to Superpower Summit. We built the CoE to make the climb up Mount Data as straightforward as possible, and we built the Altair RapidMiner platform to make the path across the Accountability Abyss as safe as possible.

In other words, if you want to make data your superpower and reach the crest of Mount Data, doing the things above will give you the best chance of success. And if you want the process to be as painless and fast as possible, work with the organization that has designed the technology specifically designed to make that happen: Altair.

To learn more about Altair’s data analytics capabilities, visit https://altair.com/data-analytics. To learn more about Altair RapidMiner, visit https://altair.com/altair-rapidminer.