Preparing your data platform for AI enablement

“We must do something about
generative AI right now.”

With Copilot and GPT hitting the
market, by now most boardrooms have heard some variation of this declaration.
But while it is important for businesses to adapt quickly to new opportunities,
it is worth noting that chasing down the latest trend may sometimes end up in
wasted resources, misaligned priorities, and might even slow down the pace of
innovation. Even as business leaders give the greenlight, they must ask
themselves: As an organization, are we truly ready to adopt this trend?

A robust data analytics
foundation is a crucial precursor to Artificial Intelligence (AI) adoption. Yet
very few
businesses can confidently state that their data is ready for AI. Your
organizational data is best suited for AI when it’s cleansed, consistent, and
centrally stored. But how does one make that happen when most organizations
continue to maintain a network of disparate systems that are poorly integrated?
When data is the differentiator for success, how does one work around data
accessibility issues? When crucial information is trapped in the form of unstructured
data in presentations, strategy papers, and customer logs that are better
suited for record-keeping, not analysis, how does one fully leverage the power of
AI? 

However, it is important to understand that starting the AI journey by going all out and investing in a data platform may not be the right approach for all businesses. For example, companies can take a use-case based approach to AI adoption which focuses on solving the business problem at hand, instead of focusing solely on the technology platform. A use-case driven approach can be especially beneficial as it helps businesses realize the impact and value of their investments faster. It may even be easier to drive organizational support for AI initiatives when it is a single AI use case, as opposed to when it is a complete overhaul to the technology architecture and way the organization works. Additionally, this approach allows for businesses to take a staged approach to data analytics while having a clear platform strategy identified from the beginning itself.

In a two-part blog series, we will explore how businesses can choose the right approach that works for them. This first blog will dive into what constitutes the ‘all-in’ approach, and explore how business leaders can better prepare their data platform for AI adoption to mitigate the risk of having to shelve AI projects midway. The second blog will look at how an iterative approach to AI may benefit some businesses.

Over two-thirds of IT leaders expect data volumes to increase by 22% on average over the next year. This growing swamp of data can become a cause for concern as the rising volume and diversity of data sources means that more effort is required to standardize the data. Businesses will have to consider several key questions:

• What are the data sources we have to work with? Sources could range from databases, cloud storage, external data sources and APIs to unstructured data sources like emails, documents and sensor data.
• Where is your data located? Is it on the Cloud, on-premise, or in a combination of both?
• What type of data do you typically work with? Whether its sensitive customer information, transactional data or feeds from your sensors, your data needs to be accurately catalogued and classified based on the requirements of your business, the sensitivity of the data and your business’ unique priorities.
• How does your data fare in terms of quality? Your AI algorithms are only as good as the data it is trained on. It is important to evaluate the quality of the data in terms of accuracy, completeness, consistency, timeliness and reliability to determine its usability.
• What dependencies exist between your data sources, how often are they updated, and how does data flow from system to system?
• Are there proper permissions in place to access and read your data sources? To understand dependencies and bottlenecks, businesses must document how data is accessed, who has access to it, and for what purpose.

Anomalies, missing values and inconsistencies in your data sets can significantly skew your analytics and AI outcomes. To ensure that your data points are reliable and do not end up introducing errors into your Machine Learning (ML) algorithms, data prep is an important step. For example, missing values related to inventory numbers or outliers in your sales data can distort your demand model predictions. If continued unchecked over a period of time these errors can have a domino effect on your business and affect your bottom-line.

Data prep work typically involves several iterative, ongoing processes including, but not limited to:

• Data collection: The first step involves collecting the data sets relevant to the business problem you are trying to solve and setting up a connection to a database/API so the extracted data can be loaded into a common repository.
• Data extraction and loading: This involves ensuring the completeness and accuracy of data sets, by removing/fixing missing values, nulls, or outliers to prevent them from skewing analysis results.
• Data cleaning: Depending on the AI use case identified, data labelling may also be required. This includes identifying raw data and adding one or more meaningful and informative labels to provide context so that an AI model can learn from it. This data-centric approach to AI ensures that better results are delivered.
• Data labelling: Depending on the AI use case identified, data labelling may also be required. This includes identifying raw data and adding one or more meaningful and informative labels to provide context so that a ML model can learn from it.
• Data transformation: This may involve normalizing variables in your data so it doesn’t disproportionately influence the AI model, and ensuring that qualitative data (such as product categories or customer segments) may also need to be encoded into a numerical format so that your data is in a unified and usable format that can be fed into the algorithms.

Data preparation is an ongoing process that can involve many more steps depending on the purpose of your analysis. As AI models evolve or when new data becomes available, data professionals will have to revisit their data prep process.

“2024 will see far greater activity around securing the right digital foundation from which to explore and build AI-focused initiatives. Without the right data and the right context, AI is difficult to fully capitalize on.”

Gaurika Wijeratne, Vice President, Data & AI, Fortude