#255 AI Due Diligence – Information Security Checks Before You Integrate AI

AI can be fantastic for relieving a lot of administrative burdens, allowing individuals to focus on more complex tasks that need a human touch. However, many are all too quick to install and integrate, which can lead to crucial vetting processes being skipped.

So many applications have also integrated various AI features, and while you may have vetted the software before these were available, those new AI features still need scrutiny before widespread use within the business.

In this episode, we dive into why there is a need for a more cautious approach to implementing AI and share some tips on basic due diligence checks you can do to ensure an AI application or integration is safe to use.  

You’ll learn

• The link between AI and increasing data breaches
• Recent incidents as a result of AI misuse or error
• Key considerations for the implementation of AI technology
• 11 Information Security checks for AI tools

Resources

• Isologyhub
• ISO 42001 Webinar
• IAF Accreditation Check

In this episode, we talk about:

[02:25] Episode Summary – Stephanie Churchman explains the need for caution when exploring the implementation of AI tools, and provides guidance on some information security checks you can perform to ensure your data stays safe.

[02:45] The link between AI and increasing data breaches: Data breaches tripled since the wide adoption of AI in early 2024 and studies are saying there is a clear link between these two events. Here in the UK alone, 32% of businesses experienced a cyber-attack or data breach in 2023, compared to 43% of businesses in 2025, with us already steadily on track to surpass that in 2026.

Does this mean people shouldn’t use AI at all? No, of course not, but we do need far more caution before you simply start using a tool.

[03:35] Recent incidents as a result of AI misuse or error:

ChatGPT copycat – There was a ChatGPT clone available as a web extension that was downloaded by some 1.5 million users. It functioned just like ChatGPT, answered queries and provided links to legit sources. But, in the background, it was scrapping passwords and gathering information that was to be sold off without users knowledge.

Sage Copilot – The popular accounting software had to temporarily suspend Sage Copilot after a data-isolation flaw occurred. This incident caused an issue where users who prompted the AI to list recent invoices ended up with incorrectly surfaced financial records belonging to unrelated businesses. This was a major security issue, especially for an application thousands of businesses rely on to track their financial records.

Google Gemini – Google Gemini was found to have been abused by bad actors for data reconnaissance. One particular group were building profiles on major cybersecurity and defense companies and were looking to gather specific technical job roles and salary information. Google’s threat intelligence team characterized this activity as a blurring of boundaries between professional research and malicious reconnaissance. Their soft touch approach allowed the bad actors to craft tailored phishing personas and to further identify potential soft targets to compromise.

[06:30] Key considerations for the implementation of AI technology: Any software or technology you plan on introducing into the business that will interact with your and your customers data should be subject to clear vetting procedures, with clear rules for use to follow.

Before integrating an AI tool, ask yourself, is the tool you want to use:

1. Relevant
2. Safe
3. Ethical

Ethical may sound strange, and will depend on what you’re using an AI for. Take CV sorting for example, many studies have shown that AI’s can have an inherited bias based on their training data. This has also now evolved into AI based recruitment tools preferring AI generated CV’s over human written ones.

From a safety standpoint, think about the data you are feeding into those recruitment tools, that’s personally identifiable information, full names, phone numbers, emails and possibly even addresses. A full profile for an individual. Is that system your using closed, do you know if you consented to having any input data used for further training? Don’t just assume that inputted data won’t be used beyond your control.

If that recruitment AI tool gets hacked, who do you think is liable for the breach? Is it the AI tool developer or the business that input the data? You think the answer would be clear, but the legality of all this is still being debated.

[09:10] 11 Information Security checks for AI tools:

#1: Have an AI Policy and AI Integration approval process in place – Many businesses will already have an AI policy in place, most are very generic, so we recommend looking at the guidance provided by ISO 42001 to see what good looks like for an AI policy.

You should also create a clear approval process that any AI tools must pass BEFORE people start using them. This should be clearly communicated to the wider team, and there should be a method to manage these checks such as a ticketing system to kick off the process.

#2: Understand where your data actually goes – Find out whether inputs are used to train the vendor’s models. These inputs can include prompts, uploaded files or even customer data depending on what the tool is. You also need to find out how long that data is retained, and whether it’s stored in a specific jurisdiction.

You can look for answers to these in a DPA (Data Processing Agreement), don’t rely on the basic marketing blurb they state on the website. If those answers aren’t provided, contact the tools support or basic enquiries to find out.

#3: Check for a SOC 2, ISO 27001, or equivalent certification – This is an easy check for vendor’s security posture. Absence of certification shouldn’t automatically disqualify a vendor or tool, but it should prompt more due diligence, not less.

Even with a certification in place, you also need to double check that it’s valid. ISO 27001 for example will need to be certified by a UKAS accredited certification body for those in the UK. For overseas, you will have your own ISO accreditation bodies, which can be verified on the IAF website.

#4: Map out third-party and subprocessor risk – Most AI tools sit on top of other infrastructure like cloud hosting, underlying foundation models and additional analytics tools. You should ask for a subprocessor list to fully understand who else touches the data.

#5: Test for prompt injection and data leakage – If the tool interacts with external content such as emails, documents or web pages, it can potentially be manipulated by malicious instructions hidden in that content. Businesses should ask vendors how they mitigate this and ideally test it themselves.

#6: Clarify access controls and permission scoping – This is especially the case for AI agents or tools with system integrations. You need to establish if the tool operates with the same permissions as the user, or whether it has broader access.

Overprivileged AI agents may operate independently with no human oversight. ‘Human in the loop’ has become a common phrase within cyber security for a reason, you always need a point of human oversight to ensure the AI is doing what it’s supposed be doing and is doing so safely.

#7: Ask about model update and versioning transparency – You need to ensure that the vendor won’t just silently swap out the underlying model for its AI tools, as this can introduce sudden behaviour changes in the tool itself.

Transparency is a key component of emerging AI security frameworks and regulations such as ISO 42001 and the EU AI Act. If a vendor isn’t willing to tell you when they’re making major changes to their tools, then it’s not a vendor you want to entertain.

#8: Evaluate the output reliability and hallucination risk in context – For security-adjacent or compliance-adjacent AI tools, factually wrong outputs are a risk.

AI can have a tendency to ‘hallucinate’ data or outcomes and then present them as fact. So, ask the vendor what guardrails exist and whether their tools’ outputs are auditable / traceable.

They should know what data was used to train their models, or where their models are pulling data from. If they don’t or can’t control what data is being used, then it’s not a tool you can 100% trust.

#9: Review incident response and breach notification commitments – If the vendor is breached, do you how quickly you would be notified, and what their recovery process looks like?

If you hold ISO 27001 and ISO 22301, or simply have a business continuity plan in place then you will already have similar procedures in place for peace of mind for your own customers, so why should you settle for any less?

And just like your clients would expect, breach notifications and expected recovery times should be contractually defined, not just assumed.

#10: Consider the supply-chain risk of the vendor itself – This tech is still relatively new, and so newer AI vendors may have smaller security teams and less mature processes than what you may be used to with more established providers. 

However, startup pace doesn’t mean you have to tolerate the start-up risk. Consider all of the previously mentioned steps, if they don’t have a lot of that in place, then they may not be mature enough yet for you to go ahead with.

This doesn’t mean you have to automatically disqualify them, if they have a clear plan of action for growth, which shows a clear focus on increased security and transparency within a reasonable timeframe, then it’s still worth considering.

#11: AI tool monitoring and Kill switch – In addition to this initial vetting procedure, you should also have a process in place to continuously monitor these AI tools too.

Many AI tools aren’t static, they’ll update and become better or possibly introduce issues as they will inevitably face the risk of bugs and other technical problems as they roll out updates. If a tool is consistently encountering issues, continuous monitoring allows this to be flagged up as a security issue.

Which is where you’ll also need a kill switch in place if an AI tool is behaving unsafely. It’s important that you know how to isolate it and remove it from your systems.

AI tools are more ingrained that your typical software, often designed to work in tandem with existing apps rather than as a standalone system. This will mean that some tools will have access to possibly sensitive data, something that needs to be protected if the AI tool experiences issues that could lead to that data being compromised.

The relevant staff, likely your IT team, need to have a clear process for what to do in those scenarios.

If you’d like any assistance with implementing ISO standards, get in touch with us, we’d be happy to help!

We’d love to hear your views and comments about the ISO Show, here’s how:

• Share the ISO Show on Twitter or Linkedin
• Leave an honest review on iTunes or Soundcloud. Your ratings and reviews really help and we read each one.

Subscribe to keep up-to-date with our latest episodes:

Stitcher | Spotify | YouTube |iTunes | Soundcloud | Mailing List