Eggs Benedict Tech · EBT Lab Notes
October 2026
AI privacy and data governance

Worried, and doing it anyway

Most organizations fear what happens to data typed into AI tools. Close to half of their people type it in regardless. What that costs, what has already happened, what is coming, and what an organization can do about it.

What we found

The fear is widespread and the behavior has not changed. In a 2025 study of 2,600 privacy and security professionals, 64% worried about sensitive information leaking, and nearly half admitted entering personal or non-public data into generative AI tools.1 Only two in five people whose organizations use these tools say a policy guides their use.2

The cost is now measured. A 2025 study of 600 breached organizations found that high use of unapproved AI tools added $670,000 to the average breach, and that one in five organizations had a breach due to it.5

Most of the exposure runs through which tool, and which tier of it, people actually use. That is something an organization can decide.

Fear
64%
of privacy and security professionals worry about sensitive information leaking through generative AI.1
Behavior
15% → 45%
share of employees using AI tools frequently, in a single year.4
Cost
$670,000
added to the average breach where use of unapproved AI tools was high.5
01 · The gap

Everyone is worried. The data goes in anyway.

Four independent studies, run on different populations, describe the same gap between what people fear and what they do.

"64% of respondents worry about inadvertently sharing sensitive information publicly or with competitors, yet nearly half admit to inputting personal employee or non-public data into GenAI tools."Cisco 2025 Data Privacy Benchmark Study, 2,600 privacy and security professionals, 12 countries1
"…almost half admit to using AI in ways that contravene organizational policies and uploading sensitive company information, such as financial, sales, or customer information, to public AI tools."KPMG and the University of Melbourne, 48,000+ people in 47 countries2

The same global study found that only two in five say a policy guides their use of these tools, and that two in three rely on AI output without evaluating it.2 Its US page reports that 72% of workers received no AI training or education.3

The 2026 Verizon Data Breach Investigations Report names the habit directly: "Shadow AI, referring to employees using unapproved AI tools at work, is now the third most common non-malicious data leakage related activity," while frequent use of AI tools "surged from 15% to 45% of employees in a single year."4

Small professional firms say the same about themselves. In the 2026 study of 1,376 US independent insurance agencies, the two biggest barriers to adopting AI were "Lack of knowledge about capability (60%) and security and privacy (48%)", both up ten points since 2024.10 Among independent agents asked for their single biggest concern, data privacy or compliance risk came first (24%) and inaccurate outputs second (22%).11 The public shares the worry: 71% of US adults expect the increased use of AI to make their personal information less secure.12

02 · The cost

What a breach costs now, and what AI adds

Breach costs, as published
MeasureFigureSource
Average cost of a data breach, global (2025 study)$4.44MIBM / Ponemon, 600 organizations5
Average cost of a data breach, United States (2025 study)$10.22Msame5
Added cost where use of unapproved AI tools was high$670,000same5
Organizations reporting a breach due to unapproved AI use1 in 5same5
Organizations reporting breaches of AI models or applications13%same5
…of which reported no AI access controls in place97%same5
Breached organizations with no AI governance policy, or still developing one63%same5
Average cost of a data breach, global (2026 study)$4.99MIBM / Ponemon, 602 organizations6
Malicious breaches that were AI-enabled, and their average cost (2026 study)1 in 4 · $6Msame; up 56% on the year6

Penalties stack on top of breach costs. In California, the privacy regulator's adjusted fines are up to $2,663 for each violation, and up to $7,988 for each intentional violation or one involving consumers known to be under 16.13

Read these as averages, not forecasts. The breach studies sample a few hundred organizations, most of them larger than a small firm. We found no published figure for organizations under 500 employees. What carries over to any size is the direction: unapproved tools, missing access controls and missing policy each push costs up.

03 · On the record

What has already happened

A

Deleted conversations, kept by court order

On 13 May 2025, a federal magistrate judge in New York directed an AI provider to "preserve and segregate all output log data that would otherwise be deleted on a going forward basis … whether such data might be deleted at a user's request" or for other reasons.7 Lesson: where conversations are stored, and on whose terms, decides who can be compelled to keep them. A user pressing "delete" is not the last word.

B

AI systems became breach targets in their own right

In the 2025 breach study, 13% of organizations reported breaches of AI models or applications, and 97% of those had no AI access controls in place.5 A year later, one in four malicious breaches was AI-enabled.6 Lesson: an AI tool is a system to secure, not just an app to approve.

C

One email, no clicks, data out

Researchers documented "a zero-click prompt injection vulnerability" in a widely used business assistant "that enabled remote, unauthenticated data exfiltration via a single crafted email."8 Lesson: an assistant that reads your mail and files can be steered by content someone else wrote. Ask what it can read, and what it can send out.

D

Regulators treat AI claims like any other claims

In September 2024 the US Federal Trade Commission announced five cases against deceptive AI claims and schemes, under the line "Using AI tools to trick, mislead, or defraud people is illegal."9 Lesson: what a vendor says about privacy and accuracy is a claim. Ask for the evidence behind it.

E

Some organizations simply banned the tools

In Cisco's 2024 study, 27% of organizations had banned generative AI at least temporarily over privacy and data-security risks.14 Lesson: a ban without an alternative tends to move the use somewhere less visible, which is what the 2026 leakage figures in section 01 describe.

04 · The tier

The same assistant can mean two different things

Most large AI providers run a consumer tier and a business tier under different terms. The difference is where most of the privacy risk sits. Three examples from providers' own pages, read in October 2026:

None of these terms is hidden. They are simply different, and the tier a person reaches for on a busy afternoon is usually the free one. The 2025 global study found most employees using free, public tools rather than ones their employer provides.2

05 · What is coming

The next risks are already named

1

Prompt injection

The security community's list of top risks for AI applications puts it first, and notes that "prompt injections do not need to be human-visible/readable, as long as the content is parsed by the model."18

2

Assistants that act

"Excessive agency": systems given "the ability to call functions or interface with other systems" that act on a prompt without anyone approving the action.19

3

Leakage and inference

The US National Institute of Standards and Technology warns that "Models may leak, generate, or correctly infer sensitive information about individuals," and that generative AI "expands the available attack surface."20

4

Attackers using AI

AI-enabled breaches rose 56% in a year.6

5

Rules arriving

The European Commission's enforcement powers over general-purpose AI model providers apply from 2 August 2026, and models placed on the market before August 2025 must comply by 2 August 2027.21 In the US, the SEC's examiners will "assess whether firms have implemented adequate policies and procedures to monitor and/or supervise their use of AI technologies."22 State technology chiefs now rank artificial intelligence as their number one priority, overtaking cybersecurity for the first time.23

06 · The professions

What lawyers, advisers and governments are being told

"…reading the contract terms and the terms of service to determine if any confidential information could be shared or used for training purposes."State Bar of Michigan, AI workgroup report, June 2025, on lawyers' duty of confidentiality24

The same report says the use of AI tools "may require informed consent from the client prior to inputting data into any AI tool," and is candid that on disclosure, "At this time in Michigan, the answer to this question is not clear."24

For investment advisers, the SEC's 2026 examination priorities look at AI supervision (section 05). For federal agencies, the White House's 2025 acquisition memorandum tells agencies to ensure contracts "permanently prohibit the use of non-public inputted agency data and outputted results to further train publicly or commercially available AI algorithms" without explicit agency consent.25 Across professions the instruction converges: know where the data goes, and write it into the contract.

07 · What to do

For each person, and for the organization

For each person

For the organization

1

A written policy, and an approved tool people will actually use

A ban or a policy without a usable alternative pushes use out of sight. Only two in five people say a policy guides them today.2

2

Vendor due diligence before buying

NIST's generative-AI profile: "Update and integrate due diligence processes for GAI acquisition and procurement vendor assessments to include intellectual property, data privacy, security, and other risks."20

3

Access controls on the AI systems themselves

97% of organizations with AI breaches had none.5

4

Least privilege: no unapproved actions

Limit what an assistant can reach and do, and require a person to approve anything with consequences.19

5

Retention you chose, with the evidence kept

Decide what may be stored, where and for how long, and keep the record of how it was configured.

6

Human review and training

Two in three people rely on AI output without evaluating it.2 Train them, and make review part of the job.

7

An incident plan and an exit plan

Know what happens when something leaks, and what is deleted, and how you would know, when you stop using a tool.20

Questions to ask any AI vendor

  1. Under the tier we would actually use, is our data used to train your models?
  2. How long are prompts and responses kept, and where?
  3. Can anyone at your company read them, and when?
  4. If a court ordered you to keep them, what would be kept?
  5. What does a "share" or "publish" feature expose?
  6. Can the assistant take actions in other systems, and who approves them?
  7. What happens when a document or email contains hidden instructions?
  8. What evidence can you give us of the configuration you describe?
  9. When we stop, what is deleted, and how would we know?
08 · Limits

What this note does not establish

Method. Every figure and quotation here was read on the publisher's own page or document on 7 October 2026. Figures are reported as published, not recomputed. Corrections are welcome.

References

Sources

  1. Cisco. "Cisco 2025 Data Privacy Benchmark Study." 2 April 2025. newsroom.cisco.com/c/r/newsroom/en/us/a/y2025/m04/cisco-2025-data-privacy-benchmark-study-privacy-landscape-grows-increasingly-complex-in-the-age-of-ai.html
  2. Gillespie, N., Lockey, S., Ward, T., Macdade, A. and Hassed, G. Trust, attitudes and use of artificial intelligence: A global study 2025. University of Melbourne and KPMG, 2025. assets.kpmg.com/content/dam/kpmgsites/xx/pdf/2025/05/trust-attitudes-and-use-of-ai-global-report.pdf
  3. KPMG US. "Trust, attitudes and use of artificial intelligence." 2025. kpmg.com/us/en/articles/2025/trust-attitudes-and-use-of-artificial-intelligence.html
  4. Verizon. "Vulnerability exploitation top breach entry point, 2026 industry-wide DBIR finds." 19 May 2026. verizon.com/about/news/breach-industry-wide-dbir-finds
  5. IBM. "IBM Report: 13% Of Organizations Reported Breaches Of AI Models Or Applications, 97% Of Which Reported Lacking Proper AI Access Controls" (Cost of a Data Breach Report 2025, Ponemon Institute). 30 July 2025. newsroom.ibm.com/2025-07-30-ibm-report-13-of-organizations-reported-breaches-of-ai-models-or-applications,-97-of-which-reported-lacking-proper-ai-access-controls
  6. IBM. "IBM Study: One in Four Malicious Breaches are AI-Enabled, Costing Companies $6 Million on Average" (Cost of a Data Breach Report 2026). 29 July 2026. newsroom.ibm.com/2026-07-29-ibm-study-one-in-four-malicious-breaches-are-ai-enabled,-costing-companies-6-million-on-average
  7. United States District Court, Southern District of New York. Order, In re OpenAI, Inc. Copyright Infringement Litigation, 25-md-3143, Magistrate Judge Ona T. Wang, 13 May 2025.
  8. "EchoLeak" (CVE-2025-32711), arXiv:2509.10540, submitted 6 September 2025. arxiv.org/abs/2509.10540
  9. US Federal Trade Commission. "FTC Announces Crackdown on Deceptive AI Claims and Schemes." 25 September 2024. ftc.gov/news-events/news/press-releases/2024/09/ftc-announces-crackdown-deceptive-ai-claims-schemes
  10. Independent Insurance Agents & Brokers of America and Future One. "Big 'I' and Future One Release 2026 Agency Universe Study." 23 September 2026. independentagent.com/news/big-i-and-future-one-release-2026-agency-universe-study/
  11. Independent Insurance Agents & Brokers of America. "Two-Thirds of Independent Agents Plan to Increase AI Use This Year" (ACT Tech Trends Report). 19 February 2026. independentagent.com/news/two-thirds-of-independent-agents-plan-to-increase-ai-use-this-year/
  12. Pew Research Center. "Americans and AI 2026: chatbots, smart devices and views on impact." 17 June 2026. pewresearch.org/internet/2026/06/17/americans-and-ai-2026-chatbots-smart-devices-and-views-on-impact/
  13. California Privacy Protection Agency. Announcement of adjusted monetary thresholds and fines, 17 December 2024. cppa.ca.gov/announcements/2024/20241217.html
  14. Cisco. "More than 1 in 4 Organizations Banned Use of GenAI Over Privacy and Data Security Risks." January 2024. newsroom.cisco.com/c/r/newsroom/en/us/a/y2024/m01/organizations-ban-use-of-generative-ai-over-data-privacy-security-cisco-study.html
  15. Anthropic. "Updates to our consumer terms." 28 August 2025. anthropic.com/news/updates-to-our-consumer-terms
  16. Google. Gemini Apps Privacy Hub. Read October 2026. support.google.com/gemini/answer/13594961
  17. Microsoft. "Data, Privacy, and Security for Microsoft 365 Copilot." Read October 2026. learn.microsoft.com/en-us/copilot/microsoft-365/microsoft-365-copilot-privacy
  18. OWASP GenAI Security Project. "LLM01:2025 Prompt Injection." genai.owasp.org/llmrisk/llm01-prompt-injection/
  19. OWASP GenAI Security Project. "LLM06:2025 Excessive Agency." genai.owasp.org/llmrisk/llm062025-excessive-agency/
  20. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). July 2024. doi.org/10.6028/NIST.AI.600-1
  21. European Commission. "Guidelines for providers of general-purpose AI models." Last updated 28 April 2026. digital-strategy.ec.europa.eu/en/policies/guidelines-gpai-providers
  22. US Securities and Exchange Commission, Division of Examinations. Fiscal Year 2026 Examination Priorities. sec.gov/files/2026-exam-priorities.pdf
  23. NASCIO. "State CIO Top Ten Policy and Technology Priorities for 2026." nascio.org/resource/state-cio-top-ten-policy-and-technology-priorities-for-2026/
  24. State Bar of Michigan, AI Workgroup. Report on artificial intelligence in the legal profession. June 2025. michbar.org/Portals/0/publications/pdfs/Age_of_AI_Report_June25.pdf
  25. Office of Management and Budget. M-25-22, "Driving Efficient Acquisition of Artificial Intelligence in Government." 3 April 2025. whitehouse.gov/wp-content/uploads/2025/02/M-25-22-Driving-Efficient-Acquisition-of-Artificial-Intelligence-in-Government.pdf