Where, Oh, Where Can My Data Be? Four LLM Security/Privacy Tiers

Often when I talk to customers about using AI tools – especially ones that use Large Language Models (LLMs) – I inevitably get asked questions about the security and privacy of their data when using these models. It is not the question itself that surprises me (it’s a great and important question), but rather often a misunderstanding users have about how the AI tools and tool providers actually handle your data and the security/privacy options that are available to secure that data. Some customers have a knee-jerk reaction of “I will never use AI with my data – period.” But in this age of “accelerating everything” because of AI, such a viewpoint could put a business at a severe competitive disadvantage. 

So what actually happens to your data when you paste that data into an AI tool that uses an LLM? Where does it actually go and, more importantly, how secure/private is that data? In this article, I discuss these questions, present four broad tiers of how you can use LLMs, and discuss the spectrum of convenience vs. control over your data for each tier. Understanding these four tiers is the difference between using AI responsibly and quietly handing your proprietary information to a system you do not control or understand.

How not to train their model…

I often hear the phrase “your data can be used to train the model” thrown around a lot, so it is worth explaining what it means. In general, when you submit a prompt to an AI tool, that text can be retained, reviewed by humans for quality purposes, and used to improve (train) future versions of the LLM. In practical terms, your information can become part of the raw material (i.e., the model) that shapes a product you do not own.

For casual use of these tools, that is probably fine. Nobody really cares if a public brainstorm about weekend plans ends up in a training set. The problem starts when the input is a client contract, a patient record, proprietary source code, an unreleased financial model, or anything covered by a confidentiality agreement or a regulation like HIPAA. For these situations, the primary concern shifts from how convenient the tool is to the risk of data exposure.

I feel the real question here is not “should we use an LLM” but rather “how can we use an LLM that fits the sensitivity of this particular data.” At Solution Street, we have defined four broad tiers of how you can use LLMs, each with an increasing level of control over your data, to give a clean way to make that call.

The four security/privacy tiers

1. Free & Open Tier

This is usually the way most people are introduced to LLMs – through a free consumer version of a popular LLM-enabled chat tool. It costs nothing, requires no setup, and works instantly. Examples include the free versions of ChatGPT, Claude, and Google Gemini.

The tradeoff is there is usually no security or privacy here by default. Your data is used to make the underlying LLM better, which means anything you submit or upload is free game. You have no idea exactly how they will use your data. For throwaway data, this is usually not a concern. But for more sensitive data, it means you give up control on how that data is used and where it shows up in the future. Most of these tools do offer some way to opt out of using your data to train the models, but this is almost always not the default, and is often buried in some settings menu. In other words, they don’t want you to opt out for a reason.

Most organizations do not use or even allow this tier of LLM-enabled AI tool usage. But time and time again, we hear from organizations’ employees that restrictions on the AI tools their organization permits them to access causes them to sometimes use non-approved tools, sometimes even the free versions, to stay productive. This is a very real concern – often called Shadow AI – that can put your data at risk. While this topic of Shadow AI is a large topic in and of itself and out of scope for this article, we suggest that educating your entire organization about the risk of using this tier of tools, while also offering a practical pathway to consider and securely onboard new AI tools, is what we recommend here to help combat Shadow AI.

So what is the best fit for this tier?:

Gathering public information; general research; personal productivity; and experimentation where the input carries no confidentiality risk.

2. Pay & Trust Tier

The next step up uses the same popular tools (ChatGPT, Claude, Microsoft 365 Copilot, etc.), but on a paid tier that ensures privacy settings are turned on (with many paid tiers – especially the enterprise tiers – privacy protections are the default, though this varies by vendor and you should always verify the current setting/policy). In this case, your data still enters the provider’s LLM and environment, but the provider commits to protecting it and discarding it rather than training on it. You are, in effect, paying for a different set of terms than the publicly available free option.

The key word in this tier is “trust.” Your data protection depends on the trust in the provider and the provider’s policy. I had a customer that was perfectly fine with entering their most sensitive business data into Microsoft 365 Excel sheets (which by nature is hosted along with their data in the cloud by Microsoft), but would never trust putting this same data into a paid enterprise version of a Microsoft 365 Copilot session. While I am not saying that they should unequivocally put that data into Microsoft 365 Copilot, they should at least look at the risks objectively, understand how their data is handled by the LLM provider, in this case Microsoft, and understand the terms of the provider’s policy. Trust also often comes only when we learn more about how things work.

It is important to note that provider policies can (and do) change from time to time. Even if your data is not used to train the models, the provider often still retains your data temporarily for logging, caching and abuse prevention purposes (typically 30 days, but this can vary) and you should consult the current policy for specific terms on how they handle and retain your data. Consideration should be made to evaluate the risk to your organization if this retained data was ever hacked or accidentally released.

This tier works well when you have a provider you are comfortable relying on and data that is sensitive but not catastrophic if a policy changes. This is typically where we see most organizations start – and often stay at – in terms of balancing data security/privacy and productivity/cost.

So what is the best fit for this tier?:

Everyday business use with sensitive information, where a reputable provider’s contractual commitments are enough for your risk tolerance. The important point here is that you need to understand and periodically monitor the specific terms and policies from your provider regarding how they handle your data at this tier.

3. Cloud Private Tier

This next step up, Cloud Private, starts to put control over your data back to you as the organization. Cloud Private means accessing (or even running dedicated instances of) popular LLM models inside of a cloud environment, using services like AWS Bedrock or Azure Foundry. You get access to strong, well-known models, but they run within the cloud provider and are accessed through your walled section of infrastructure that you control.

You are in control of the cloud infrastructure and can control how your data flows and is stored. The core promise of this arrangement is that the LLM provider cannot train on your data. You are no longer trusting a consumer product’s settings; you are relying on enterprise cloud infrastructure with the security posture and compliance certifications that come with it. In this tier, we often see customers that are in heavily-regulated industries, have government regulations, and/or already run serious workloads in the cloud.

The tradeoff here is cost and complexity. You need cloud expertise, you take on configuration, management and operations, and you pay for the consumption or computation that you use. You are also usually responsible for providing the middleware and user interface for accessing these models; this can be provided through either open source or custom built software. The cost difference between this tier and the “Pay and Trust” could be substantially more expensive, so this is not a tier that we suggest customers take lightly or start with until they fully look at the cost/benefit. Solution Street is here to help with a full analysis of your specific needs if you are considering this tier.

So what is the best fit for this tier?:

Very sensitive or regulated data at organizations that already operate in AWS or Azure, have the budget and staff to support this tier of solution, and need strong (potentially frontier) models without giving up data control

4. Private Model Tier

At the far end of the data control spectrum is the Private Model tier. In this tier, you host and run an open-weight (or open source) LLM model – such as Llama – yourself on your own hardware or privately provisioned cloud compute/GPUs. You are in full control – the model runs in your environment, data never leaves your environment, and you are in control from end to end. This is the strongest possible answer to “where does my data go,” because the answer is simply “nowhere.” Or more specifically, wherever you want it to go; again you are in full control here. For the most sensitive workloads, that is often exactly what you want. 

An example of this is an internal tool we developed here at Solution Street to support business operations. We made the decision to go with a Private Model solution based on several factors: the sensitivity of the data, the ability to process this data satisfactorily using open-weight models (vs. more expensive frontier models), the atomic nature of the use case required, and our ability to operate this solution cheaply (actually on a well-provisioned Mac). It has worked very well over a couple of years now, however it is a very specific solution for a very specific problem.

The cost here is development and operations. You are responsible for the hardware, the model deployment, the software (user interface, middleware, data storage, etc.), the tuning, the scaling, and the ongoing maintenance. Open-weight/open source models are strong and improving quickly, but running one well is a real engineering commitment. Interestingly, this tier is not necessarily the most expensive option, specifically when the solution is implemented for a very specific use case (such as the one Solution Street implemented) and not a more general use case like an organization-wide generic chat solution (where the cost could be many magnitudes above the other tiers).

This tier is also only applicable when the private, open-weighted model is able to sufficiently process your workload and does not require the latest frontier model. Finally, special consideration should be made to the selection of the model itself, since even “free” and open-weight models may have restrictive terms and conditions and even carry some security concerns, such as has been raised recently with some of the non-U.S. models currently available. We typically only see this tier implemented in organizations that have the deep technical capability and the security requirements to justify the effort.

So what is the best fit for this tier?:

Highly sensitive, proprietary, or regulated data where full control is a hard requirement, the workload can be processed by open-weight (vs. frontier) models, and the organization has the technical capacity and budget to support it.

How to choose a tier?

The four tiers form a spectrum: as you go up the tiers, so does your security and control; but usually at the expense of convenience, cost, and effort. There is no universally correct tier, only the tier that matches a given piece of data.

A few factors to consider when making this decision:

• Data sensitivity. The more damaging a leak would be, the further up on the spectrum you should sit.
• Regulatory requirements. Compliance obligations like HIPAA or Government regulations often push you toward Cloud Private or Private Model.
• Budget and effort. The Free & Open tier is, well, free; the higher tiers cost money, deep expertise, and often both.
• Technical capability. Cloud Private and Private Model assume you have, or can hire, the skills to run them well.
• Scale. Heavy, ongoing, or changing patterns of usage can change the equation on which option is most economical.

In our experience, we have seen that most organizations do not pick one tier for everything. They match the selected tier(s) to the task and data requirements, for example using a Pay & Trust tool for most company-related business and a Private Model deployment for the crown jewels.

The bottom line

The goal here is not to find the single best tier but to match the tier and tools to the sensitivity of the data in front of you. If you view the four tiers as a spectrum rather than a single decision, the path becomes clearer; use the convenient/less costly options where the stakes are lower, and move towards the higher tiers of control as the stakes rise.

If you are weighing these tradeoffs and want help figuring out which option fits your data, your regulations, and your team, that is exactly the kind of problem we help clients solve. Reach out to Solution Street and we can work through it together.