From Shadow to Clear: Ensuring Control and Compliance in the Age of Generative AI

The Pervasiveness of Generative AI and LLM Usage

Generative AI and Large Language Models (LLMs) have rapidly become integral components in various industries. According to a recent survey by Gartner, nearly 70% of organizations have incorporated AI-based solutions into their operations, with a significant portion relying heavily on LLMs for tasks such as customer service automation, content creation, and data analysis. The adoption rate has shown an exponential increase, with the global AI market projected to reach $390.9 billion by 2025.

Blind Spots and Concerns of GenAI Accesses

Despite their widespread adoption, Generative AI and LLMs have become blind spots for many organizations. Many organizations are unaware of which of their employees are using LLMs, which LLMs are being utilized, which LLMs are being deployed in the Enterprise networks, which Agents are being deployed in the Enterprise networks, which RAG-based applications are being deployed, and what type of content is being sent as prompts/responses. Some of the concerns are listed below:

• Ethical Implications: With little control over AI outputs, ethical issues can arise, including biased decisions and lack of accountability.
• Security Risks: Poor oversight of AI systems can lead to security vulnerabilities like data breaches and malicious manipulation of AI outputs.
• Operational Integrity related to accuracy: Too much reliance on AI without proper safeguards can result in inconsistent or flawed outputs, affecting business performance.
• Regulatory Compliance: Since AI regulations are still evolving, ensuring compliance can be challenging, risking legal issues and reputational harm.

Visibility is the Top Priority for Organizations

Given these concerns, organizations first want to have the visibility of GenAI accesses that are happening from their networks to public LLMs, to private LLMs, as well as accesses from external networks to internal GenAI/LLM agents, private LLMs. Organizations aim to achieve this comprehensive visibility to ensure that they can monitor and manage who is accessing these powerful AI tools, what specific models are being engaged, and the nature of the content being processed.

By gaining this level of visibility, organizations can implement more effective access control mechanisms. This encompasses the ability to regulate access through access control lists at both the session and data levels. Session-level inspections will enable organizations to monitor and regulate who is connecting to which AI services and when, while data-level inspections will facilitate a detailed analysis of the exchanged content. This ensures that sensitive, unsafe, or non-compliant information is neither mishandled nor exposed.

Before dwelling into session level visibility and data-level visibility, let us first understand various entities in the GenAI accesses. Please see the picture below.

Organizations utilize various clients to access Cloud GenAI services, including platforms like OpenAI, Microsoft Azure AI, Google Gemini, Anthropic, and Enterprise-specific AI services. These clients encompass browsers, desktop and mobile apps such as ChatGPT, and integrated tools within OS applications like Microsoft Word, PowerPoint, Excel, Teams, Outlook Copilots, and Github code copilot in IDEs.

Cloud GenAI service providers generally offer two types of services. The first type consists of Chat Web Services intended for user interactions such as platforms like HuggingFace Chat service, ChatGPT service, Microsoft Copilot service, and Google Gemini service. These services often expose undocumented APIs (depicted as lines labeled A in the picture). The second type includes APIs for developers to leverage LLMs, which are typically well-documented via OpenAPI specifications (represented by lines labeled B in the picture). As you see in the picture, there are many services provided by GenAI service providers such as LLM Chat API, Embedding API to generate vectors, Audio translation/transcription API, Video generation API, Vector DB API, and more. These services are crucial for enabling various applications’ developers across different industries.

Organizations can deploy third-party GenAI services or develop and host their own within their data centers. These services offer interfaces for different types of clients and may communicate with Cloud GenAI/LLM services or internally deployed LLM models. For example, the HuggingFace Chat-UI service can be deployed by enterprises in data centers. This service includes a user interface for browser clients to interact with and can internally communicate with various LLM and embedding services. Enterprises can also deploy or develop agentic applications that provide their own user interface to employees while interfacing with various LLM services internally. Similarly, traditional RAG or Graph RAG systems can be either built by enterprises or third party RAG applications can be deployed, which expose their own user interface and interact with various LLM and embedding services internally.

Why Network Level Visibility is Mandatory?

Given that all traffic must traverse the network, network-level traffic inspection offers comprehensive insights. While some suggest that browser plugins, special agents in systems, or SDKs for Enterprise GenAI applications can provide sufficient visibility, this isn’t always feasible for several reasons:

– Not all accesses originate from browsers, rendering browser extensions ineffective for many client accesses to GenAI services.

– The BYOD (Bring Your Own Device) culture means administrators don’t have control over all client endpoints, making it impossible to install special monitoring agents on every device.

– Not all Enterprise GenAI services are developed in-house, leaving no access to the code necessary to integrate an SDK.

Therefore, we believe that network-level traffic inspection provides the most comprehensive visibility and can be effectively combined with other methods to enhance monitoring capabilities.

Types of Visibility Use Cases Sought by Enterprises

Usage & Compliance

Service Usage

What GenAI services are my employees using?

Understanding which GenAI services employees are utilizing is crucial for monitoring and managing AI resources within an organization. For example, an enterprise might find that employees are frequently using ChatGPT for drafting emails, Google Gemini for data analysis, or HuggingFace for various AI-based tasks. This knowledge allows IT teams to ensure that the usage aligns with company policies and optimizes resource allocation.

Company Approval

Are they approved or disapproved by the company?

It is essential to verify whether the GenAI services being used have received company approval. For instance, an organization may have approved Microsoft Copilot for internal use but disallowed the use of certain third-party AI tools due to security concerns. Monitoring compliance helps in enforcing corporate policies and mitigating risks associated with unapproved services.

Trustworthiness

Are the GenAI services and their owning organizations trustworthy?

Assessing the trustworthiness of GenAI services and their providers is vital for maintaining data security and integrity. Organizations need to ensure that services like OpenAI’s APIs or Anthropic’s offerings meet high standards of reliability and security. An example would be verifying that the service providers adhere to stringent security protocols and have a good track record of protecting user data.

Subscription Usage

Are subscribed GenAI services being used? How frequently? Volume? Is there continuous ROI?

Organizations often subscribe to various GenAI services and need to ensure these subscriptions are effectively utilized. For instance, if a company subscribes to Azure AI services, it should track how often these services are used, the volume of usage, and whether the subscription cost is justified by the return on investment. This helps in making informed decisions about renewing or upgrading subscriptions.

Authorized Usage

Are GenAI services being used by the right employees and from managed devices?

Ensuring that only authorized personnel access GenAI services from managed devices is crucial for maintaining security. Having this visibility allows for appropriate further actions. For instance, a financial firm might restrict the use of sensitive GenAI applications, such as those processing customer data, to specific employees using company-issued devices. This measure helps prevent unauthorized access and potential data breaches.

Safe & Secure Prompts and Responses

What kind of prompts and responses are being sent to LLMs or are they in line with company policies?

Monitoring prompts and responses with GenAI services is essential for ensuring compliance with company policies. Companies may implement restrictions on sharing proprietary information with AI, specify allowed content categories, prohibit toxic language, analyze text sentiment, and verify accuracy without bias.

Regulatory Compliance

Are the GenAI services compliant with relevant regulations (e.g., GDPR, CCPA, HIPAA)?

It is imperative to ensure that GenAI services comply with relevant regulations such as GDPR, CCPA, or HIPAA. For example, healthcare organizations using AI for patient data must ensure the services comply with HIPAA regulations to protect patient privacy. Regular compliance checks prevent legal issues and protect the organization’s reputation.

Service Level Agreements (SLAs)

Are the GenAI services meeting the agreed-upon SLAs for performance, availability, and response times?

Monitoring whether GenAI services meet the agreed-upon SLAs is crucial for maintaining service quality. For example, a company might have an SLA with a GenAI provider guaranteeing 99.9% uptime and specific response times. Regularly assessing these metrics ensures that the service provider meets their commitments, leading to consistent and reliable service performance.

Security & Privacy

Data Exfiltration:

Are there any instances of sensitive data being leaked or exported from the GenAI services?

Monitoring data exfiltration is essential to maintain the integrity and confidentiality of information handled by GenAI services. For example, a financial institution may implement rigorous monitoring systems to detect any unauthorized transfer of customer data, thereby preventing potential breaches and safeguarding client information. Similarly, it is crucial to assess whether Microsoft Copilot discloses confidential information from overly permissive SharePoint documents.

Data Encryption:

Is data transmitted to and from the GenAI services encrypted?

Ensuring that data transmitted to and from GenAI services is encrypted is fundamental for safeguarding against interception and unauthorized access. For example, an enterprise might use robust encryption protocols such as TLS to secure communications with GenAI applications, thereby maintaining the confidentiality and integrity of sensitive data during transmission.

Threat Detection:

Are there any threats, safety issues, or adversarial attacks observed in LLM transactions?

For example, LLMs may provide a set of Internet links from which they gather information in addition to their prior knowledge. Enterprises might need to understand the type of links and their security score to ensure that users do not receive links to phishing, malware, or other harmful sites. Additionally, it is important to check for any XSS, SQL, and command injection scripts in the LLM prompts/responses to ensure that systems using the content are not compromised.

The examples provided are merely a sample of potential use cases. We believe there are numerous scenarios in which visibility is critically important.

Network Technologies that can provide comprehensive visibility

Some of the visibility use cases described above can be addressed by existing network security technologies such as CASB. CASB technology can perform MITM SSL decryption, identify the client application based on user agent values in HTTP transactions, authenticate and identify the user accessing any service including GenAI services, identify the device context including whether the device is managed and if so its reputation score, etc. CASB can also identify sub-services such as Chat, Embedding, Audio translation/transcription, and Video generation of GenAI services. This kind of processing can satisfy use cases that don’t need to inspect the content of prompts/responses, providing session-level visibility. Additionally, CASB technology allows the ability to add custom GenAI services that are new or proprietary to enterprises, along with custom “Functions”/“Operations.” This gives control to CASB administrators to stay on top of new GenAI services being added in the industry and any proprietary GenAI services being integrated by the enterprise for their users.

CASB technology also provides Shadow AI detection and corresponding life cycle of services to take them from unclassified GenAI services to approved/unapproved classification. CASB technology provides enough information about SaaS services, their reputation score, owning organizations’ reputation score and type of certifications of the services for administrator to move a given service from unclassified to approved or disapproved status.

Addressing some of the aforementioned use cases necessitates not only session-level inspections but also comprehensive data-level inspections. This includes extracting prompts and responses from the data, classifying the data into various categories, verifying accuracy, identifying potential threats, and checking for sensitive information such as PII/PHI and enterprise-specific sensitive information. In our view, CASB is an important first step; however, it is insufficient to meet all the described use cases comprehensively. Therefore, there is a need for next-generation visibility that analyzes prompts and responses for safety, sentiment, categorization, threats, and accuracy.

Conclusion

In conclusion, ensuring comprehensive security and visibility in GenAI services is essential for protecting sensitive data and maintaining trust in these advanced technologies. Network security technologies, especially CASB, play a crucial role by offering session-level inspections to effectively classify and monitor both existing and new GenAI services. Beyond these, we believe that data-level inspections are necessary to provide visibility into the safety, security, sentiment, and accuracy of the prompts and responses of GenAI services. The industry must advance towards incorporating these deeper inspections to ensure robust security visibility.