LLMOps, Observability, and Security & Privacy tools in AWS Marketplace (Part 3)

In this article, I will share the LLMOps, Observability & Monitoring, and Security & Privacy tools in AWS Marketplace from partners. This is 3rd in a series of articles on 'Classifying Generative AI Partner Offerings in AWS Marketplace'. Previous articles are here: Part 1: Classification, Part 2: Foundation Models & Vector Databases.

These two categories are extremely critical for achieving operational efficiency, increasing speed to market and ensuring responsible and safe use when putting Generative AI into production.

Large Language Model Ops (LLMOps) encompasses the practices, techniques and tools used for the operational management of large language models in production environments. (Source)

My observations with these tools include:

• These tools are critical to accelerating time to value for Generative AI projects and reducing operational burden.
• Tools will evolve in capability and numbers as we are in fairly early stages of Generative AI production deployments.
• Explainability remains a key blocker for many enterprises (particularly in highly regulated industries) and some of these tools provide that feature.
• Compatibility of these tools with other tools in the Generative AI stack will be an important decision criteria for enterprises.

LLMOps & Observability Tools (Count 7)

Fulfillment Method: SaaS

CRAFT AI’s MLOps Platform - Industrialize AI The MLOps platform that accelerates the deployment and management of Large Language Models in production.

Comet ML’s Comet MLOps Platform integrates with customer’s existing infrastructure and tools so they can reproduce, debug, manage, visualize, and optimize model - from training runs to production monitoring. By adding two lines of code to their notebook or script and they can automatically start tracking code, hyperparameters, metrics, and more, so they can compare and reproduce training runs.

Arthur’s Arthur: The AI Performance Company is the leading performance solution for all model types, including LLMs, CV, and NLP, with exclusive support for multi-class, multi-label, and tabular regression models that provides visibility into model performance with unique, actionable insights. Arthur AI offers firewall and evaluation capabilities for LLMs.

Abacus.ai’ Abacus.Ai Platform is the world's first AI-assisted data science and end-to-end LLM and MLOps platform that enables real-time deep learning and AI agents at scale for common enterprise use cases.

Seaplane.io’s Seaplane provides the fastest, easiest way for data scientists to deploy DS pipelines and leverage LLMs and other ML models without the need for MLEng or MLOps support.

Union.ai’s Union Cloud As a managed Flyte? solution, Union Cloud empowers data and ML teams to concentrate on ML production. It eliminates infrastructure constraints and complex setup processes, streamlining their workflow for 10x productivity.

Fulfillment Method: Amazon Machine Image

LangGenius’ Dify: Next-Gen LLMOps is an LLM application development platform that has helped built over 100,000 applications. It integrates BaaS and LLMOps, covering the essential tech stack for building generative AI-native applications, including a built-in RAG engine. Dify allows customers to deploy their own version of Assistants API and GPTs, based on any LLMs.

Security & Privacy Tools (Count 10)

While listed under a common category of Security and Privacy tools these tools fall into two sub-categories: 1/ Securing Generative AI applications and ensuring privacy (AIShield, AVM, Portal26, Airlock, MDACA, Private AI, Tonic Textual) and 2/ Generative AI applications to cybersecurity and compliance (MixMode, Blink, RiskAssessment AI).

Fulfillment Method: SaaS

Bosch Software & Digital Solutions AIShield.GuArdIan - Guardrail for Enterprise LLM adoption is a cutting-edge solution designed to support secure coding practices with generative AI, ensuring legal compliance and adherence to organizational policies.

Add Value Machine’s Enterprise Generative AI Security Hub helps customers meet both their compliance requirements and security needs. It operates as a control point for customer's LLMs, inspecting data such as PHI, PII, or PCI.

Portal26 is an award-winning Generative AI (GenAI) visibility and data security leader. Its Generative AI Visibility and AI Trust, Risk, and Security Management (AI TRiSM) SaaS Platform helps organizations embrace and accelerate the competitive promise of Gen AI throughout the enterprise. It serves as the essential foundation to provide organizations with the full visibility to understand Generative AI use - along with the governance, security, and education to manage it well.

MixMode’s MixMode Real-time Threat Detection and Response is a patented generative AI solution born out of dynamical systems that self-learns an environment without rules or training data. MixMode's AI constantly adapts itself to the specific dynamics of an individual network rather than using the rigid legacy machine learning models typically found in other cybersecurity solutions. The MixMode Platform is the only solution fully integrated with AWS capable of ingesting and analyzing large volumes of data within an AWS environment, including VPC Flow Logs and CloudTrail, to provide real-time threat detection and response.

Compliance.sh / RiskAssessmentAI’s Security Compliance Automated with AI makes it easy for organizations to get compliant with ISO 27001, SOC 2 Type II, HIPAA, GDPR and more with the help of AI. Built for startups, scale ups and enterprises.

Blink Ops’ Blink - Generative AI powered security automation copilot. By simply typing a prompt, any security pro can generate workflows to streamline operational tasks in AWS.

Fulfillment Method: Amazon Machine Image / Amazon ECS / Amazon EKS

Philterd’s Airlock is a quality control solution for AI applications to help prevent the disclosure of sensitive information, such as PII and PHI, and to control the sentiment and offensiveness of AI-generated text.

SpinSys’ MDACA PrivateGPT with Keycloak is an enterprise version of GPT that combines advanced AI capabilities with data privacy and customization. It empowers organizations with seamless integration, real-time assistance, and versatile applications to enhance productivity, decision-making, and customer service.

Private AI’s Private AI: Detect, Redact and Anonymize PII in 50+ Languages helps detect, anonymize, & replace Personally Identifiable Information (PII) with exceptional accuracy and supports 50+ languages, including French, German, Spanish and Japanese.

Tonic AI’s Tonic Textual is a sensitive data redaction platform and fully featured SDK for protecting your unstructured free-text data. Tonic Textual identifies sensitive datapoints from a block of text, redacts them, and synthesizes realistic text data to maintain data utility without compromising sensitive information. With just a few clicks Tonic Textual can be embedded into your data and ML pipelines to provide customers with realistic, de-identified text data that is safe to use for model training, MLOps, LLMOps, and building data pipelines.

Note: This list is current based on my review as of February 2nd. If there are any LLMOps, observability, security & privacy tools I am missing, feel free to mention that and I will make an update.