Covering Disruptive Technology Powering Business in The Digital Age

DataStax, the generative AI data company, has announced it is supporting enterprise retrieval-augmented generation (RAG) use cases by integrating the new NVIDIA NIM inference microservices and NeMo Retriever microservices with Astra DB to deliver high-performance RAG data solutions for superior customer experiences.

With this integration, users will be able to create instantaneous vector embeddings 20x faster than other popular cloud embedding services and benefit from an 80% reduction in cost for services.

Organisations building generative AI applications face the daunting technological complexities, security, and cost barriers associated with vectorising both existing and newly acquired unstructured data for seamless integration into large language models (LLMs). The urgency of generating embeddings in near-real time and effectively indexing data within a vector database on standard hardware further compounds these challenges.

“In today’s dynamic landscape of AI innovation, RAG has emerged as the pivotal differentiator for enterprises building genAI applications with popular large language frameworks,” said Chet Kapoor, Chairman and CEO at DataStax. “With a wealth of unstructured data at their disposal, ranging from software logs to customer chat history, enterprises hold a cache of valuable domain knowledge and real-time insights essential for generative AI applications, but still face challenges. Integrating NVIDIA NIM into RAGStack cuts down the barriers enterprises are facing to bring them the high-performing RAG solutions they need to make significant strides in their genAI application development.”

DataStax and NVIDIA to Address Major Issues

DataStax is collaborating with NVIDIA to help solve this problem. NVIDIA NeMo Retriever generates over 800 embeddings per second per GPU, pairing well with DataStax Astra DB, which is able to ingest new embeddings at more than 4000 transactions per second at single-digit millisecond latencies, on low-cost commodity storage solutions/disks. This deployment model greatly reduces the total cost of ownership for users and performs lightning-fast embedding generation and indexing.

With embedded inferencing built on NVIDIA NeMo and NVIDIA Triton Inference Server software, DataStax AstraDB vector performance of RAG use cases running on NVIDIA H100 Tensor Core GPUs achieved 9.48ms latency embedding and indexing documents, which is a 20x improvement.

“Enterprises are looking to leverage their vast amounts of unstructured data to build more advanced generative AI applications,” said Kari Briski Cice President of AI Software at NVIDIA. “Using the integration of NVIDIA NIM and NeMo Retriever microservices with the DataStax Astra DB, businesses can significantly reduce latency and harness the full power of AI-driven data solutions.”

When combined with NVIDIA NeMo Retriever, Astra DB and DataStax Enterprise (DataStax’s on-premise offering) provide a fast vector database RAG solution that’s built on a scalable NoSQL database that can run on any storage medium. Out-of-the-box integration with RAGStack (powered by LangChain and LlamaIndex) makes it easy for developers to replace their existing embedding model with NIM. In addition, using the RAGStack compatibility matrix tester, enterprises can validate the availability and performance of various combinations of embedding and LLM models for common RAG pipelines.

DataStax is also launching, in developer preview, a new feature called Vectorize that performs embedding generations at the database tier, enabling customers to leverage Astra DB to easily generate embeddings using its own NeMo microservices instance, instead of their own, passing the cost savings directly to the customer.

“At Skypoint, we have a strict SLA of five seconds to generate responses for our frontline healthcare providers,” said Tisson Mathew, CEO at and Founder of Skypoint. “Hitting this SLA is especially difficult in the scenario that there are multiple LLM and vector search queries. Being able to shave off time from generating embeddings is of vast importance to improving the user experience.”

(0)(0)