We thank Greg Pereira and Robert Shaw from the llm-d team for their support in bringing llm-d to AWS. In the agentic and reasoning era, large language models (LLMs) generate 10x more tokens and compute through complex reasoning chains compared to single-shot replies. Agentic AI workflows also create highly variable demands and another exponential increase in …
Introducing Disaggregated Inference on AWS powered by llm-d Read More »
This post is cowritten with Ilija Subanovic and Michael Rice from Workhuman. Workhuman’s customer service and analytics team were drowning in one-time reporting requests from seven million users worldwide—a common challenge with legacy reporting tools at scale. Business intelligence (BI) admins faced mounting pressure as their teams became overwhelmed with these requests. By rebuilding their …
Building and managing machine learning (ML) features at scale is one of the most critical and complex challenges in modern data science workflows. Organizations often struggle with fragmented feature pipelines, inconsistent data definitions, and redundant engineering efforts across teams. Without a centralized system for storing and reusing features, models risk being trained on outdated or …
EAGLE is the state-of-the-art method for speculative decoding in large language model (LLM) inference, but its autoregressive drafting creates a hidden bottleneck: the more tokens that you speculate, the more sequential forward passes the drafter needs. Eventually those overhead eats into your gains. P-EAGLE removes this ceiling by generating all K draft tokens in a …
P-EAGLE: Faster LLM inference with Parallel Speculative Decoding in vLLM Read More »
As organizations scale their generative AI workloads on Amazon Bedrock, operational visibility into inference performance and resource consumption becomes critical. Teams running latency-sensitive applications must understand how quickly models begin generating responses. Teams managing high-throughput workloads must understand how their requests consume quota so they can avoid unexpected throttling. Until now, gaining this visibility required …
Deploying AI agents safely in regulated industries is challenging. Without proper boundaries, agents that access sensitive data or execute transactions can pose significant security risks. Unlike traditional software, an AI agent chooses actions to achieve a goal by invoking tools, accessing data, and adapting its reasoning using data from its environment and users. This autonomy …
Secure AI agents with Policy in Amazon Bedrock AgentCore Read More »
This post shows you how to build a scalable multimodal video search system that enables natural language search across large video datasets using Amazon Nova models and Amazon OpenSearch Service. You will learn how to move beyond manual tagging and keyword-based searches to enable semantic search that captures the full richness of video content. We …
Multimodal embeddings at scale: AI data lake for media and entertainment workloads Read More »
This post is a collaboration between AWS, NVIDIA and Heidi. Automatic speech recognition (ASR), often called speech-to-text (STT) is becoming increasingly critical across industries like healthcare, customer service, and media production. While pre-trained models offer strong capabilities for general speech, fine-tuning for specific domains and use cases can enhance accuracy and performance. In this post, …
Fine-tuning NVIDIA Nemotron Speech ASR on Amazon EC2 for domain adaptation Read More »
A new Google AI initiative aims to improve heart health outcomes for people living in remote Australian communities.
Agentic AI isn’t a feature you turn on. It’s a shift in how work is defined, who does it, and how decisions get made. Most enterprises learn this the hard way. They launch pilots that stall the moment they hit real processes, systems, and governance. The pattern repeats: vague use cases, prototypes that can’t survive …
Operationalizing Agentic AI Part 1: A Stakeholder’s Guide Read More »
