How Starya Built an AWS Inference Platform for Its Healthcare AI Agents with Elevata

As Starya's agents handled more conversations, language model usage became a major variable cost. At the start of the engagement, Starya reported spending approximately R$320,000 per month on third-party model services. Costs grew with the amount of information each model processed, and the team needed clearer visibility into the cost of serving individual customers.

Changing models also carried a product risk. An agent that schedules appointments must choose the right tools, work with the information returned by those tools, and communicate naturally with the person requesting help. A model that performs well on a general benchmark may behave differently in that workflow.

Starya needed to establish which models could support its own agents and whether dedicated GPU infrastructure could serve them reliably. The decision had to account for response quality, latency, workload volume, and infrastructure utilization together. The company also wanted greater control over the environment processing its AI workloads, particularly for healthcare applications.

Starya and Elevata worked through model evaluation, infrastructure testing, and a controlled production rollout.

The first evaluation compared 17 models using 95 examples from a representative Starya scheduling agent. The assessment examined tone of voice, tool use, safety, and proactivity, alongside response correctness and grounding: whether answers stayed faithful to the available information. Starya contributed its evaluation pipeline and experience operating the agents; Elevata provided the AWS inference environment and engineering support.

The evaluation identified open-weight models that could compete with the existing services on important quality measures. Gemma scored above the deployed baseline on correctness and grounding, while Qwen achieved the strongest grounding result in the tested group. Differences in tool use and conversational behavior helped the teams identify candidates for further validation in Starya's workflows.

Elevata then built the serving infrastructure on AWS. Open-weight models ran through vLLM on Amazon EKS, using GPU instances in Amazon EC2. Amazon S3 stored model weights. A shared model gateway connected NebulaOS agents with dedicated GPU inference, Amazon Bedrock, and other configured model services.

This gateway gave the teams a common point for routing requests, monitoring usage, and attributing consumption to individual customers. It also allowed the model and infrastructure serving a workload to evolve without requiring every customer application to move at the same time.

The teams tested the infrastructure before expanding production traffic. Load testing examined response times, capacity, routing overhead, and scaling behavior. KEDA and Karpenter provided scaling mechanisms for model replicas and GPU nodes, while monitoring helped the engineers understand how demand and long prompts affected performance.

The rollout introduced selected enterprise agents to the AWS GPU path, with alternative model services available through the gateway. The architecture also supported Amazon Bedrock as a primary route, giving Starya flexibility to adjust its model mix as quality, performance, and cost changed.

During the initial production rollout in late June 2026, six enterprise agents used the AWS GPU infrastructure. Starya reported that 90.97% of traffic for those migrated agents was served by Gemma on dedicated GPUs. This demonstrated that an open-weight model could handle most of the selected agents' traffic in a live customer environment.

The rollout also made the performance tradeoff visible. For those workloads, response latency was approximately 6.5 seconds on the GPU path, compared with 2.8 seconds on the API path. Starya accepted that tradeoff for the selected enterprise text agents. Model quality, response time, and infrastructure economics could now be evaluated against the needs of each workflow.

Starya paired its existing model-evaluation process with an AWS platform for running dedicated inference and a gateway that made customer-level usage easier to attribute. Combining this infrastructure with Amazon Bedrock gave the team more options for serving agents as their requirements changed.

For Starya, the engagement connected AI engineering with the economics of operating a product. The team could decide which models and serving paths suited each workload, measure the consequences, and refine the platform while continuing to develop its agents.

Learn more about Starya and NebulaOS at starya.ai.