Case Study on Optimizing AWS g6e-Based LLM Inference Batch Workloads of Neosapiens

Neosapience operates an AI actor service called Typecast based on AI voice synthesis and language intelligence technology.

Since its establishment in 2017, it has researched deep learning-based emotional expression and multilingual TTS technologies.

This post analyzes the optimization of lightweight LLM inference on Amazon EC2 instances, focusing on throughput and latency.

The combination of g6e (L40S) and INT8 was found to provide the best balance for operational conditions.

Neosapience aims to refine its optimal instance-precision combination in light of evolving AI product trends.