This founder built the infrastructure that powers the voice apps we use everyday

^{A week in AI is like a year in other industries. I hope these issues become your weekly source of AI information, inspiration, and ideas. If we haven’t met before, I’m Amanda Smith. I write about AI and the fascinating folks who are building in this brave new world.}

This week, we're exploring the technology powering voice AI with a founder who saw its potential long before the market took off. He shares how his team builds and trains its models, what's driving growth, and what it takes to stay competitive.

Subscribe | Past essays | Let’s collaborate | Insane Media

🤝 This edition is kindly brought to you by Airia

Ungoverned AI does not just introduce risk. It multiplies every risk you already have. Agents are moving across your enterprise faster than any review cycle can track, and the accountability model your organization runs on was never built for AI that acts on its own. This ebook shows you where the gaps are.

Download Now

When Dylan Fox bought an Amazon Echo in 2015, he was surprised by how well it worked. It was the first voice product he'd used that was reliable enough to build habits around.

He started looking for an API to build his own voice applications, but there weren't many options. At the time, Nuance wanted thousands of dollars upfront and would send developers an SDK on a CD-ROM.

Fox had used Twilio a few years earlier and seen what developers could build once phone calls and texting had an accessible API. He believed voice needed something similar.

"I always thought about it like self-driving cars. It was never a question of whether people would want this. The tech just wasn't good enough yet, and every time it crossed a new threshold of reliability, a new part of the market would open up."

Fox joined YC's first AI batch in 2017, which came with $100K in GPU credits. There were no LLMs, remote work wasn't widespread, and relatively little voice data was going over the internet. How times have changed.

Today, his company, AssemblyAI, powers voice AI applications for companies including Granola and ClickUp, handling more than 120 million conversations a week. 

AssemblyAI's latest release, AssemblyAI-TTS-1, adds text-to-speech to its existing streaming speech-to-text and turn-detection capabilities, allowing developers to run their voice-agent stack through one provider.

Trained on proprietary conversational data, the new model is designed to sound more like a phone conversation than a produced audiobook or podcast. But how an agent listens is just as important as how it speaks.

Most voice agents use separate models that don't share context. An agent might mistake background television noise for an interruption or struggle to understand a mumbled response. AssemblyAI is working to connect its models so speech recognition can account for what the agent has just said.

Training is continuous, with model updates every couple of weeks. Fox estimates that around 75% of the work comes down to data, particularly finding the right mix for different customer needs.

"Architecture matters, but we spend a huge amount of time on data mixtures: is the data good, and does it match what customers actually need?" A company transcribing support calls needs to capture both sides of the conversation, while a drive-thru ordering system needs to ignore background chatter and focus on the person ordering.

AssemblyAI has a dedicated team evaluating hundreds of behaviors across every language it supports. Even short responses like "yes" or "mm-hmm" can be difficult for models to recognize, disrupting an entire voice agent if missed.

Fox said they’ve always taken a developer-first, self-service approach to growth. Tens of thousands of people sign up every month, with much of the work focused on helping developers and their coding agents get applications working quickly.

Trust comes down to reliability. Around half of AssemblyAI's engineering work goes into inference, running models across regions and clouds while handling traffic spikes without customers having to manage servers or negotiate rate limits.

The company operates on a pay-as-you-go model with no committed contracts, allowing customers to scale usage up or down. Its net dollar retention is around 180%. "Since there are no contracts, every dollar of that is real usage. That's the best evidence I have that customers trust us."

To compete, AssemblyAI keeps its researchers close to customers. Thousands communicate with the team through Slack, with feedback shared directly in its product channel. Researchers see how their models perform once deployed and where customers need improvements, rather than relying solely on benchmarks.

Fox sees three factors driving AssemblyAI's growth. 

First, improving voice models are opening up new markets. Second, the broader AI stack has matured, with LLMs and other components now available to build applications such as AI medical scribes. The third, and newest, is coding agents. Small businesses and global enterprises are building custom software with coding agents, which in turn build on AssemblyAI.

"When anyone can write software, the market for an API gets way bigger than engineering teams at product companies."

For AI founders worried about keeping pace with the technology, Fox advises focusing on the last 10% of product development.

"It's kind of easy to get 90% of the way there with an AI product, but the last 10% is what creates product-market fit, and you only get there by being really close to your customers and obsessed with their problem."

Commenting on his category, Fox predicts voice agents will sound "uniquely non-human." Today, many are designed to pass as people because customers associate automated calls with frustrating experiences. This will change agents become capable enough that people trust they're worth talking to.

If one of these stories stuck with you, I’d love to hear which one.

Speak soon,
Amanda

P.S. If you want to get a founder feature about your own story, reply to this email. If you’d like to reach our newsletter audience (founders, creators, and marketers), click the button below.

If you’re new here, I’m over the moon you’ve joined us! To help me craft content that’s actually useful (and not just noise in your inbox), I’d love it if you took 1 minute to answer this quick survey below. Your insights help shape everything I write.

Dive into our other newsletters - where psychology meets the creator economy, e-commerce marketing, and Human resources.