How Websites Detect Your Voice: Loudness, Voice Detection and Speech Recognition

Three very different jobs

When a website "listens", it can be doing one of three quite different things. It can measure how loud the sound is. It can decide whether the sound is a human voice rather than a fan or traffic. Or it can work out the actual words you said. These three jobs need very different amounts of computing power, and they have very different consequences for your privacy, so it is worth knowing which one a site is doing.

How sound reaches a web page

Your microphone turns air pressure into an electrical signal, and the sound card measures that signal thousands of times a second (commonly 44,100 or 48,000 times). Each measurement, called a sample, is just a number saying how far the microphone membrane moved at that instant. A web page that has your permission receives a stream of these numbers through a browser feature called the Web Audio API.

The browser handles this stream inside your device. A page can analyse the numbers, draw a wave, or pass them on, but nothing leaves your device unless the page's own code sends it somewhere. That is why the next question, what the page does with the numbers, matters more than the permission itself.

Job 1: measuring loudness

The simplest thing a page can do is measure volume. It takes a short slice of samples, say a twentieth of a second, and calculates their average size (engineers use the root mean square, or RMS). Loud sounds give big numbers; silence gives numbers close to zero. Converted to decibels, this is the value behind every volume meter you have seen in a video call.

Loudness alone tells a page that something is making noise, but not what. A clap, a dog bark and a sentence can all look alike. On the other hand, loudness measuring is extremely light, works offline, and reveals almost nothing about you: a single number that changes a few times a second contains no words, no voice print and no recording.

This is the approach Lootbeez uses for voice mining. The game reads the loudness level on your device to know that you are speaking, and the number is thrown away a moment later. No sound is saved or uploaded.

Job 2: telling a voice from noise

Voice activity detection (VAD) goes one step further. Human speech has typical patterns: most of its energy sits between roughly 100 and 4,000 hertz, it rises and falls in syllables a few times a second, and voiced sounds have a regular pitch. A VAD looks for these patterns, often by splitting the sound into frequency bands with a Fourier transform, and decides "speech" or "not speech" for each short slice.

Video call apps use VAD to switch off your microphone stream while you are silent, which saves data, and to show who is talking. Modern VADs are often small neural networks trained on many hours of speech and noise. They still run comfortably on a phone and, like loudness meters, they produce a yes or no rather than words.

No detector is perfect. Singing, laughing and loud television can count as speech, and very quiet speakers can be missed. That is why games and apps that react to the voice usually add simple rules on top, such as needing sound for a minimum time.

Job 3: understanding the words

Speech recognition, also called speech-to-text, turns speech into written words. It is far harder: the system has to deal with accents, background noise, mumbling and words that sound alike. Today it is done by large neural networks trained on huge amounts of transcribed audio.

Some recognition runs on the device, and phones have become very good at this. Much speech recognition, though, still sends audio to a server where bigger models run. Browsers offer a Web Speech API for recognition, and depending on the browser and device it may process the audio on a company's servers. A site that uses it should say so in its privacy policy.

Speech recognition is what voice assistants, dictation and automatic captions use. It is very useful, but it is also the job that involves your actual words, so it deserves the most care when you decide which sites to allow.

How to tell what a site is doing

• Read the privacy policy. A good one states plainly whether audio is recorded, sent to a server, or turned into text, and who processes it.
• Watch what the site does with your voice. If it reacts to the words you say (answers questions, types your sentences), it is doing recognition. If it only reacts to whether you are talking or how loudly, it is probably measuring loudness or using VAD.
• Look at the browser indicators. Browsers show a microphone icon or a red dot in the tab while a page is using the microphone. If it stays on after you leave a feature, close the tab.
• Use the permission settings. You can allow the microphone for one visit only, or block it, from the icon on the left of the address bar.

Why it matters

Voice features can be fun and accessible: they help people who find typing hard, they make games more physical, and they let you use a device with your hands full. Knowing the difference between a loudness meter and a speech recogniser helps you choose with confidence. A site that only needs to know you are talking should not need your words, and a trustworthy site will tell you exactly which of the three jobs it does.