Human voices first: building a Shakespeare app with audio from real performances

When we set out to build a Shakespeare app with audio, the easy version was obvious: point a text-to-speech engine at the plays and ship a voice for every line by the weekend. We didn't, because the best way to experience Shakespeare is to hear a person perform it, and a person performing it is what Yorick plays by default. Its default voices are full-cast recordings made by LibriVox volunteers, cut speech by speech and matched to our text, and a human being listened to each of those speeches before it reached you.

This post is about why we made that choice, what it lets you do in the app, and how much of the work turned out to be people rather than software.

Why a performance beats a reading

Shakespeare didn't write books. He wrote parts, for actors to speak to a crowd that had paid to listen, and the text on the page is something like a score. Hamlet, of all people, says so: he spends the start of Act 3, Scene 2 coaching a company of actors, and his first instruction is about the voice, not the words: Speak the speech, I pray you, as I pronounced it to you, trippingly on the tongue.

You can read Lady Macbeth's sleepwalking scene silently and understand it. Hear someone actually say Out, damned spot! out, I say! One: two:, with the break in the voice where the counting starts, and you know what is happening to her in a way no footnote gives you. Pace, breath, the choice of which word to lean on: that is where the meaning lives, and it is exactly the part a synthetic voice is guessing at.

So the rule we built around is simple. Wherever a human performance exists, it is the default. Everything else in the app is there to help you get to it, or to help you give one yourself.

Where the voices come from

LibriVox is a volunteer project that records public-domain books and releases the results into the public domain. Its dramatic readings are something special: full casts of volunteers, each taking a part, recording separately and stitched together by a coordinator. Between them they have recorded 38 of Shakespeare's plays. Some readers are trained actors and some are not, and the recordings have the texture of real people doing something they love, which is the point.

The catch is that a LibriVox recording is an hour or three of audio per play, split by act. Yorick needs something much finer: this speech, by this character, starting here and ending there, so that when you are rehearsing Horatio the app can play Hamlet's line and then stop and wait for yours.

Matching the recordings to the text

Turning a long recording into speech-sized clips is a problem modern AI is genuinely good at, and we used it heavily. Roughly, for every play:

• Transcribe the audio with a speech recognition model, so every word has a timestamp.
• Work out who is speaking when, from the voices themselves.
• Line the transcript up against our text, speech by speech, in several passes: confident matches first as anchors, then the gaps between them, then a language model brought in for the hard stretches where the simpler matching gives up.
• Cut a clip for each speech and file it under its character.

That gets most of the way. It does not get all of the way, and the reasons are human ones. The volunteers read from whatever edition they had to hand, and our text is the Globe edition, so words differ. Some readers say the stage directions aloud and some don't. One volunteer often doubles several small parts. A messenger's two-word reply runs straight into the next speech with no pause at all. Speech recognition, faced with trippingly on the tongue, does its best.

Then a person listened to every speech

So we built a review desk: a clip editor that walks through a play one speech at a time, shows the expected text beside what the machine heard and the waveform of the recording, and plays the clip. If it is right, one key confirms it and moves on. If the start clips a breath or the end swallows the next character's first word, you drag the boundary and confirm. If it is hopeless, it gets replaced.

Across the 38 plays that came to 31,915 speeches, every one of them listened to. A little under half were right as the pipeline cut them. The rest, 17,182 of them, needed a hand: usually a few hundred milliseconds at one edge, sometimes a clip that belonged to the speech before. A hundred were beyond saving. Group speeches, the "All Witches" lines and crowd shouts, are getting a separate pass of their own, because deciding which of three voices on the recording is the Second Witch is a judgement call, not a matching problem.

The most useful lesson came the hard way. Once a person has reviewed a clip, the software must never quietly redo it. We found that out when re-running an alignment silently re-matched clips that had already been confirmed by hand. Since then, human corrections are the first thing the pipeline reads and the last thing it is allowed to change.

What a Shakespeare app with audio from real performances lets you do

All of that is invisible when it works, which is the idea. What you see is this:

• Listen to any scene, free, on this site. Every scene page in the reader has a Listen button on almost every speech, playing the volunteer performance of that exact speech. Start with Hamlet's O, what a rogue and peasant slave am I!, which is, fittingly, a speech about how an actor's performance can move people more than the real thing.
• Choose who reads each part. On Cast Your Characters you mark each role "Me" or "Yorick". For Yorick's roles you pick the sound: the default human recording, a community performance another user has shared, a generated voice, or your phone's own voice. Different roles can have different voices in the same scene.
• Perform against real voices. Take a part yourself and Yorick plays the other characters' recorded lines, then listens for yours. It follows along with speech recognition and is forgiving about it, so you don't need to be word-perfect to move on.
• Record your own, and share it. Your takes are kept line by line. Keep the ones you like, and if you choose to share them, your performance becomes one of the options other people can cast. The library of human voices grows with the people using it.

AI where it helps, people where it matters

None of this is an anti-AI position. Yorick is full of current AI, and we are proud of it: the speech recognition that follows you through a scene, the alignment that made the recordings usable at all, the translations, context notes and performance tips beside every speech, and an AI you can ask about a line, a character or a scene. Those notes and chats are AI-generated and labelled as such, and they are rehearsal aids, not scholarship. There is also a generated cast, one consistent synthetic voice per character, for anyone who wants a different sound.

But we drew the line in the same place every time. The AI does the tedious work and gets out of the way, so that the thing at the centre is a human voice: a volunteer's from LibriVox, another user's, or yours. A synthetic voice can say every word of Hamlet correctly. Only a person can decide what it means to say it.

If you have never heard a scene performed, the reader is the place to start: pick a play, press Listen, and follow along. Our guide to which Shakespeare play to start with has a scene to open in each of the six. And if a part catches you, try saying it back.