It sounds exactly like a person.Who asked for that?

This week I sat at my laptop and listened to a man chatting with a machine. They interrupted each other, laughed in the middle of a sentence and paused the way people pause when they are thinking. It was a demo of a new AI speech model, software that listens and talks back. Small labels on the screen said which voice was the human and which was the machine. Without them, I couldn’t have told you. I played a few more demos to be sure, and it didn’t get easier.

My first thought was: wow, this is really, really good. My second thought came a few seconds later: okay, and who needs this? Why exactly?

I have been asking that question ever since. The clearest answer I found is the one that scares me: phone calls where your daughter’s voice begs you for money, and it isn’t your daughter. At the end of this post are five simple steps that protect you and your family.

Meet Dessa

The model is called Dessa. It comes from Block, the American payments company formerly called Square. It listens and talks at the same time, like a person on the phone, and Block says it “laughs the moment something’s funny, mid-sentence, the way people do.” The company’s own summary: “Talking with it feels like talking with a person.”

Block says it runs on a powerful laptop, with no big computer centre behind it. You can’t download it yet. There is a waitlist, and a small group of AI safety researchers gets early access “to find the risks we haven’t anticipated.”

Block gives three reasons for building it: “curiosity (aka fun), better understanding between humans and AI, and safety.” By safety, Block means that today’s talking AI doesn’t notice when you are upset or in pain, and just keeps going. That’s a fair point. But the page says nothing about fraud. It also doesn’t say whether people on a call will be told they are talking to a machine.

I looked hard for the use cases

Here is every use I could think of or find:

• People who lose their voice, for example through ALS, a disease of the nerves that slowly takes away speech.
• Dubbing and translation, so a podcast can be heard in Spanish in the host’s own voice.
• Language practice, where you talk out loud without embarrassing yourself in front of a real person.
• Hotlines at night, when no person is on duty.
• Video games, where characters could talk with you and with each other in a way that feels alive. I can see that one: there the illusion is the whole point, and every player knows it’s a game.

Then I asked the same question about each one: does the voice need to pass as a human for this to work? Mostly it doesn’t. Someone who lost their voice needs a voice to speak with. They don’t need to trick anyone. A language learner needs a patient partner, and a hotline at night needs to answer the question. A voice that sounds a little like a machine can do both.

That left two uses where fooling people is the point: fraud, and making a machine feel like a person.

Fraud no longer sounds fake

In December 2024 the FBI warned that criminals “generate short audio clips containing a loved one’s voice to impersonate a close relative in a crisis situation”, asking for money or a ransom. One of its tips was to listen closely to the tone and the choice of words. That only works while the fake still sounds a little wrong.

People miss more fakes than you might think. In a 2023 study at University College London, 529 people listened to real and fake speech. They spotted the fakes 73 percent of the time, so about one fake in four got through. Showing them examples first “only improves results slightly”. Those were older models, and the listeners knew some fakes were included. On a real call at night, with a crying voice that sounds like your daughter, I doubt anyone does better.

It doesn’t stop at families. In early 2024, a worker at the engineering firm Arup in Hong Kong joined a video call with his company’s head of finance and several colleagues. All of them were fakes. He sent about 25 million dollars in 15 transfers.

For years I told myself that these calls catch older people who don’t hear well. I don’t believe that any more, because at this quality I think they can catch anyone, me included.

A machine that feels like a friend

The second use worries me almost as much. When OpenAI, the company behind ChatGPT, made its chatbot’s voice far more lifelike in 2024, it wrote in its own safety report that the risk of people getting emotionally attached “may be heightened by the audio capabilities”. Testers used language like “This is our last day together.” OpenAI also wrote that people “might form social relationships with the AI, reducing their need for human interaction.”

To be fair: a large study by MIT and OpenAI with 981 people found no clear difference between talking to a chatbot by voice and by text. It did find that the people who used the chatbot most came out worse, for example on loneliness and on depending on the chatbot.

Psychiatrists writing in the medical journal The Lancet Psychiatry this year warn that chatbots may feed false beliefs in people who are already at risk. Newspapers call this “AI psychosis”. It isn’t a diagnosis, and it’s unclear whether chatbots can cause it in healthy people.

We already treat chatbots like people, and I think a voice that laughs at your jokes and sighs when you’re sad makes it even easier to forget that nobody is there.

The good uses already exist, with locks on them

To my surprise, the best use on my list, a voice for people who are losing theirs, already exists. And it comes with limits built in.

Apple’s Personal Voice is made for people at risk of losing their speech. You read a set of sentences aloud that the phone picks for you. The voice is made and stored on your own device, locked behind your Face ID, Touch ID or passcode, and you can only make a voice that sounds like you. Clinics also help ALS patients record their own voice while they can still speak, a practice called voice banking.

When OpenAI released its realistic voices, it allowed only a few preset voices made with voice actors and added a filter that blocks anything else. Spotify translated podcasts with the hosts’ agreement, in their own voices.

Every good use I found works with your own voice and your consent, inside a closed system. None of them needs a voice that anyone can switch on at home and use to pretend to be a real person.

The problem is a laptop with nobody watching

Block’s boss, Jack Dorsey, called Dessa “small enough to run on your macbook”. Under his post, someone asked whether people will be able to run it on their own computers. His whole answer: “yes”. There is no download yet. But once a model like this sits on people’s own computers, nobody can turn it off again. There would be no way to see who misuses it, or to add a hidden mark to its audio that shows it was made by a machine. If there is a real use for it, release it for that use, with locks on it. Handing it to everyone opens a door to hell, and I don’t think we could close it again.

There is a story behind the name that I can’t stop thinking about. In 2019, a Toronto company also called Dessa made a fake voice of Joe Rogan, a famous American podcast host. It called the result “the most realistic AI simulation of a voice we’ve heard to date”. It also warned that “only a few seconds of audio are needed to create a life-like replica of anyone’s voice on the planet”. Then it kept the model to itself.

Square, the company that is now Block, bought that company in 2020. Two of the engineers behind the Rogan voice are on today’s Dessa team. In 2019 their company warned about scams; the 2026 page doesn’t mention them. I don’t know what changed, and I’d like to.

Dessa is the second model of this kind I’ve seen announced, and both times the replies on X, the platform that used to be Twitter, split the same way. I read through more than 200 replies under Jack Dorsey’s post. Lots of “wow” and “insane”, and people asking for access. And then the others: “Who asked for this???” “Robots should sound like f***ing robots.” “Our elderly will be so f***ed.” One person wrote that the examples are “insanely good” but “very dystopian”, and that they can’t think of a good use case “not involving AI psychosis”. Another simply joked that a lot of guys in San Francisco are about to start dating their MacBooks.

To be fair, some replies saw a real benefit. One person wrote that it could ease loneliness for people who get no human contact at all. I take that seriously. I’m just not sure a machine that hides what it is is the right answer to loneliness. Among all the questions asking why, I didn’t find a good answer to the why.

We’ve been here before

In 1952, Kent cigarettes got a new filter called Micronite. From 1952 until at least mid-1956, according to a study in Cancer Research, that filter contained crocidolite, a form of asbestos. A 1955 Kent ad promised “proof of greater health protection.” Or take leaded petrol: refinery workers died of poisoning from the lead additive in 1924, and it took until 2021 for the last country to stop selling it.

A voice model is nowhere near as deadly as asbestos. I still get the same feeling, that one day we’ll look back at this week and wonder why so many of us cheered.

Machines should say they are machines

A machine that talks to a person should say, at the start, that it’s a machine. Europe now agrees: since 2 August 2026, the EU’s AI Act says people must be told when they are talking to an AI system, unless it’s obvious. AI-made audio also has to carry a hidden mark that software can detect; systems that were already on sale get until December. That’s good, but a law doesn’t reach a model running quietly on someone’s own computer.

We changed our own rules this week too. SaaSolution, the free software directory behind this blog, no longer lists AI callers that don’t say at the start that they are an AI. The same goes for tools sold as sounding exactly like a human, and for copying someone’s voice without their consent. We removed two products under this rule. Read-aloud apps, dubbing and language tutors stay, because with them everyone knows a machine is talking.

“Don’t trust the voice.”

US Federal Trade Commission, consumer alert, 2023

Before the first fake call reaches you

1. Agree on a safe word with your family. The FBI says it in plain words: “Create a secret word or phrase with your family to verify their identity.” Pick something that has never appeared online, and agree on it face to face rather than in the family group chat. If someone calls in a panic and asks for money, ask for the word. No word, no money.
2. Hang up and call back. Use a number you already know is theirs, not the one that just called you.
3. Treat urgency as the warning sign. Money, secrecy and “right now” in one call is reason enough to stop, however real the voice sounds.
4. At work, never move money on a call alone. Agree that every payment request is confirmed a second way, for example in person or through a number you already have, even when the boss is on video.
5. Put less of your voice online. A few seconds of recording can be enough to copy a voice. The FBI suggests limiting public videos and voice recordings and making social media accounts private where you can.

I’m impressed by what Block built, and I’m also still waiting for an answer to my second thought. Until someone gives a better one than curiosity, I’d rather my family had a safe word.

Sources

• Block, “Dessa”, block.xyz/dessa (read on 24 October 2026); Jack Dorsey’s announcement on X, 23 October 2026, and the replies under it (read on 24 October 2026)
• Square press release, “Welcoming the Dessa Team to Square”, 7 February 2020; The Register and Vice on Dessa’s fake Joe Rogan voice, 17 May 2019
• FBI Internet Crime Complaint Center, “Criminals Use Generative Artificial Intelligence to Facilitate Financial Fraud”, PSA I-120324-PSA, 3 December 2024
• US Federal Trade Commission, “Scammers use AI to enhance their family emergency schemes”, 20 March 2023
• Kimberly T. Mai, Sergi Bray, Toby Davies, Lewis D. Griffin, “Warning: Humans cannot reliably detect speech deepfakes”, PLOS ONE, 2023
• CNN, “British engineering giant Arup revealed as $25 million deepfake scam victim”, 16 May 2024
• OpenAI, “GPT-4o System Card”, 8 August 2024
• Cathy Mengying Fang and others (MIT Media Lab and OpenAI), “How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use”, preprint, 2025
• Hamilton Morrin and others, “Artificial intelligence-associated delusions and large language models”, The Lancet Psychiatry, 2026
• Apple Newsroom, “Apple previews Live Speech, Personal Voice, and more”, 16 May 2023; Apple Support, “Create a Personal Voice”
• Spotify Newsroom, Voice Translation pilot, 25 September 2023
• William E. Longo, Mark W. Rigler, John Slade, “Crocidolite asbestos fibers in smoke from original Kent cigarettes”, Cancer Research, 1995; Stanford Research into the Impact of Tobacco Advertising, 1955 Kent advertisement
• William Kovarik, “Ethyl-leaded gasoline”, International Journal of Occupational and Environmental Health, 2005; UN News, 30 August 2021
• EU Artificial Intelligence Act, Article 50 and Article 113; Garrigues, “AI Digital Omnibus regulation has been published”, July 2026, on the transition period for marking