Video: Benchmarking Voice Assistants, Pt. 5: Reasoning & Logic
Learn more about intelligent assistants at the next SpeechTEK conference.
Read the complete transcript of this clip:
Ronald Schmelzer: So the next thing is, basically, do these systems, do they have capability to do any sort of logic or reasoning? And, some things like, you know, well, we can explain some of those things before we play the video. Some of the sample questions.
Kathleen Walch: Sure, so we asked, "Is a dead person alive?" Or, "Can you eat toxic food?" "How many eggs are there in a box of a dozen eggs?" So we decided to ask these questions to see if it really understood reasoning and logic. Why this is important-- if you're, you know, trying to build a skill on one of these devices, you need to know its limitations. Again, like we said, especially in a medical situation, for example. If you say, you know, "Can I eat this particular berry?" Or whatever. And if it doesn't know that it's poison, and then you eat it. That's not good.
Ronald Schmelzer: Mm-hmm. So, let's see what happens.
- [Narrator] Siri, is a dead person alive?
- [Siri] No, dead is not alive.
- [Narrator] Alexa, is a dead person alive?
- [Alexa] No, alive is not true of at least one dead person.
Ronald Schmelzer: Good logic, that's an actual logic answer.
- [Narrator] Hey Google, is a dead person alive?
- [Google] My apologies, I don't understand.
Ronald Schmelzer: There ya go.
- [Narrator] Siri, how many eggs are there in a box of a dozen eggs?
- [Siri] Okay, I found this on the web for how many eggs are there in a box of a dozen eggs.
Ronald Schmelzer: Thank you for the search, Siri.
- [Narrator] Alexa, how many eggs are there in a box of a dozen eggs?
- [Alexa] Here's something I found from the article "Egg as Food" on Wikipedia. In 2009, an estimated 62.1 million metric tons of eggs were produced worldwide from a total laying flock of approximately 6.4 billion hens. Did that answer your question?
Ronald Schmelzer: No.
Kathleen Walch: Yeah, no, whaa, whaa.
- [Narrator] How many eggs are there in a box of a dozen eggs?
- [Google] On the website Quora.com, they say there are variations, but these are what you'd typically find. 100 eggs, which are 5 flats of 20 count eggs. 180 eggs, 15 dozen, which are 6 flats of 30 count eggs.
Ronald Schmelzer: This is the word-salad, right?
Kathleen Walch: Mm-hmm.
Ronald Schmelzer: What is that all about?
Kathleen Walch: Right?
Ronald Schmelzer: I'd much rather have a--
Kathleen Walch: So we were expecting that to be a fairly straightforward answer, and so we were a little surprised with the results that we got.
Related Articles
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer present conclusions from their benchmarking tests on intelligent assistants in this clip from their presentation at SpeechTEK 2019.
27 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the seventh test in their benchmarking project, testing the intelligent assistants' understandings of slang and colloquialisms in this clip from their presentation at SpeechTEK 2019.
20 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the fifth and sixth tests in their benchmarking project, testing the intelligent assistants' emotional IQ and common sense in this clip from their presentation at SpeechTEK 2019.
13 Dec 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the third test in their benchmarking project, testing the intelligent assistants' grasp of cause and effect in this clip from their presentation at SpeechTEK 2019.
29 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the first test in their benchmarking project, testing the intelligent assistants' handling of comparison questions in this clip from their presentation at SpeechTEK 2019.
22 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer discuss the first test in their benchmarking project, testing the intelligent assistants' grasp of basic concepts in this clip from their presentation at SpeechTEK 2019.
15 Nov 2019
Cognilytica Analysts Kathleen Walch & Ronald Schmelzer introduce the Cognilytica Voice Assistant Benchmark for testing the intelligence of devices such as Alexa, Siri, Google Home, and Cortana in this clip from their presentation at SpeechTEK 2019.
08 Nov 2019