Testing the AI Chatbots

With a range of AI Chatbots to choose from now, I decided to do my own test of them, to get an idea of the differences between them, also to test their accuracy and other features. This set of tests will also enable me to establish some ideas of how to test AI chatbots, as they’re now everywhere and may eventually replace traditional interfaces and even apps.

To be included in the tests, I established the following basic criteria for the AI chatbots:

must be available for free (or at least have a free version)
should ideally have an iOS app, on which I would perform the testing
also, ideally, should provide sources of its results, for provenance and to enable checking of answers

I narrowed the AI Chatbots to test down to the following ones:

Perplexity AI – iOS app
OpenAI ChatGPT – iOS app
Microsoft Copilot – iOS app
Google Gemini – as currently no iOS app available, tested via Chrome browser

I tested the AI Chatbots with a wide range of questions and these included:

simple questions with factual answers that can be determined as correct or incorrect
some trickier questions with factual answers to add some complexity
questions on specific areas of knowledge, such as movies, music, sport, even air fryers!
plus some questions where the wording is garbled to see if it can still get the meaning of the question
some whimsical questions, which do have a meaning, to see if it can grasp those and what kind of answer it supplies
some basic garbage questions to check error handling

To enable ratings and comparison between them, I established the following scores:

Ease of Use / Usability / UX / UI
Accuracy
Response time
Error handling
Source quotation/Provenance
Working with the response e.g. further options
Overall score

For consistency, I used the same set of questions across all the AI Chatbots and marked them accordingly based on the results and also made notes on any bugs or errors I found.

All the posts

I have created a post for each AI Chatbot test in order to fully explore the test results, a post which shows the questions asked, plus a Results post, where I crown the winning AI Chatbot. You can jump straight there or read each test post via the links below.

– Testing the AI Chatbots

– Testing the AI Chatbots: Questions

– Testing the AI Chatbots: Perplexity AI

– Testing the AI Chatbots: OpenAI ChatGPT

– Testing the AI Chatbots: Microsoft Copilot

– Testing the AI Chatbots: Google Gemini

– Testing the AI Chatbots: Results

Note: The heading images used in these posts were created via Bing Image Creator

About my testing services: iOS App Testing / Android App Testing / Website Testing / AI Chatbot Testing

We hired Pat to help with QA of a music streaming app. We needed someone with that magic “6th sense” of where the bugs are lurking; the ones which developers tend to miss. Pat more than fulfilled this need, with highly methodical approach and a great eye for detail.

Subsequently Pat has provided QA services for a personal app project of mine, finding a significant issue which had been lurking for some time.

Pat is a real asset to any team – highly competent technically but also easy going and a great guy to work with. Highly recommended!

Andrew Ebling, Head of App Development, Composed

Pat is an excellent one-stop-shop for testing Apps and Websites. He is a self-starter who just knows what needs to be done and gets on with it, and is responsive to requests for any specific testing requirements. He provides test plans and reports as a matter of course and was a pleasure to work with. I will definitely be working with Pat again if the opportunity arises.

Andrew Birley, PM, Storythings

We’ve worked with Pat Walsh on a number of very different projects, from interactive video projects to mobile apps. His work is very thorough and has quickly become an essential and important phase of all the production work we do at Storythings. He’s thorough, diligent, and a pleasure to work with.

Matt Locke, Director, Storythings

Pat Walsh provided Kingston Valdes with a professional and personable service that was excellent value. The site was built and tested quickly, we are delighted with the results, and we have already had positive feedback from some of our clients. If you’re thinking of building a new site or adapting an existing one, I’d certainly recommend speaking with Pat about it.

Kathy Valdes, Kingston Valdes

Pat Walsh fixed my site within hours of my contacting him – his WordPress website fix service clearly works!

Douglas Green, Douglas Software

Pat has been absolutely incredible at testing each of our projects he works on. He’s very professional, organised and diligent. I look forward to using him again in the future.

Oliver Thurtle, Director of Technology, Drum

Pat performs comprehensive website testing for us, across all the main browsers and platforms, including mobile devices. His flexible approach to testing and clear feedback enable us to have confidence that the business end of our website is still working smoothly after system updates and changes.

Steve Liu, Manager, Wicked Lasers

Pat came to Composed to respond to our (increasingly urgent) need of QA on mobile and web and I can testify he more than fulfilled the task. On the Android side, his very structured approach (first exploratory testing, then focus on defining test plans for a variety of scopes) and his clear knowledge of the latest mobile technologies have quickly paid off. He was involved directly with the developers (including myself) and this very dynamic relationship enabled us to annihilate bugs swiftly and efficiently.

David Ferrand, Android Developer, Composed

Couldn’t recommend Pat highly enough. He goes beyond what you’d expect of a QA tester, commenting more broadly on user experience and even design. He’s quick, efficient and thorough – couldn’t ask for any more.

Rob Calvert, Director, Click

Testing the AI Chatbots

Testing the AI Chatbots

Share this to social media...