Microsoft’s AI-powered Bing demo included several minor bugs

Microsoft has been wowing the press and public alike with its new AI-powered Bing. But it turns out the technology made several mistakes during Microsoft’s public demo last week.

The errors included fabricated information about a Gap financial earnings report, which was noticed by engineer and writer Dmitri Brereton, who published a post(opens in a new window) going over the errors.

During the Microsoft demo(opens in a new window), which can be viewed online, the company asked ChatGPT’s Bing to provide key information on Gap’s third-quarter earnings report. The program continues to do so, except that the summary part is completely disabled.

(Credit: Microsoft)

For example, AI-powered Bing says Gap reported an operating margin of 5.9%. However, the company’s earnings report(opens in a new window) clearly states that the operating margin was 4.6%.

AI-powered Bing also says that Gap has projected net sales growth in the double digits. But the actual report states that “Net sales could decline in the mid-single digits year-over-year in the fourth quarter of FY 2022.”

In an earlier part of the demo, Microsoft also used the new Bing to query “What are the pros and cons of the 3 best-selling pet vacuums?” The search engine is quick to produce a result listing the pros and cons of three pet vacuums.

(Credit: Microsoft)

However, Brereton notes that Bing made a mistake in describing the Bissell Pet Hair Eraser handheld vac. It lists the product with a “short cord length of 16 feet” as a con. However, the vacuum is a cordless model designed to be portable. Also, Bing seems to provide an answer for the most recommended pet vacuums, not necessarily the top sellers.

Elsewhere in the demo, Bing made a mistake listing nightlife recommendations for Mexico City. The search engine says that an establishment called Cecconi’s Bar “has a website where you can make reservations and view the menu”.(opens in a new window) Make a reservation.

In addition, Microsoft also touted Bing’s ability to create a quiz about ’90s music. The search engine did just that, accurately listing the correct musician for each question. But strangely, all the answers to the 10-question quiz are “Answer A”, with no variation.

(Credit: Microsoft)

Microsoft did not immediately respond to a request for comment. But the company’s FAQ(opens in a new window) for the new Bing, technology won’t always be accurate. “Bing sometimes misrepresents the information it finds, and you may see responses that look convincing but are incomplete, inaccurate, or inappropriate,” the company wrote.

In fact, Microsoft’s own demo showcases the technology by synthesizing complex topics into easy-to-read articles and lists, for the most part. But in some cases the technology might be off – and it might be hard for a user to tell unless they check the response carefully. That said, Microsoft has previously said it’s working to refine ChatGPT’s integration with Bing.

Still, the accuracy of Bing and other AI-powered chatbots will undoubtedly face more scrutiny as they are released to the public. Google’s own ChatGPT rival Bard also mistakenly posted an inaccuracy in a public demo last week.