AI chatbot giving “dangerously inaccurate” information

An AI chatbot launched by the New York City government to help business owners access relevant legal information is lying and, in some cases, even giving users false information about illegal activities, The Markup reports.

The report, published in collaboration with local nonprofit newsrooms Documented and The City, includes numerous inaccuracies in the chatbot’s answers to questions about housing policies, workers’ rights and other topics.

New York City Mayor Adams’ administration rolled out the chatbot in October, in addition to the MyCity portal, which launched in March 2023 as a “one-stop shop for city services and benefits.” Powered by Microsoft’s Azure AI, the chatbot targets existing and aspiring business owners and is promoted as a source of “actionable and reliable information” straight from city government sites.

However, as the city stated, this is only a pilot program and studies are ongoing. The disclaimer on the website also emphasizes that “inaccurate, harmful or biased content may sometimes be produced.”

Still, based on what The Markup says and the examples it gives, there’s a lot more misinformation in the tests than one might expect. In fact, one housing policy expert who spoke to The Markup said the chatbot was “dangerously buggy” at worst.