New York City’s initiative to assist small business owners with an artificial intelligence-powered chatbot has sparked controversy and criticism due to its dissemination of erroneous guidance that misrepresents local policies and even suggests violating the law.
Despite concerns raised by experts and tech news outlet The Markup, the city has opted to maintain the tool on its official government website, drawing scrutiny over its decision-making process.
Buggy System Raises Concerns
Launched in October as a “one-stop shop” for business owners navigating bureaucratic procedures, the chatbot offers algorithmically generated responses to user queries.
However, reports indicate that the tool continues to provide inaccurate information, prompting concerns about the risks associated with governments embracing AI-powered systems without adequate safeguards.
The chatbot has been criticized for offering false guidance on various legal and policy matters, including employment practices, waste management initiatives, and health regulations.
From suggesting it is permissible to fire workers for reporting sexual harassment to advising businesses to serve cheese nibbled on by rodents, the bot’s responses have raised eyebrows and highlighted the need for rigorous oversight.
Response from Microsoft and Mayor Eric Adams
Microsoft, the provider of the AI services powering the chatbot, has pledged to collaborate with city officials to enhance the accuracy and reliability of the tool.
Meanwhile, Mayor Eric Adams defended the decision to maintain the chatbot on the government website, characterizing the discovery of issues as a natural part of refining new technology.
Computer science professor Julia Stoyanovich and other experts have criticized the city’s approach, calling it “reckless and irresponsible.”
They caution against the potential consequences of relying on AI systems that lack proper oversight and accountability. Concerns about the pitfalls of large language models, like the one employed by the chatbot, have been voiced by scientists for some time.
Cautionary Tale for Public Sector
The incident serves as a cautionary tale for other cities considering similar AI-powered initiatives. Los Angeles, for example, has adopted a more curated approach to its chatbots, avoiding reliance on large language models to mitigate the risk of misinformation.
Suresh Venkatasubramanian of Brown University emphasizes the importance of clarity in the purpose and implementation of chatbot technology, urging cities to carefully consider the potential consequences.
As New York City grapples with the fallout from its flawed AI-powered chatbot, the incident underscores the importance of responsible implementation and oversight in the adoption of AI technologies by governments.
Image credits: Wallpapers.com
