NY’s AI Chatbot for small businesses suggests them to breaks laws, steal wages

New York City's MyCity chatbot often touted as a tool to help local business owners and landlords navigate city laws, is under scrutiny after offering dubious legal advice and potentially encouraging lawbreaking, according to a report from The Markup.

Introduced as part of Mayor Eric Adams' "New York City Artificial Intelligence Plan" last October, the MyCity bot was hailed as a comprehensive effort to leverage innovative AI technology for the city's benefit. Powered by Microsoft, the bot aimed to assist small businesses and property owners in understanding complex legal matters.

However, investigations by The Markup revealed troubling inaccuracies in the bot's responses. Despite drawing information from the NYC Department of Small Business Services, the bot consistently provided erroneous advice, sometimes encouraging users to violate the law.

For instance, when asked about accepting meal vouchers, the bot incorrectly stated that buildings are not required to accept them, contradicting anti-discrimination laws. Similarly, it wrongly affirmed that employers could take a portion of employees' tips, and even steal wages overlooking labour regulations.

More from Tech

Furthermore, the bot's response to a question about firing an employee for disclosing financial information demonstrated a lack of understanding of employment law, potentially exposing businesses to legal risks.

A spokesperson for the NYC Office of Technology and Innovation defended the bot, citing its pilot phase and claiming it had provided thousands with accurate answers. They acknowledged the need for ongoing improvements but emphasized the bot's disclaimers on risks.

Despite assurances, experts caution against relying on the bot's advice, highlighting the potential legal ramifications for users. With concerns over misleading guidance, users are urged to seek professional legal counsel for accurate information.

While the MyCity chatbot aims to streamline legal inquiries for businesses, its shortcomings underscore the importance of caution and independent verification when seeking legal advice. Users should exercise prudence to avoid unintended legal consequences.

(With inputs from agencies)