Incident library · Government · United States

    New York City government: Chatbot later shut down

    An official city chatbot told business owners they could take workers' tips and refuse cash, both against the law.
    The MarkupMar 2024

    NYC's AI Chatbot Tells Businesses to Break the Law

    Chatbot later shut down

    Reported contributing factor

    A public chatbot gave legal answers that were wrong, and stayed online with a disclaimer after the errors were reported.

    Reported contributing factor

    The reported contributing factor

    A public chatbot gave legal answers that were wrong, and stayed online with a disclaimer after the errors were reported.

    Incidents usually have several contributing factors. This is the one public reporting names; it is not presented as the sole cause.

    Which StackUp assessment checks for this gap

    Find out if your organisation has this gap

    Sources

    Published Mar 2024. Sources last checked Nov 2026. Spotted an error or an update? Tell us and we will correct it.

    • Ars TechnicaApr 2025

      Company apologizes after AI support agent invents policy that causes user uproar

      Public apology and cancellations

      Stated by the organisation

      An AI agent answered customers directly without a clear label or human check, and invented a company policy.

    • Silicon RepublicJan 2025

      Apple calls a halt to AI notification summaries amid backlash

      AI feature suspended after complaints

      Stated by the organisation

      AI summaries were shown to users as if they came from the publisher, without a clear warning that they could be wrong.

    • The RegisterFeb 2024

      Air Canada must pay after chatbot lies to grieving passenger

      Held liable for its chatbot's answers

      Found by the tribunal

      The airline was found not to have taken reasonable care that its chatbot's answers were accurate.

    Find out which of these gaps you have

    The free Readiness Check takes ten questions and shows the incidents tied to your weakest answers.