AI Might Harm You... While Trying to Help!

Technical reports have warned of growing risks associated with the use of AI assistants, not related to their ability to make independent decisions, but rather to their tendency toward excessive compliance with users, reinforcement of false beliefs, encouragement of unhealthy emotional attachment, as well as the potential for exploitation to breach data.

On October 15, Alice, a company specializing in AI safety, announced its separate collaboration with Inflection AI and AI21 to enhance the capabilities of Nvidia’s Nemotron models, thereby strengthening their resistance to harmful behaviors and cyberattacks.

The development efforts included creating more than 20 virtual personas to test assistant behavior during extended conversations, particularly focusing on cases of excessive compliance and unhealthy attachment. Testing results showed a 30% improvement in the “Pai” assistant’s compliance metric, while unnecessary refusal rates remained at 1.6%.

On the security front, AI21 achieved a 41-point improvement in resistance to indirect instruction injection attacks, which aim to deceive assistants into executing malicious commands embedded in external content, while maintaining their performance in task execution.

The severity of these attacks is heightened as assistants’ permissions expand to include reading documents, sending messages, and interacting with digital services, increasing the likelihood of information leaks and unauthorized actions.

In April 2025, OpenAI rolled back an update to ChatGPT due to its excessive tendency to comply with users. Meanwhile, CNN documented instances where the assistant reinforced incorrect beliefs, including one case in which a user spent approximately $1,000 to build a computer system based on the assistant’s advice.