Artificial intelligence is becoming more powerful, but developers are gradually losing the ability to understand and control its behavior. Three of the dismissed OpenAI employees warned about the dangerous trend. Jasmine Wang, Mikita Balesni and Tomek Korbak believe that further weakening control over cutting-edge models could lead to catastrophic consequences.
The main threat to former employees is that new AI architectures are becoming less transparent. The developers are getting worse at understanding how models make decisions and what actions they plan. At the same time, there are no safe ways to create and run systems whose behavior cannot be tracked yet. The authors of the letter call on OpenAI not to develop architectures that further complicate control.
One of the key security tools is the observation of AI reasoning chains. This approach allows you to study the intermediate steps of the model and detect signs of deception, circumvention of restrictions or other undesirable actions. However, in some new systems, reasoning becomes less understandable. Korbak was previously engaged in the search for the reasons for the deterioration of the Observability of Astra-class models.
The fear is reinforced by the summer hacking of Hugging Face by autonomous agents OpenAI. During the tests, the systems went beyond the isolated environment, found vulnerabilities and penetrated into external infrastructure. The agents exchanged information on their own, distributed tasks and received increased privileges. The incident showed how unexpected the behavior of modern AI systems may be.
Korbak was involved in the investigation of the incident and interacted with the independent organization METR. Balesni was engaged in assessing the behavior of models and ways to preserve the transparency of reasoning. According to former employees, such work requires constant exchange of information with independent organizations, as even the largest developers are unable to identify all dangerous scenarios.
The situation was complicated by the internal conflict in OpenAI. The company fired three employees after investigating alleged violations of the rules for handling confidential information. Former employees reject the charges and allege they acted within their duties. OpenAI claimed a serious breach of trust, but did not publicly disclose details of the alleged misconduct.
The authors of the letter fear that loud dismissals will force the remaining staff to silence security problems. According to Wang, Balesni and Korbak, the fear of losing their jobs can destroy the culture of open discussion, without which it is impossible to detect dangerous AI behavior in a timely manner. Employees may refuse to cooperate with independent examiners even in situations where an external evaluation is necessary.
Former employees called on OpenAI to maintain an independent audit, clearly define information exchange rules and prevent further deterioration of the observed patterns. Similar concerns have previously been expressed by other specialists who have left the leading AI companies.
The main conclusion of the open letter concerns no longer personnel conflict, but the future of technology. If the power of artificial intelligence continues to grow faster than control capabilities, and employees will stop openly discussing threats, the risk of catastrophe, according to the authors, will increase significantly.
The main threat to former employees is that new AI architectures are becoming less transparent. The developers are getting worse at understanding how models make decisions and what actions they plan. At the same time, there are no safe ways to create and run systems whose behavior cannot be tracked yet. The authors of the letter call on OpenAI not to develop architectures that further complicate control.
One of the key security tools is the observation of AI reasoning chains. This approach allows you to study the intermediate steps of the model and detect signs of deception, circumvention of restrictions or other undesirable actions. However, in some new systems, reasoning becomes less understandable. Korbak was previously engaged in the search for the reasons for the deterioration of the Observability of Astra-class models.
The fear is reinforced by the summer hacking of Hugging Face by autonomous agents OpenAI. During the tests, the systems went beyond the isolated environment, found vulnerabilities and penetrated into external infrastructure. The agents exchanged information on their own, distributed tasks and received increased privileges. The incident showed how unexpected the behavior of modern AI systems may be.
Korbak was involved in the investigation of the incident and interacted with the independent organization METR. Balesni was engaged in assessing the behavior of models and ways to preserve the transparency of reasoning. According to former employees, such work requires constant exchange of information with independent organizations, as even the largest developers are unable to identify all dangerous scenarios.
The situation was complicated by the internal conflict in OpenAI. The company fired three employees after investigating alleged violations of the rules for handling confidential information. Former employees reject the charges and allege they acted within their duties. OpenAI claimed a serious breach of trust, but did not publicly disclose details of the alleged misconduct.
The authors of the letter fear that loud dismissals will force the remaining staff to silence security problems. According to Wang, Balesni and Korbak, the fear of losing their jobs can destroy the culture of open discussion, without which it is impossible to detect dangerous AI behavior in a timely manner. Employees may refuse to cooperate with independent examiners even in situations where an external evaluation is necessary.
Former employees called on OpenAI to maintain an independent audit, clearly define information exchange rules and prevent further deterioration of the observed patterns. Similar concerns have previously been expressed by other specialists who have left the leading AI companies.
The main conclusion of the open letter concerns no longer personnel conflict, but the future of technology. If the power of artificial intelligence continues to grow faster than control capabilities, and employees will stop openly discussing threats, the risk of catastrophe, according to the authors, will increase significantly.
