Codex 日志错误可能导致数 TB 数据写入本地 SSD。
Codex logging bug may write TBs to local SSDs

原始链接: https://github.com/openai/codex/issues/28224

由于 Codex 的 SQLite 反馈日志机制,该程序正导致 SSD 出现过度的写入损耗。目前该应用程序默认将所有目标的日志级别设为 `Level::TRACE`,导致每年产生约 640 TB 的写入量——这足以在不到 12 个月内耗尽一块标准 1 TB SSD 的寿命额定值。 日志中充斥着来自内部依赖项(如 `inotify` 事件、`tokio-tungstenite` 内部组件)的高频冗余信息以及重复的 OpenTelemetry 事件。由于系统持续进行插入并立即清理数据的操作,底层数据库承受着严重的写入放大。 **拟议解决方案:** * **限制日志级别:** 取消 SQLite 接收端全局 `TRACE` 的默认设置,并提高高频、低价值目标的日志阈值。 * **优化数据:** 用汇总的遥测数据(如持续时间、成功状态、令牌使用量)代替原始的 Websocket/SSE 有效负载日志。 * **限制遥测:** 除非调试需要,否则停止持久化存储镜像的 `codex_otel` 事件。 * **实施上限:** 引入全局数据库大小或写入量限制,以防止日志失控。 * **配置项:** 提供一个明确的“出口”,允许根据需要完全禁用反馈日志记录。

OpenAI 的 Codex 出现了一个漏洞,可能导致系统向本地 SSD 写入数 TB 的数据。这一严重问题在 Hacker News 上引发了强烈反弹。 评论者们对 OpenAI 自身的 AI 工具引发严重技术故障这一讽刺现象感到沮丧。讨论的焦点在于“感觉编程”(vibe-coding)的广泛风险,以及对 AI 软件开发的过度依赖。批评者认为,这一事件凸显了旗舰 AI 产品在严谨的人工监管和测试方面的缺失。 许多用户对这样一个严重的问题在一周内未得到解决感到担忧,认为这证明了 AI 驱动的开发工作流程在缺乏严格质量控制的情况下,不仅难以维护,而且具有潜在危险。该讨论帖成为了一个警示:虽然 AI 可以提高生产力,但如果开发人员放弃标准的工程实践和监管,也可能导致灾难性的技术债务。
相关文章

原文

Issue

Codex is continuously writing a large amount of data to the local SQLite feedback log database:

  • ~/.codex/logs_2.sqlite
  • ~/.codex/logs_2.sqlite-wal
  • ~/.codex/logs_2.sqlite-shm

On my machine, after about 21 days of uptime, the main SSD has written about 37 TB. Process/file-level checks show Codex SQLite logs are the main continuous writer.

That extrapolates to roughly 640 TB/year. On a 1 TB SSD, that is about 640 full-drive writes per year. Some consumer SSDs are rated around 600 TBW, so this could consume roughly a full drive's warranted write endurance in less than a year.

Evidence

Current retained rows in logs_2.sqlite:

metric value
retained rows 681,774
estimated retained log content 1,035.6 MiB

Level distribution:

level estimated MiB byte %
TRACE 732.5 70.7%
INFO 266.5 25.7%
DEBUG 30.6 3.0%
WARN 5.9 0.6%

Largest target+level pairs:

target level estimated MiB
codex_api::endpoint::responses_websocket TRACE 527.4
codex_otel.log_only INFO 141.2
codex_otel.trace_safe INFO 121.2
log TRACE 97.4
codex_client::transport TRACE 60.1
codex_core::stream_events_utils DEBUG 27.5
codex_api::sse::responses TRACE 19.1

The top sources are mostly global TRACE logs, mirrored telemetry logs, and raw websocket/SSE payload logging. TRACE alone is about 70.7% of retained bytes. codex_otel.log_only + codex_otel.trace_safe add another 25.3%. Filtering these categories should remove roughly 96% of retained log bytes in this sample without fully disabling feedback logs.

Sanitized examples from the most frequent TRACE source: target=log

These are high-frequency retained samples. Raw websocket/SSE payload bodies are intentionally not included because they may contain private conversation content.

128,764x TRACE log: inotify event: ... mask: OPEN, name: Some("ld.so.cache")
 37,982x TRACE log: inotify event: ... mask: OPEN, name: Some("locale.alias")
 23,843x TRACE log: inotify event: ... mask: OPEN, name: Some("passwd")
  3,639x TRACE log: <tokio-tungstenite checkout>/src/compat.rs:131 AllowStd.with_context
  3,505x TRACE log: <tokio-tungstenite checkout>/src/lib.rs:245 WebSocketStream.with_context
  3,362x TRACE log: <tokio-tungstenite checkout>/src/compat.rs:154 Read.read
  3,356x TRACE log: <tokio-tungstenite checkout>/src/compat.rs:157 Read.with_context read -> poll_read
  3,230x TRACE log: <tokio-tungstenite checkout>/src/lib.rs:294 Stream.poll_next
  3,227x TRACE log: <tokio-tungstenite checkout>/src/lib.rs:304 Stream.with_context poll_next -> read()
  3,213x TRACE log: inotify event: ... mask: OPEN, name: Some("nsswitch.conf")
  2,001x TRACE log: WouldBlock
  1,217x TRACE log: Masked: false
  1,169x TRACE log: Opcode: Data(Text)
  1,169x TRACE log: First: 11000001
Sanitized examples from frequent INFO sources

The dominant INFO sources are mostly repeated OpenTelemetry mirror events. IDs are redacted.

843x INFO codex_client::custom_ca:
  using system root certificates because no CA override environment variable was selected ...

334x INFO codex_otel.trace_safe:
  session_loop{thread_id=<redacted>}:submission_dispatch{otel.name="op.dispatch.user_input" submission.id=<redacted> codex.op="user_input"}:turn{otel.name="session_task.turn" thread.id=<redacted> ...}

333x INFO codex_otel.log_only:
  session_loop{thread_id=<redacted>}:submission_dispatch{otel.name="op.dispatch.user_input" submission.id=<redacted> codex.op="user_input"}:turn{otel.name="session_task.turn" thread.id=<redacted> ...}

332x INFO codex_otel.log_only:
  session_loop{thread_id=<redacted>}:submission_dispatch{otel.name="op.dispatch.user_input_with_turn_context" submission.id=<redacted> codex.op="user_input_with_turn_context"}:turn{otel.name="session_task.turn" thread.id=<redacted> ...}

332x INFO codex_otel.trace_safe:
  session_loop{thread_id=<redacted>}:submission_dispatch{otel.name="op.dispatch.user_input_with_turn_context" submission.id=<redacted> codex.op="user_input_with_turn_context"}:turn{otel.name="session_task.turn" thread.id=<redacted> ...}

Write amplification

The retained DB size hides the real write volume. In a 15-second sample:

metric before after
retained rows 681,774 681,774
max row id 5,003,347,015 5,003,383,226

About 36,211 rows were inserted in 15 seconds, while retained row count stayed flat. This suggests continuous insert-and-prune write amplification: rows are inserted, indexed, written to WAL, then pruned.

Likely cause

The SQLite feedback log sink is installed with a global TRACE default:

Targets::new().with_default(Level::TRACE)

This persists all targets at TRACE level by default, including dependency/internal logs and large raw protocol payloads.

Proposed fix

Keep feedback logs enabled, but narrow what is persisted by default:

  1. Do not use global TRACE for the SQLite feedback log sink.
  2. Drop or raise thresholds for low-value dependency noise, especially target=log, hyper_util, tokio-tungstenite internals, inotify spam, and low-level OpenTelemetry SDK logs.
  3. Avoid persisting full raw websocket/SSE payloads by default. Store summaries instead: event kind, duration, success/error, token usage, and payload byte length.
  4. Avoid persisting mirrored codex_otel.log_only / codex_otel.trace_safe events unless they are explicitly useful for feedback debugging.
  5. Add a global logs DB size/write cap. Per-thread caps are not enough when many threads/processes exist.

An optional escape hatch such as sqlite_logs_enabled = false would still be useful, but the main fix should be better default filtering.

Related issues and discussions

联系我们 contact @ memedata.com