使用关键内存优化解锁高性能PostgreSQL
Unlocking high-performance PostgreSQL with key memory optimizations

原始链接: https://stormatics.tech/blogs/unlocking-high-performance-postgresql-key-memory-optimizations

## PostgreSQL 内存管理:共享缓冲区 & Work_Mem PostgreSQL 利用内存来优化性能。**共享缓冲区**作为频繁访问数据的缓存;后续查询可以从这些缓冲区获得“命中”,从而减少磁盘读取。虽然在生产环境中进行一些磁盘 I/O 是正常的,但目标是避免其成为瓶颈——平滑的读/写和稳定的延迟是关键。存在工具可以检查共享缓冲区的使用情况。 **Work_mem** 是一个更微妙的设置,定义了查询中*每个操作*(排序、连接等)分配的内存。它“危险”在于会迅速增加:查询中的每个并行 worker 都可以消耗高达 `work_mem` 的内存,即使没有应用程序更改,也可能导致内存不足错误。 默认的 4MB 通常足以满足简单的工作负载,但分析查询可能需要增加。`work_mem` 不足会导致溢出到磁盘(临时文件、较慢的排序),而过大的值会导致内存压力。通过使用 `EXPLAIN (ANALYZE, BUFFERS)` 监控查询性能,识别基于磁盘的操作并相应地进行调整,来实现调优。

黑客新闻 新 | 过去 | 评论 | 提问 | 展示 | 招聘 | 提交 登录 解锁高性能PostgreSQL的关键内存优化 (stormatics.tech) 13点 由 camille_134 1小时前 | 隐藏 | 过去 | 收藏 | 讨论 指南 | 常见问题 | 列表 | API | 安全 | 法律 | 申请YC | 联系 搜索:
相关文章

原文

In the output, look for :

  • buffer hits – served from shared_buffers
  • buffer reads – pulled from disk

If you run the same query again, most of the time, the second run shows far more hits because now the pages are already in shared_buffers.

Important Note

In large production workloads, not everything can or should fit in memory. So you will see disk reads, and that’s normal. The goal isn’t that everything must be a cache hit. The goal is :

  • Disk I/O shouldn’t be your bottleneck, and 
  • Reads and writes should be smooth and 
  • Latency shouldn’t spike because the cache is too small or mis-sized

If you want deeper visibility into what is currently stored in shared_buffers or which tables are occupying memory, PostgreSQL gives you tools for that. Extensions like:

let you inspect shared buffers directly and understand memory usage patterns.

work_mem

After shared_buffers, the next memory parameter we need to focus on is work_mem.

And this is probably the most dangerous memory setting in PostgreSQL if you don’t fully understand how it works – not because it’s bad, but because it multiplies quietly. Many production outages caused by out-of-memory errors can be traced back to a misunderstanding of work_mem.

work_mem defines the limit or the maximum amount of memory allocated for executing operations such as:

  • Sorting, when performing operations like ORDER BY, DISTINCT, and GROUP BY.
  • JOINs usage (with hashing to build in-memory hash-tables, for example, for the hash join).
  • Set operations like UNION, INTERSECT, and EXCEPT.
  • Creating the bitmap arrays for the bitmap scan method

This parameter affects the efficiency of query execution and the overall performance of the database. It’s important to note that work_mem is allocated for each operation, not per the PostgreSQL session. This is a crucial detail, as a single SQL query can perform multiple sorting or join operations, each of which will consume its own area of memory. And some of these can be paralleized by PostgreSQL, and when that happens, each parallel worker uses up to work_mem per operation. If an operation runs sequentially, it can use up to work_mem. But if the same operation runs under a Gather node with, say, five parallel workers, then that single operation can consume:

5 × work_mem

This is exactly how databases run out of memory, even when the application hasn’t changed, because work_mem multiplies across:

  • Parallel workers
  • Multiple memory-intensive operations in a query
  • Concurrent queries running at the same time

This is why the most important thing to remember is that work_mem is per operation, and it can be used multiple times inside a single query, across many concurrent queries. 

How do we tune work_mem?

By default, PostgreSQL sets work_mem to 4MB. For many simple OLTP workloads with high concurrency, this is actually fine. But for analytical or reporting queries, 4MB is often too small. 


If the  work_mem is too small, PostgreSQL starts spilling to disk, and you’ll typically see:

  • Temporary files are being created
  • Sorts switching to disk-based algorithms
  • Increased disk I/O and latency spikes

If the work_mem is too large, it will cause memory pressure or worst OOM kills.

We can measure if work_mem needs tuning using:

EXPLAIN (ANALYZE, BUFFERS)
SELECT …
联系我们 contact @ memedata.com