One day, a scheduled task running on a 1C1G small instance failed to execute. Checking the logs revealed an OOM error.

I previously wrote about a quick combination of Java dump tricks, but that focused more on issues like deadlocks. This article focuses on memory OOM issues.

The log is as follows:

1
2
2025-01-10 11:55:01.155  INFO 7036 --- [ol-224-thread-1] c.p.bs.service.impl.ScheduleServiceImpl  : fetchAndSaveStockFundFlow schedule job start
Exception in thread "pool-224-thread-1" java.lang.OutOfMemoryError: Java heap space

The code at the time was written in a brute-force manner: the thread pool was unnamed, all relevant data was retrieved all at once, remote services were called to fetch data, and then everything was dumped into the database in one go.

Therefore, you could guess with your eyes closed that a small 1G instance is allocated only around 300MB of heap memory by default. Coupled with 5 threads configured to operate simultaneously for each task, OOM was only a matter of time even without a memory leak.

DUMP

The first step is to dump the memory snapshot, which is generally done using jmap or jhsdb.

1
jmap -dump:live,format=b,file=heapDump.hprof [PID]
1
jhsdb jmap --binaryheap --pid [PID]

Although other bloggers recommended using jhsdb, running it resulted in an OOM as well—a true nesting doll scenario.

After executing jmap, the heapDump.hprof file was generated in the execution directory.

Next, you need to find a way to export the file from the server. There are many methods, and scp works fine.

1
scp username@servername:/remote_path/filename ~/local_destination

You can also set up a file server using Nginx, which makes it convenient to download other data at any time. (Note data security issues: do not store sensitive data in the directory, as memory dumps can also expose sensitive data.)

Setting Up an Nginx File Server

First, install Nginx

1
yum install nginx

Create the public directory used by the file server

1
mkdir -p /var/www/

Note: Please do not create the server directory under the /root folder, as it will cause a 403 Permission Denied error.

Example error:

1
2025/01/10 15:25:15 [error] 30497#30497: *18 "/root/fileroot/index.html" is forbidden (13: Permission denied), client: IP, server: , request: "GET / HTTP/1.1", host: "IP"

Modify permissions and owner. Since the default user is root, sudo is omitted here.

1
2
3
# 修改权限和所有者
chown -R nginx:nginx /var/www/
chmod -R 755 /var/www/

Create Nginx configuration: /etc/nginx/conf.d/file.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
server {
  listen 80;
  location / {
    root /var/www/;
    autoindex on;
    autoindex_exact_size off;
    autoindex_localtime on;
    # 添加跨域设置
    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header 'Access-Control-Allow-Headers' 'Authorization';

    # 开启访问日志记录
    access_log /var/log/nginx/access.log;
  }
}

Start the service

1
systemctl start nginx

Move the dump result from the first step to the /var/www directory. Open your browser and enter the IP address to view the contents of the folder.

If it is a one-time server, simply use systemctl stop nginx to stop Nginx after downloading.

Analysis

There are many analysis tools, but IntelliJ IDEA can directly open and analyze hprof files.

Analyzing hprof
Analyzing hprof

You can see which DTOs have a high count. Obviously, the problem stems from repeatedly “retrieving all relevant data at once”, which generated too many StockBaseInfoEntity instances, occupying 159.8MB. As mentioned above, Java’s default maximum heap memory is 25% of the machine’s memory, making it very easy to hit an OOM.

Code Optimization

Staticizing Common Data

Since multiple tasks require the same common data, and the original code repeatedly fetched the common data in full, the memory continuously bloated.

Therefore, staticizing common data allows multiple read operations to point to the same memory location.

However, keep in mind: 1. Data needs to be initialized during project cold startup, and 2. Thread safety concerns.

Batch Partitioning

Change querying all, processing all, and writing all to operating in batches of 1,000 or 100 records at a time.

You can partition directly using Guava. To avoid adding new dependencies here, native for loops are used:

1
2
3
4
5
6
7
for (int i = 0; i < stockBaseInfoEntities.size(); i += Base.BATCH_SIZE) {
    Date lowestDate = fetchService.fetchAndSaveBsPointInstruct(stockBaseInfoEntities.subList(i, Math.min(i + Base.BATCH_SIZE, stockBaseInfoEntities.size())));
    if (lowestDate == null) {
        continue;
    }
    fetchService.calcLastSignalDaysAndCloseDiff();
}

Naming the Thread Pool

Here, Hutool’s NamedThreadFactory class was used directly.

1
private final ExecutorService instructExecutorService = Executors.newFixedThreadPool(BsInstruct.THREAD_NUM, new NamedThreadFactory("InstructExecutor", false));

Modifying JAR Runtime Parameters

Adjusting the heap size:

1
java -Xms128m -Xmx750m -jar yourapp.jar

Adding GC-related parameters:

1
java -Xms128m -Xmx750m -XX:+PrintGCDetails -jar yourapp.jar

After Optimization

338MB → 59.6MB

After optimization
After optimization

Appendix: Java Extra Options

Running java -X allows you to view all extra options and their explanations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
java -X

    -Xbatch           禁用后台编译
    -Xbootclasspath/a:<以 : 分隔的目录和 zip/jar 文件>
                      附加在引导类路径末尾
    -Xcheck:jni       对 JNI 函数执行其他检查
    -Xcomp            强制在首次调用时编译方法
    -Xdebug           不执行任何操作；已过时，将在未来发行版中删除。
    -Xdiag            显示附加诊断消息
    -Xfuture          启用最严格的检查，预期将来的默认值。
                      此选项已过时，可能会在
                      未来发行版中删除。
    -Xint             仅解释模式执行
    -Xinternalversion
                      显示比 -version 选项更详细的
                      JVM 版本信息
    -Xlog:<opts>      配置或启用采用 Java 虚拟
                      机 (Java Virtual Machine, JVM) 统一记录框架进行事件记录。使用 -Xlog:help
                      可了解详细信息。
    -Xloggc:<file>    将 GC 状态记录在文件中（带时间戳）。
                      此选项已过时，可能会在
                      将来的发行版中删除。它将替换为 -Xlog:gc:<file>。
    -Xmixed           混合模式执行（默认值）
    -Xmn<size>        为年轻代（新生代）设置初始和最大堆大小
                      （以字节为单位）
    -Xms<size>        设置初始 Java 堆大小
    -Xmx<size>        设置最大 Java 堆大小
    -Xnoclassgc       禁用类垃圾收集
    -Xrs              减少 Java/VM 对操作系统信号的使用（请参见文档）
    -Xshare:auto      在可能的情况下使用共享类数据（默认值）
    -Xshare:off       不尝试使用共享类数据
    -Xshare:on        要求使用共享类数据，否则将失败。
                      这是一个测试选项，可能导致间歇性
                      故障。不应在生产环境中使用它。
    -XshowSettings    显示所有设置并继续
    -XshowSettings:all
                      显示所有设置并继续
    -XshowSettings:locale
                      显示所有与区域设置相关的设置并继续
    -XshowSettings:properties
                      显示所有属性设置并继续
    -XshowSettings:vm
                      显示所有与 vm 相关的设置并继续
    -XshowSettings:security
                      显示所有安全设置并继续
    -XshowSettings:security:all
                      显示所有安全设置并继续
    -XshowSettings:security:properties
                      显示安全属性并继续
    -XshowSettings:security:providers
                      显示静态安全提供方设置并继续
    -XshowSettings:security:tls
                      显示与 TLS 相关的安全设置并继续
    -XshowSettings:system
                      （仅 Linux）显示主机系统或容器
                      配置并继续
    -Xss<size>        设置 Java 线程堆栈大小
                      实际大小可以舍入到
                      操作系统要求的系统页面大小的倍数。
    -Xverify          设置字节码验证器的模式
                      请注意，选项 -Xverify:none 已过时，
                      可能会在未来发行版中删除。
    --add-reads <module>=<target-module>(,<target-module>)*
                      更新 <module> 以读取 <target-module>，而无论
                      模块如何声明。
                      <target-module> 可以是 ALL-UNNAMED，将读取所有未命名
                      模块。
    --add-exports <module>/<package>=<target-module>(,<target-module>)*
                      更新 <module> 以将 <package> 导出到 <target-module>，
                      而无论模块如何声明。
                      <target-module> 可以是 ALL-UNNAMED，将导出到所有
                      未命名模块。
    --add-opens <module>/<package>=<target-module>(,<target-module>)*
                      更新 <module> 以在 <target-module> 中打开
                      <package>，而无论模块如何声明。
    --limit-modules <module name>[,<module name>...]
                      限制可观察模块的领域
    --patch-module <module>=<file>(:<file>)*
                      使用 JAR 文件或目录中的类和资源
                      覆盖或增强模块。
    --source <version>
                      设置源文件模式中源的版本。
    --finalization=<value>
                      控制 JVM 是否执行对象最终处理，
                      其中 <value> 为 "enabled" 或 "disabled" 之一。
                      默认情况下，最终处理处于启用状态。

GC log options, adapted from https://www.cnblogs.com/dupengpeng/p/17620200.html :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
-XX:+PrintGC <==> -verbose:gc  打印简要日志信息
-XX:+PrintGCDetails            打印详细日志信息
-XX:+PrintGCTimeStamps  打印程序启动到GC发生的时间，搭配-XX:+PrintGCDetails使用
-XX:+PrintGCDateStamps  打印GC发生时的时间戳，搭配-XX:+PrintGCDetails使用
-XX:+PrintHeapAtGC  打印GC前后的堆信息，如下图
-Xloggc:<file> 输出GC导指定路径下的文件中
-XX:+TraceClassLoading  监控类的加载
-XX:+PrintGCApplicationStoppedTime  打印GC时线程的停顿时间
-XX:+PrintGCApplicationConcurrentTime  打印垃圾收集之前应用未中断的执行时间
-XX:+PrintReferenceGC 打印回收了多少种不同引用类型的引用
-XX:+PrintTenuringDistribution  打印JVM在每次MinorGC后当前使用的Survivor中对象的年龄分布
-XX:+UseGCLogFileRotation 启用GC日志文件的自动转储
-XX:NumberOfGCLogFiles=1  设置GC日志文件的循环数目
-XX:GCLogFileSize=1M  设置GC日志文件的大小

Appendix 2: Viewing Heap Information with jhsdb jmap

1
jhsdb jmap --pid [PID] --heap
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
Attaching to process ID 31724, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 21.0.5+9-LTS-239

using thread-local object allocation.
Mark Sweep Compact GC

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 786432000 (750.0MB)
   NewSize                  = 44695552 (42.625MB)
   MaxNewSize               = 262144000 (250.0MB)
   OldSize                  = 89522176 (85.375MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 22020096 (21.0MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 63569920 (60.625MB)
   used     = 972208 (0.9271697998046875MB)
   free     = 62597712 (59.69783020019531MB)
   1.5293522471005154% used
Eden Space:
   capacity = 56557568 (53.9375MB)
   used     = 972208 (0.9271697998046875MB)
   free     = 55585360 (53.01033020019531MB)
   1.7189706601245656% used
From Space:
   capacity = 7012352 (6.6875MB)
   used     = 0 (0.0MB)
   free     = 7012352 (6.6875MB)
   0.0% used
To Space:
   capacity = 7012352 (6.6875MB)
   used     = 0 (0.0MB)
   free     = 7012352 (6.6875MB)
   0.0% used
tenured generation:
   capacity = 140910592 (134.3828125MB)
   used     = 38475656 (36.69324493408203MB)
   free     = 102434936 (97.68956756591797MB)
   27.305013380399394% used

Appendix 3: Viewing Class Memory Usage with jhsdb jmap histo

Sorted in descending order of memory usage. Note that it will print a large amount of information, so piping to more is recommended.

1
jhsdb jmap --pid [PID] --histo | more

Among the large pile of information above, only one is familiar. This entity is the staticized object mentioned earlier, confirming that the memory leak issue has been resolved.

histo
histo