One day, I discovered that all threads in the scheduled task thread pool were completely blocked:

Infinite failure blocking
Infinite failure blocking

The code is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
// 由于任务规模、执行频次较低，直接无界队列。如果高频任务建议new ThreadPoolExecutor
private final ExecutorService royalFlushExecutor = Executors.newFixedThreadPool(RoyalFlush.THREAD_NUM, new NamedThreadFactory("royalFlushExecutor", false));

@Override
public List<StockRoyalFlushDto> fetchRoyalFlushData(List<StockBaseInfoEntity> stockBaseInfoEntities) {
    String currentTime = DateUtil.currentTimeStr();

    List<StockRoyalFlushDto> results = new ArrayList<>();

    List<Future<StockRoyalFlushDto>> futures = new ArrayList<>();

    System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");

    AtomicInteger current = new AtomicInteger(0);
    for (StockBaseInfoEntity stockBaseInfoEntity : stockBaseInfoEntities) {
        String url = String.format(String.format(RoyalFlush.AI_URL, stockBaseInfoEntity.getCode() + RoyalFlush.QUERY));
        futures.add(royalFlushExecutor.submit(() -> {
            log.info("royalFlushData-当前进度:{}/{}-启动时间:{}", current.getAndIncrement(), stockBaseInfoEntities.size(), currentTime);
            return fetchAndParseData(stockBaseInfoEntity, url);
        }));
    }

    for (Future<StockRoyalFlushDto> future : futures) {
        try {
            StockRoyalFlushDto stockRoyalFlushDto = future.get(1, java.util.concurrent.TimeUnit.SECONDS);
            if (stockRoyalFlushDto != null) {
                results.add(stockRoyalFlushDto);
            }
        } catch (Exception e) {
            log.error("royalFlushData-任务执行失败:{} e:{}", e.getMessage(), e.toString());
        }
    }

    return results;
}

When trying to reproduce the issue, I found that Thread Pool 1 disappeared after a while:

Thread Pool 1 is gone
Thread Pool 1 is gone

Troubleshooting and Localization

Using ps aux | grep java can locate the process ID. Of course, right after INFO in the log is the process ID, which was 22385.

Use the top -H -p command to find the process:

1
top -H -p 22385
Stuck thread
Stuck thread

Use jstack to view thread states and stack trace information:

1
jstack 22385 | grep -C 50 22426
jstack results
jstack results

We can see that the thread is in the RUNNABLE state. Subconsciously, one might think a RUNNABLE thread is not executing, but RUNNABLE is actually divided into two states: Running and Ready.

In other words, a thread in the RUNNABLE state might be actively running, or it might not be executing and is waiting for CPU resource allocation.

Therefore, for a thread in the running state, when it runs halfway through a task and the CPU executing the thread is scheduled to do other things, the thread temporarily stops running. However, its state remains unchanged as RUNNABLE because it could be scheduled back at any time to continue executing the task.

Source Code Investigation and Fix

After locating the logic entry point, inspecting the underlying code of the dependent method revealed that timeout was -1, meaning no timeout:

Default timeout is -1, meaning no limit
Default timeout is -1, meaning no limit

Add a timeout setting above line 90 of the original code:

Timeout setting
Timeout setting

The issue was successfully resolved. In fact, the official documentation also prompts that modifications must be made to prevent bringing down the system.

Hutool official documentation prompt
Hutool official documentation prompt

Summary

  • Always set timeouts for remote calls
  • When threads in a thread pool are blocked, even if a timeout is configured when retrieving results, it only allows the retrieval logic to proceed—the executing thread remains blocked there
  • In the latest versions of Java, jstack allows querying without converting thread IDs to octal
  • It is crucial to log thread information—do not casually use sout to print information