Opened 3 years ago

Closed 3 years ago

#265 closed (fixed)

Cylc processes with high CPU usage

Reported by: Martin Dix Owned by: Martin Dix
Priority: major Component: Accessdev Server
Keywords: Cc:

Description

Top sometimes shows cylc processes with CPU usage consistently around 50%. The cause of this seems to be that they are continually writing files to cylc-run/SUITE/state, e.g. in this case at a frequency of about 1.5 s. Note that cylc keeps only the most recent 10 files

  69956 Mar  1 14:32 state.20160301T033212.313480Z
  69956 Mar  1 14:32 state.20160301T033213.906387Z
  69956 Mar  1 14:32 state.20160301T033215.450920Z
  69956 Mar  1 14:32 state.20160301T033217.007761Z
  69956 Mar  1 14:32 state.20160301T033218.547831Z
  69956 Mar  1 14:32 state.20160301T033220.227907Z
  69956 Mar  1 14:32 state.20160301T033221.708751Z
  69956 Mar  1 14:32 state.20160301T033223.677265Z
  69956 Mar  1 14:32 state.20160301T033225.011035Z
  69956 Mar  1 14:32 state.20160301T033226.531301Z

These files are all identical apart from the internal timestamp. In suites behaving normally these files are only written when something changes (e.g. a task finishes). Here however nothing seems to be changing at this high frequency. The log files don't show anything strange.

I've only seen this behaviour in complex NWP suites with lots of tasks. The high CPU usage can also come and go while the suite runs.

Seems more likely to be a cylc bug rather than anything to do with our local configuration?

https://groups.google.com/forum/#!topic/cylc/RyTpSfar61A

Change History (4)

comment:2 Changed 3 years ago by Martin Dix

Owner: set to Martin Dix
Status: newassigned

Still sometimes see high CPU usage (~30%) even with 6.9.1 suites, though there's no longer the constant writing of state files.

It seems to occur in APS3 trial suites when there are internally queued tasks, though it seems independent of the actual number of queued tasks. When there are no queued tasks the CPU usage drops back to normal (1-2%).

Simpler suites with queued tasks don't show the problem.

comment:3 Changed 3 years ago by Martin Dix

comment:4 Changed 3 years ago by Martin Dix

Resolution: fixed
Status: assignedclosed

Fixed in cylc 6.10.0.

Note: See TracTickets for help on using tickets.