Some tips and tricks to improve scaling¶
Calculate number of cores required for Fluent jobs¶
This section deals with figuring out the number for the --ntasks-per-node
flag, which may vary as per the user's requirement. This becomes crucial, as one of the primary objectives of utilizing a High
Performance Computing facility is to enhance performance by
minimizing job execution time. Selecting incorrect values for --ntasks-per-node
and nodes when submitting a job can lead to either a resource error or an increase in communication overheads. This, in turn, extends the job's execution time, which is undesirable, as the resources are contested.
For this, it is important to note is that Ansys recommends using 200,000 cells/core (for research licenses) and 100,000 cells/core (for student licenses) for parallel runs. This is a safe estimate, although users are welcome to calculate by tweaking this number as this is application-dependent too.
For instance, combustion and multiphase examples will use much more memory compared to a standard external flow aerodynamics or heat transfer problem, due to the additional equations. However, this is difficult for Ansys to advise as their customer base uses various hardwares with different RAM and CPU core-count configurations. Fluent also does not have an inbuilt memory estimator (yet), as given the variety and combinations of models, cell types, and material properties and edge-cases, this is not a trivial problem.
Therefore, as a catch-all rule, 200,000 cells/core for research license holders, and 100,000 cells/core for student license holders is the recommended metric.
Example¶
Take for instance the cells in the following example mesh are roughly 11 million. (Note, the number of cells in Ansys console can be obtained from the command, /report/mesh-size
).
>/report/mesh-size
number of interior nodes = 4893059
number of interior faces = 6790856
number of interior cells = 11165614
number of boundary nodes = 321278
number of boundary edges = 5145
number of boundary faces = 112069
For student licenses, this will translate to 11,165,614/100,000 = 111.65 cores ~112 cores, while for research license holders, this will be 11,165,614/200,000 = 55.82 cores ~ 56 cores. For all partitions, the following should follow,
License | VSC | Partition | Cores/node | --ntasks-per-node | --nodes |
---|---|---|---|---|---|
Education | 5 | zen3_0512, zen3_1024, zen3_2048 | 128 | 112 | 1 |
Education | 4 | skylake_0096, skylake_0384, skylake_0768 | 48 | 112 | 3 |
Research | 5 | zen3_0512, zen3_1024, zen3_2048 | 128 | 56 | 1 |
Research | 4 | skylake_0096, skylake_0384, skylake_0768 | 48 | 56 | 2 |
Here, please note that the memory request for these would vary, again depending on the applications.
Users need to consider the communication vs computation overhead
associated with their jobs. What this means is that if a user has not estimated the cells/core metric properly, the runtime could potentially be higher. For instance, taking zen3_0512
partition, if 2 nodes are used, but only 70 cores are chosen (say) as --ntasks-per-node
, while it can actually go upto 128 cores without extending nodes, the job will now be split across two different nodes, while remaining heavily under-utilized. This is attributed to the fact that Fluent must now keep track of and assemble results from two separate nodes, which will naturally increase the communication overhead. Simply put, more nodes or cores do not always mean faster results, as there will be a communication overhead that must be accounted for.
How can I know which Ansys license I can use?
On the login node, the user can do the following steps,
module load ANSYS/2024R1
/opt/sw/vsc4/VSC/x86_64/generic/ansys_inc/v241/licensingclient/linx64/lmutil lmstat
The response of the above will be something like, implying an ansys-r (research) licence.
(base) zen jz@l51:~$ /opt/sw/vsc4/VSC/x86_64/generic/ansys_inc/v241/licensingclient/linx64/lmutil lmstat
lmutil - Copyright (c) 1989-2023 Flexera. All Rights Reserved.
Flexible License Manager status on Wed 12/11/2024 13:46
License server status: 1055@ansys-r.it.tuwien.ac.at
License file(s) on ansys-r.it.tuwien.ac.at: C:\Program Files\ANSYS Inc\Shared Files\Licensing\license_files\ansyslmd.lic:
ansys-r.it.tuwien.ac.at: license server UP (MASTER) v11.19.5
Vendor daemon status (on ansys-r.it.tuwien.ac.at):
ansyslmd: UP v11.19.5
Fluent with GPU¶
Using GPU acceleration is usually discouraged for research license holders because there are only a few applications !1 within Ansys that benefit from this as per the Ansys customer forum and documentation. For the student licenses, they are not supported at all. Hence they are not recommended. The ideal method recommended for the same, at least for a local installation of Ansys, is briefly explained in this youtube video.
-
View factors related applications (Radiation). ↩