Hecate Usage Guidelines

Hecate does not have a queueing system.
We hope that we do not have to implement one.
Hecate is used by users from three departments: Astrophysics, Computer Sciences and the Princeton Plasma Physics Laboratory.
The user community has a size of about 20.
The environment is in general friendly and any anti-social behavior is prohibited.
The general rule is: users should not jam Hecate by either use more CPUs or more RAM that it has.

Each user should check the usage of CPUs and RAM using "top" and "pscpu" before starting a job.

We have had some recent job conflicts with people submitting jobs requiring large amounts of memory, large numbers of CPUs or both. In order to avoid such conflicts and to maximize the efficiency of using Hecate, we propose that every user of hecate obeys these guidelines .

Distribution of CPU and memory on Hecate
There are 64 processors available on hecate, numbered 0-63, combined into 8 modules and 32 memory nodes: 4 node cards per module, 2 cpus per node card.
16GB Ram total / 32 node cards = 500MB/node card
If you use the command "topology" you will see which module and node card each CPU is attached to. Here is how the machine looks face on:

module #
CPU # 2
8-15 4
40-47 6
24-31 8
56-63

module #
CPU # 1
0-7 3
32-39 5
16-23 7
48-55

It is advantagous to run, when possible, with modules that are close to one another to decrease the distance that each node will talk over. For example if you wanted to use 16 processors you would NOT want to work on cpus 0-7 and 56-63 simultaneously since these are the most widely separated modules.
A problem that has arisen lately is one where a single cpu user allocates a large chunk of memory on the same node card where other user's multi-cpu job is running. This is bad for both parties since the cpus compete for the memory on that node card. In order to prevent this from happening, every user should run his or her jobs, both serial and parallel, on specific processors. The following sections describe how to allocate specific CPUs for your jobs.

Running parallel jobs

First of all, users who wish to use large numbers of CPUs should first decide on which CPUs to run their jobs. Use one of the tools to check how the operating system is being used (ascii tools: top and osview or opengl/x-windows tools: gmemusage, gr_osview, gr_top).
Then, you start your job with either dplace or sysmp. An example of running dplace is this:
dplace -place pfile -mustrun a.out < in > out
where "a.out" is your executable, "in" and "out" the input and output files, and the file "pfile" looks like this:
```
#physical placement_file for 4 specific memories and 8 threads
memories 4 in topology physical near \
/hw/module/2/slot/n1/node \
/hw/module/2/slot/n2/node \
/hw/module/2/slot/n3/node \
/hw/module/2/slot/n4/node
threads 8
distribute threads across memories
```
This would run a code on 8 processors with 4 memory nodes, assigning them to module 2 (CPU 8-15). Of course, you need to modify the file to meet your CPU needs.

Running serial jobs
Serial jobs should be run with runon, which will keep your program running on the processor specified. The usage is this:
runon n a.out < in > out
where n is the CPU on which the job will run. Please make sure, using the system tools, that the CPU you choose is not already taken by another serial or parallel job.

module # CPU #	2 8-15	4 40-47	6 24-31	8 56-63
module # CPU #	1 0-7	3 32-39	5 16-23	7 48-55

Hecate Usage Guidelines

Distribution of CPU and memory on Hecate

Running parallel jobs

Running serial jobs