Frequently Asked Questions

If you only need to use a different shell temporarily and don't want to fully change your own shell, you can initiate a new shell by invoking it (e.g. with the tcsh or bash command, followed by pressing 'Enter'). Alternatively, you can pass commands as arguments to the shell language (e.g. tcsh -c "echo hello").

For ease of e-mail navigation and deletion, we have adopted four standard headlines for all of our messages:

• CHPC DOWNTIME - for announcements related to planned or unplanned computer downtimes
• CHPC PRESENTATION - for CHPC lectures and presentations
• CHPC ALLOCATIONS - for announcements regarding the CHPC resource allocation
• CHPC INFORMATION - for announcements different than the three above

This is either due to planned or unplanned downtime or a localized failure of the login server.

In the case of a localized failure of the login node (e.g. the node you are immediately connected to when accessing CHPC systems), please attempt to connect to another CHPC node (e.g. notchpeak2 instead of notchpeak1, or a different cluster altogether). If the error persists, please make us aware by contacting us at helpdesk@chpc.utah.edu.

If you are attempting to connect to CHPC systems through the FastX software or through Open OnDemand, and your login attempts are failing, the most likely culprit is your home directory being over quota- both FastX and Open OnDemand write to your home directory.

To fix this, you need to SSH to CHPC clusters (without FastX) and clean your home directory.

With this, your lockout could be initiated by CHPC systems OR the UofU campus.

In the case of lockout from CHPC systems:

These time parameters are not made public for security reasons and vary between CHPC environments and the campus authentication, but, in general, the period of disablement is an hour or less from the last failed login attempt.

The best approach to deal with this is to try to log into a different machine (e.g., kingspeak2 instead of kingspeak1, or a different cluster altogether) or wait until login is enabled again on the login node/cluster you were first attempting to access. It is counter-productive to try to log in again to the same login node, even with the correct password, as the disablement timer will reset.

In the case of lockout from UofU campus:

To check if campus authentication works, try any of the CIS logins. If you cannot successfully log in to CIS, contact the campus help desk and inform them of the AD SSH lockout and ask them to unlock you.

Some access tools like FastX and Open OnDemand rely on storing small files in the user's home directory upon logging in; access to these tools will fail when your home directory is over quota.

• If your storage is over 50GB, delete or move unnecessary files.
  • To find what files/directories are taking the most space in your home directory, run the ncdu command in your home directory to show disk space used per directory (the largest directory will be at the top).
• If you want to keep your files permanently, explore other storage solutions at CHPC 

• To map network drives to CHPC file servers
• When using a remote desktop to connect to CHPC machines 
• When connecting to the Protected Environment resources

Below details the naming schema for each partition. The naming schema is a quick way to understand 1) the type of resources available on that partition and 2) your level of access to that partition.

Note that, below, <cluster> refers to any of the CHPC clusters (currently Granite, Notchpeak, Kingspeak, Lonepeak, and Redwood).

• Partition: <cluster>
  • Only contains CPUs.
• Partition: <cluster>-guest
  • Only contains CPUs.
  • This partition allows for jobs to be run in 'guest-mode' on owner nodes. 
  • Jobs run on owner nodes are subject to preemption.
• Partition: <cluster>-gpu
  • Contains CPUs and GPUs.
  • To use a GPU, add --gres=gpu to your Slurm command.
• Partition: <cluster>-gpu-guest
  • Contains CPUs and GPUs.
  • This partition allows for jobs to be run in 'guest-mode' on owner nodes. 
  • Jobs run on owner nodes are subject to preemption.
• Partition: <cluster>-dtn
  • This partition is connected to our Data Transfer Nodes and is useful for transferring large amounts of data at fast speeds, high performance, and low latency.
  • Do not run computational jobs on the dtn nodes. They are for data transfer only.
• Partition: <cluster>-freecycle
  • Your job should run on the freecycle partition only if you ran out of an allocation for the Granite, Notchpeak, or Redwood clusters.
  • Freecycle jobs do not require an allocation, but are subject to preemption.

This error message can indicate one of three things: 

1. 1. You supplied the incorrect account name or account/partition information to Slurm.

      • Solution: If you suspect that you supplied the incorrect account name or account/partition information to Slurm, you can check your allocation with the  mychpc batch command.

   2. You are attempting to run a job on the Notchpeak, Granite, or Redwood general partitions, but your group does not have an allocation or has used all of their allocation for the current quarter.

      • Solution: To view your allocation status and see if you have run out of allocation for the current quarter, see your group information on the CHPC Portal. Your group may want to consider completing a small allocation request to supply your group with a small allocation for the remainder of the quarter. Submitting a larger allocation request will automatically grant you a small allocation for the current quarter (if you do not have one) and will put your large request in for the following quarter.

   3. The partition you selected is on a cluster different from the cluster you are submitting the job from.

      • Solution: Either update the partition, account, and QoS combination requested or use the -M flag in your sbatch command. e.g., if you are logged into the Notchpeak cluster, but are using the kingspeak partition, you can submit your Slurm script (called MySlurmScript.sh in this example) with this command: sbatch -M kingspeak MySlurmScript.sh

Your job is requesting more nodes than is permitted per cluster. The request limit is typically about half of the total general nodes available, e.g., 32 on Notchpeak, 24 on Kingspeak, and 106 on Lonepeak. Your job will not start unless an exception is approved via a special reservation or QOS.

This is typically because 1) the node is either down and requires maintenance or 2) your job's requested walltime overlaps with an existing reservation on that node, e.g. for a scheduled downtime. Check the CHPC status page for updates on scheduled downtimes.

While interactive jobs (srun) will fail immediately, batch scripts (sbatch) will enter the queue showing a ReqNodeNotAvail reason code.

1. High Resource Requests: Requesting excessive memory, CPU cores, or long wall-times can lower your priority or restrict your job to a few available nodes. Reducing these requests can help your job start sooner.
2. Resource Availability: A status of (Resources) indicates the scheduler is waiting for nodes that meet your specific requirements. Note that general availability charts may not account for node fragmentation.
3. Queue Priority: A status of (Priority) means other jobs with higher priority scores are ahead of yours in the queue.

The CHPC sometimes has to perform necessary maintenance on compute nodes to keep them in a health state - for instances such as this, we put reservations in place to prevent jobs from running during maintenance.

As a maintenance window approaches, you may notice that jobs will not be started by the scheduler, Slurm, if they overlap with the reservation window. This is intentional: it serves to ensure that no jobs are running when the clusters are undergoing maintenance, which would cause running jobs to be interrupted and could potentially result in incomplete or corrupt files.

Jobs that will finish before maintenance begins (i.e., jobs that do not overlap with reservations) will still be started by Slurm, while longer jobs will remain in the Slurm queue until maintenance is complete and reservations are released. If you see a message like ReqNodeNotAvail, May be reserved for other job or ReqNodeNotAvail, Reserved for maintenance when submitting or viewing the status of a job, please try shortening the runtime of the job (if possible for your job) so it does not overlap with a maintenance-related reservation.

Hostname       Partition     Node Num_CPU  CPUload  Memsize  Freemem  Joblist
                            State Use/Tot              (MB)     (MB)  JobId User ...
   kp016      kingspeak*    alloc  16  16    1.00*    64000    55494  7430561 u0123456  
   kp378      schmidt-kp    alloc  28  28    1.00*   256000   250656  7430496 u0123456  

 Another possibility is that the job is running low on memory, although we now limit the maximum memory used by the job via Slurm so this is much less common than it used to be. However, if you notice low free memory along with low CPU utilization, like in the example below, try to submit on nodes with more memory by using the #SBATCH --mem=xxx option.

The memory you request enforces strict memory limits on cluster nodes to ensure fair access and stability. In rare cases where Slurm's memory checks fail to catch an over-limit job, excessive memory consumption can degrade node performance or cause it to crash entirely, triggering a system administration warning.

1. Monitor Live Resource Usage: Run the pestat command in your terminal to check the memory load of your active jobs.

2. Inspect the Node Directly: If memory consumption appears critically high, SSH directly into the affected compute node and run the top command. Look for high memory utilization alongside kswapd processes consuming CPU time, which indicates that the program is using too much memory.

3. Cancel Problematic Jobs: If a runaway job is overwhelming a node, cancel it immediately to prevent crashing the system.

4. Adjust Your Memory Allocation: Modify your submission script to request appropriate memory using the #SBATCH --mem=xxx directive, or optimize your code to reduce its memory footprint.

#!/bin/bash
#SBATCH --nodes=12
....
JOB_TASKS=`echo $SLURM_JOB_CPUS_PER_NODE | sed -e 's/[()]//g' -e 's/x/*/g' -e 's/,/+/g' | bc`

mpirun -np $JOB_TASKS program_to_run

This error can be a result of the way the file system writes files. For performance reasons, it writes parts of the file into a memory buffer, which gets periodically written to the disk. If another machine tries to access the file before the machine writes the file to the disk, this error occurs. The CHPC uses NFS storage for all our home directories and group spaces; this error is well described here.

There are several ways to deal with this:

1. Use the Linux sync command to forcefully flush the memory buffers to the disk. Do this on the machine where the file writing and file reading occurs before the file is accessed. If multiple nodes are requested, add the following line to your Slurm batch script to ensure that all compute nodes in the job sync: "srun -n $SLURM_NNODES --ntasks-per-node=1 sync".
2. Sometimes, adding the Linux sleep command can help to provide an extra time window for the syncing to occur.
3. Inside the code, use fflush for C/C++ or flush for Fortran. For other languages, such as Python and MATLAB, Google the language with "for flush" to see what options are available.

If none of these options help, please try another file system to see if the error persists (e.g.,/scratch/general/vast or /scratch/local), and then let us know.

alias emacs emacsclient -a \"\"

Note the escaped double quote characters (\"\"). This will start Emacs as a daemon if it's not started already, and proceed to run in client mode.

Note that by default emacsclient starts in the terminal. To force start Emacs GUI, add the "-c" flag, e.g., (assuming the aforementioned alias is in place)."

emacs -c myfile.txt

Another solution, suggested by a user, is to add this line to your .emacs file in your home directory:

(setq locate-dominating-stop-dir-regexp "\\`\\(?:/uufs/chpc.utah.edu/common/home/[^\\/]+/\\|~/\\)\\'")

Read the file without any content conversion:

 M-x find-file-literally <Enter> filename <Enter>

After opening the file, you can tell Emacs to encode the file accordingly, e.g., to syntax highlight shell scripts:

M-x sh-mode <Enter>

To make this change permanent, open the ~/.emacs file to add:

(global-set-key "\C-c\C-f" 'find-file-literally)

In the email you received from Arbiter, there will be a section detailing the high-impact processes that caused Arbiter to trigger. In the example below, you can see that the user was running a Python process that exceeded the limits we place on our login nodes. At this point, it would be appropriate for the user to terminate any Python process running on the login node and start the same job in an interactive session or on a compute node via Slurm.

 A high-performance computing (HPC) cluster consists of many computers, known as nodes. Most of the nodes are compute nodes, which can be used for large or long-running computations. Other nodes include the login nodes, a shared resource that allows users to login to the CHPC and edit files, develop and debug software, and request computational nodes for jobs. They are, in effect, the front doors to an HPC cluster; they are not where you should run large calculations.

You are receiving this message because you are running a large or long-running processes on a login node. This can affect the responsiveness of the nodes for other CHPC users. It is required that you use compute nodes for your computations at the CHPC.

To use compute nodes, you will need to interact with the job scheduler, Slurm. Slurm allocates resources by accounting for availability and the size of your job. We have documentation about Slurm on the CHPC website, and we recommend attending our introductory Slurm presentations if you haven’t used an HPC cluster before or would like a review of Slurm.

 You will no longer receive emails from Arbiter when you cancel any running processes that are triggering Arbiter. If you are unsure what processes are triggering Arbiter, follow the steps below to identify them:

1. 1. Log into the Login Node 
      • Login to the exact node that is listed in the Arbiter email - processes are node-specific and cannot be managed from another machine
      • If the warning was triggered by a stray FastX session, log into that specific node via FastX and explicitly terminate the session.
   2. Identify the processes that are using significant resources
      • Use the command below to identify the process ID (PID) of each process that is triggering Arbiter:
      • ps -Fu $USER

   3. Stop the Process

      • Use the kill command followed by the process's ID (PID):

      • Standard stop: kill <PID>  (replace <PID> with your actual PID)

      • Force stop: If the process is unresponsive, use kill -9 <PID>  to terminate it immediately (replace <PID>  with your actual PID)

Some processes may continue running for some time, even if you’ve logged out of the node and don’t believe you have any processes running. Please see the instructions above for finding and stopping running background processes.

If you are still unsure what is triggering the Arbiter warnings, please contact us at helpdesk@chpc.utah.edu.

First, try to remove the session files by running 

rm -rf ~/.local/share/rstudio/sessions/*.

If that does not help, move the whole RStudio settings directory.

mv ~/.local/share/rstudio ~/.local/share/rstudio-old.

Be aware that this will reset some customizations you may have done. If this does not work, move the user settings,  

mv ~.config/rstudio ~/.config/rstudio-old .

OnDemand saves session files to your home directory. One of these session files is called output.log - this file contains a log of your specific app session. You can inspect the output.log file to understand why your job failed. Follow these steps to inspect your output.log file:

1. Initiate a terminal session. In OnDemand, you can do this by clicking 'Clusters' -> 'Shell Access'

2. Move to the directory that contains the correct output.log file. Replace <SESSION-ID> with the correct OnDemand session, located on the job tile in OnDemand:

cd ~/ondemand/data/sys/dashboard/batch_connect/sys/jupyter_app/output/<SESSION-ID>

3. View the output.log file:

cat output.log

4. Inspect the contents of the output.log file. Typically, if jupyter is missing, one of the last lines in the file will say something similiar to: "Error: jupyter: command not found".

If you are using a CHPC-installed python module, try changing the python version you are selecting. Some of the newest python versions do not have jupyter installed.

If you are using a custom environment, such as conda, mamba, or pyvenv, you will need to install jupyter into the custom environment.

Follow these steps to restore your connection without losing your current progress:

1. Refresh the Main Page: Go back to your main Open OnDemand tab and reload the page.

2. Re-Authenticate: Log back in with your credentials when prompted.

3. Open Your Dashboard: Navigate to the My Interactive Sessions tab in Open OnDemand.

4. Relaunch the Session: Find your active session in the list and click the blue Launch button. This will open your session in a new browser tab, preserving your current workspace and state.

5. Close the Old Tab: Immediately close the old browser tab.

Important: Do not continue working or trying to save files in the old tab. Because it is unauthenticated, any edits made there will be lost and cannot be saved to the server.

• 1. Lack of Parallelization: HPC clusters achieve speed by distributing workloads across multiple processors. If your code runs serially, it won't leverage the hardware. Depending on your workflow, you can parallelize using independent tasks, OpenMP, MPI, or interpreted languages like Python, R, or MATLAB.

Resource: Review our Introduction to Parallel Computing lecture or contact us for guidance.

• 2. Incorrect Task Counts or Hardcoded Values: Failing to supply the correct number of tasks or hardcoding task counts directly into code instead of using dynamic variables leads to poor execution. Always utilize Slurm environment variables like $SLURM_NTASKS or $SLURM_CPUS_PER_NODE in your Slurm scripts and program input files.

• 3. Inefficient Parallelization: Poorly implemented parallel logic (especially with MPI) can create severe communication overhead. If you need help profiling and optimizing your code's parallel performance, please contact us.

• 4. Hardware or Software Cluster Issues: If you have thoroughly ruled out code and configuration errors, the slowdown may stem from cluster-level hardware degradation or software environment bugs.

These are the possible methods that you can take to resolve this error:

• Choose a Compatible Binary: If the software provides multiple downloads, select a pre-compiled binary built for an OS version closer to RHEL 8 / Rocky Linux 8.
• Run via a Container: If alternative binaries or source builds are impractical, run the program inside a container (such as Apptainer) configured with the specific OS the program was built for.

• Build from Source: Attempt to compile the program natively on our systems, or use a package manager like Spack to manage and build dependencies correctly.