5.632 Considerations Before Upgrading

Torque is flexible in regards to how it can be upgraded. In most cases, a Torque "shutdown" followed by a configure, make, make install procedure as documented in this guide is all that is required (see Installing ). This process will preserve existing configuration and in most cases, existing workload.

A few considerations are included below:

• If upgrading from OpenPBS, PBSPro, or Torque 1.0.3 or earlier, queued jobs whether active or idle will be lost. In such situations, job queues should be completely drained of all jobs.
• If not using the pbs_mom -r or -p flag (see Command-line Arguments), running jobs may be lost. In such cases, running jobs should be allowed to be completed or should be requeued before upgrading Torque.
• pbs_mom and pbs_server daemons of differing versions may be run together. However, not all combinations have been tested and unexpected failures may occur.
• When upgrading from early versions of Torque (pre-4.0) and Moab, you may encounter a problem where Moab core files are regularly created in /opt/moab. This can be caused by old Torque library files used by Moab that try to authorize with the old Torque pbs_iff authorization daemon. You can resolve the problem by removing the old version library files from /usr/local/lib.

To upgrade

1. Build new release (do not install).
2. Stop all Torque daemons (see qterm and momctl -s).
3. Install new Torque (use make install).
4. Start all Torque daemons.

5.632.1 Rolling Upgrade

If you are upgrading to a new point release of your current version (for example, from 4.2.2 to 4.2.3) and not to a new major release from your current version (for example, from 4.1 to 4.2), you can use the following procedure to upgrade Torque without taking your nodes offline.

To perform a rolling upgrade in Torque

1. Enable the pbs_mom flag on the MOMs you want to upgrade. The enablemomrestart option causes a MOM to check if its binary has been updated and restart itself at a safe point when no jobs are running. You can enable this in the MOM configuration file, but it is recommended that you use momctl instead.

> momctl -q enablemomrestart=1 -h :ALL

The enablemomrestart flag is enabled on all nodes.

2. Replace the pbs_mom binary, located in /usr/local/bin by default. pbs_mom will continue to run uninterrupted because the pbs_mom binary has already been loaded in RAM.

   > torque-package-mom-linux-x86_64.sh --install

   The next time pbs_mom is in an idle state, it will check for changes in the binary. If pbs_mom detects that the binary on disk has changed, it will restart automatically, causing the new pbs_mom version to load.

   After the pbs_mom restarts on each node, the enablemomrestart parameter will be set back to false (0) for that node.

© 2016 Adaptive Computing