You are here: 18 Miscellaneous > Enabling High Availability Features

18.2 Enabling High Availability Features

Contact Adaptive Computing before attempting to implement any type of high availability.

The "native" high availability implementation, as described here, is only suitable for Moab Basic Edition. Contact Adaptive Computing for information on high availability for Enterprise Edition.

18.2.1 High Availability Overview

High availability allows Moab to run on two different machines: a primary and secondary server. The configuration method to achieve this behavior takes advantage of a networked file system to configure two Moab servers with only one operating at a time.

If you use a shared file system for high availability and Moab is configured to use a database, Moab must be an ODBC build, not SQLite.

When configured to run on a networked file system — any networked file system that supports file locking is supported — the first Moab server that starts locks a particular file. The second Moab server waits on that lock and only begins scheduling when it gains control of the lock on the file. This method achieves near instantaneous turnover between failures and eliminates the need for two Moab servers to synchronize information periodically as the two Moab servers access the same database/checkpoint file.

As Moab uses timestamping in the lock file to implement high availability, the clocks on both servers require synchronization; all machines in a cluster must be synchronized to the same time server.

Moab high availability and Torque high availability operate independently of each other. If a job is submitted with msub and the primary Moab server is down, msub tries to connect to the fallback Moab server. Once the job is given to Torque, if Torque can't connect to the primary pbs_server, it tries to connect to the fallback pbs_server. For example:

A job is submitted with msub, but Moab is down on server01, so msub contacts Moab running on server02.

A job is submitted with msub and Moab hands it off to Torque, but pbs_server is down on server01, so qsub contacts pbs_server running on server02.

When you shut down or restart Moab on both servers, you must run the command twice. A single shutdown (mschedctl -k) or restart (mschedctl -R) command will go to the primary server and kill it, causing the secondary server to fall back and start operating. To kill the secondary server, resubmit the command.

Do not use anything but a plain simple NFS fileshare that is not used by anybody or anything else (i.e., only Moab can use the fileshare).

Do not use any general-purpose NAS, do not use any parallel file system, and do not use company-wide shared infrastructure to set up Moab high availability using "native" high availability.

When mounting a remote machine to the NFS server using the /etc/fstab file, do not use the option "noatime". This option disables access time updates on the file system which Moab relies on to prevent scheduling conflicts between multiple Moab servers.

18.2.2 Configuring High Availability on a Networked File System

Because the two Moab servers access the same files, configuration is only required in the moab.cfg file. The two hosts that run Moab must be configured with the SERVER and FBSERVER parameters. File lock is turned on using the FLAGS=filelockha flag. Specify the lock file with the HALOCKFILE parameter. The following example illustrates a possible configuration:

SCHEDCFG[Moab]  SERVER=host1:42559
SCHEDCFG[Moab]  FBSERVER=host2
SCHEDCFG[Moab]  FLAGS=filelockha
SCHEDCFG[Moab]  HALOCKFILE=/opt/moab/.moab_lock
			

Use the HALOCKUPDATETIME parameter to specify how frequently the primary server updates the timestamp on the lock file. Use the HALOCKCHECKTIME parameter to specify how frequently the secondary server checks the timestamp on the lock file.

HALOCKCHECKTIME 9
HALOCKUPDATETIME 3			

In the preceding example, the secondary server checks the lock file for updates every 9 seconds. The HALOCKUPDATETIME parameter is set to 3 seconds, permitting the primary server three opportunities to update the timestamp for each time the secondary server checks the timestamp on the lock file.

FBSERVER does not take a port number. The primary server's port is used for both the primary server and the fallback server.

18.2.3 Confirming High Availability on a Networked File System

Administrators can run the mdiag -S -v command to view which Moab server is currently scheduling and responding to client requests.

18.2.4 Other High Availability Configuration

Moab has many features to improve the availability of a cluster beyond the ability to automatically relocate to another execution server. The following table describes some of these features.

Feature Description
AMCFG[] BACKUPHOST If using the Moab Accounting Manager, you may enable high availability with the accounting manager by specifying a backup server as in the following example:
AMCFG[mam] BACKUPHOST=headnode2					
FBSERVER If you are communicating to an HA configuration from the grid head, you can use this parameter to point Moab to the fallback server.
JOBACTIONONNODEFAILURE If a node allocated to an active job fails, it is possible for the job to continue running indefinitely even though the output it produces is of no value. Setting this parameter allows the scheduler to automatically preempt these jobs when a node failure is detected, possibly allowing the job to run elsewhere and also allowing other allocated nodes to be used by other jobs.
SCHEDCFG[] RECOVERYACTION If a catastrophic failure event occurs (SIGSEGV or SIGILL signal is triggered), Moab can be configured to automatically restart, trap the failure, ignore the failure, or behave in the default manner for the specified signal. These actions are specified using the values RESTART, TRAP, IGNORE, or DIE, as in the following example:
SCHEDCFG[bas] MODE=NORMAL RECOVERYACTION=RESTART
						
SCHEDCFG[] <Failover Trigger Definition>

HA failover trigger.

SCHEDCFG[Moab] TRIGGER=atype=exec,etype=failure,action="/opt/moab/hafailover.sh"	

© 2016 Adaptive Computing