Conventions

Appendix E: Security

Moab provides role and host based authorization, encryption*, and DES, HMAC, and MD5 based authentication. The following sections describe these features in more detail.

• Authorization
• Role Based Authorization Security Configuration
• Authentication
• Mauth Authentication
• Munge Authentication
• Server Response Control
• Interface Development Notes
• Host Security
• Minimal Host Security Enforcement
• Medium Host Security Enforcement
• Strict Host Security Enforcement
• Access Portal Security

Authorization

Role Based Authorization Security Configuration

Moab provides access control mechanisms to limit how the scheduling environment is managed. The primary means of accomplishing this is through limiting the users and hosts that are trusted and have access to privileged commands and data.

With regard to users, Moab breaks access into three distinct levels.

Level 1 Moab Admin (Administrator Access)

Level 1 Moab administrators have global access to information and unlimited control over scheduling operations. By default, they are allowed to control scheduler configuration, policies, jobs, reservations, and all scheduling functions. They are also granted access to all available statistics and state information. Level 1 administrators are specified using the ADMINCFG[1] parameter.

Level 2 Moab Admin (Operator Access)

Level 2 Moab administrators are specified using the ADMINCFG[2] parameter. By default, the users listed under this parameter are allowed to change all job attributes and are granted access to all informational Moab commands.

Level 3 Moab Admin (Help Desk Access)

Level 3 administrators are specified via the ADMINCFG[3] parameter. By default, they are allowed access to all informational Moab commands. They cannot change scheduler or job attributes.

Configuring Role Based Access

Moab allows site specific tuning of exactly which functions are available to each administrator level. Moab also provides two additional administrator levels (ADMINCFG[4] and ADMINCFG[5]) that may be used for site specific needs.

To configure Moab role based access, use the ADMINCFG parameter.

ADMINCFG[1]    USERS=root,john SERVICES=ALL NAME=admin
ADMINCFG[3]	USERS=joe,mary  SERVICES=mdiag,mrsvctl,mcredctl NAME=power
ADMINCFG[5]	USERS=joy,blake SERVICES=NONE NAME=users
...

A NONE in services will still allow users to run showq and checkjob on their own jobs.

To determine the role of system users and what commands they can run, use the mcredctl -q role user:<USERID> command.

Using the SERVICES attribute of the ADMINCFG parameter, access to an arbitrary selection of services can be enabled on a per administrator-level basis. Possible services include the following:

Account and Class/Queue Admins

While the ADMINCFG parameter allows organizations to provide controlled access to scheduling objects, it does not allow for distribution along organizational boundaries. For example, a site may set up a level 3 administrator to be able to view statistics, diagnose jobs, and modify job priorities; it does not provide a way to differentiate one type of job from another. If a site administrator wanted to allow control based on the queue or account associated with a job, they would best accomplish this using the credential MANAGERS attribute.

A credential manager allows a user to be trusted to administer workload and policies for an associated subgroup of jobs. For example, in the configuration below, a number of queue and account managers are configured.

CLASSCFG[orion] MANAGERS=johns
CLASSCFG[xray]  MANAGERS=steve2
CLASSCFG[gamma] MANAGERS=steve2,jpw
ACCOUNTCFG[bio] MANAGERS=charles

By default, the specified managers can do anything to a job that the actual job owner could do. By default, this would include the ability to view cumulative and per job statistics, see job details, modify job priorities and holds, cancel and preempt jobs, and otherwise adjust policies and constraints within the associated credential.

Authentication (Interface Security)

Moab supports password-challenge, DES, HMAC, and MD5 based authentication. Authentication protocols may be specified on a per interface basis allowing independent realms of trust with per realm secret keys and even per realm authentication protocols.

Mauth Authentication

Mauth is a tool provided with Moab that provides client authentication services. With mauth enabled, each client request is packaged with the client ID, a timestamp, and an encrypted key of the entire request generated using the shared secret key.

This tool is enabled by providing a secret key. A random key is selected when the Moab ./configure script is run and may be regenerated at any time by rerunning ./configure and rebuilding Moab.  If desired, this random key may be overridden by specifying a new key in the protected .moab.key file as in the example below:

Moab must be shut down before setting a new secret key. Use the service moab stop or mschedctl -k commands to shut down Moab.

> vi /opt/moab/etc/.moab.key
(insert key)
> cat /opt/moab/etc/.moab.key
XXXXXXXX
# secure file by setting owner read-only permissions
> chmod 400 /opt/moab/etc/.moab.key
# verify file is owned by root and permissions allow only root to read file
> ls -l /opt/moab/etc.moab.key
-r-------- 1 root root 15 2007-04-05 03:47 /opt/moab/etc/.moab.key

All directories in the path containing .moab.key must be owned by the root or primary Moab user It must not be writable by "other" in its permissions.

If .moab.key is used, this protected file will need to be on each host that is authorized to run Moab client commands.

By default, this file will be owned by the user root and its contents will be read by the mauth tool which provides client authorization services. If desired, the ownership of this file can be changed so long as this file is readable by the Moab server and the mauth tool. This can be accomplished if the Moab primary administrator, the owner of mauth, and the owner of .moab.key are the same.

By default, it is up to the individual cluster administrators to determine whether to use the .moab.key file. For sites with source code, the use of .moab.key can be mandated by using ./configure --with-keyfile.

By default, mauth is located in the install bin directory. If an alternate name or alternate file location is desired, this can be specified by setting the AUTHCMD attribute of the CLIENTCFG parameter within the moab.cfg file as in the following example.

CLIENTCFG  AUTHCMD=/opt/sbin/mauth

Configuring Peer-Specific Secret Keys

Peer-specific secret keys can be specified using the CLIENTCFG parameter. This key information must be kept secret and consequently can only be specified in the moab-private.cfg file. With regard to security, there are two key attributes that can be set. (Other resource managers or clients such as Moab Accounting Manager or a SLURM/Wiki interface can also use the attributes to configure their authentication algorithms. The default, unless otherwise stated, is always DES. These attributes are listed in the table below:

CLIENTCFG[RM:clusterB] AUTH=admin1 KEY=14335443

CLIENTCFG[AM:mam] AUTHTYPE=HMAC64

CLIENTCFG[RM:clusterA] HOST=orx.pb13.com  KEY=banana6

CLIENTCFG[RM:clusterA] KEY=banana6

The CLIENTCFG parameter takes a string index indicating which peer service will use the specified attributes. In most cases, this string is simply the defined name of the peer service. However, for the special cases of resource and allocation managers, the peer name should be prepended with the prefix RM: or AM: respectively, as in CLIENTCFG[AM:mam] or CLIENTCFG[RM:devcluster].

The first character of any secret key can be viewed by trusted administrators using specific diagnostic commands to analyze Moab interfaces. If needed, increase the length of the secret keys to maintain the desired security level.

Munge Authentication

Moab also integrates with MUNGE, an open source authentication service created by Lawrence Livermore National Laboratory (http://home.gna.org/munge/). MUNGE works with Moab to authenticate user credentials being passed between the Moab client and the Moab server or from Moab server to Moab server.

To set up MUNGE in a cluster, download and install MUNGE on every node in the cluster by following the installation steps found at http://home.gna.org/munge/. The MUNGE secret key must reside on each node in the cluster. Before starting the Moab daemon, the MUNGE daemon must be running on all nodes.

To enable Moab to use MUNGE for authentication purposes, specify the MUNGE executable path in the moab.cfg file using CLIENTCFG and AUTHCMD as in the following example. The MUNGE executable path must reside in each client's moab.cfg file as well.

CLIENTCFG AUTHCMD=/usr/bin/munge

Moab requires that the MUNGE and UNMUNGE executable names be "munge" and "unmunge" respectively. It also assumes that the UNMUNGE executable resides in the same directory as the MUNGE executable.

Configuring Munge Command Options

Moab also integrates with MUNGE command line options. For example, to set up Moab to use a specific socket that was created when the MUNGE daemon was started, use CLIENTCFG and AUTHCMDOPTIONS to specify the newly created socket. The AUTHCMDOPTIONS attribute, like AUTHCMD, must also reside in the client's moab.cfg file.

CLIENTCFG     AUTHCMD=/usr/bin/munge 
CLIENTCFG     AUTHCMDOPTIONS="-S /var/run/munge/munge.socket.2"

Server Response Control

If a request is received that is corrupt or cannot be authenticated, Moab will report some limited information to the client indicating the source of the failure, such as "bad key," "malformed header," and so forth. In the case of highly secure environments, or to minimize the impact of sniffing or denial of service attacks, Moab can be configured to simply drop invalid requests. This is accomplished by adding the DROPBADREQUEST attribute to the CLIENTCFG parameter in the moab-private.cfg file as in the following example:

CLIENTCFG[DEFAULT] DROPBADREQUEST=TRUE

Interface Development Notes

Sample checksum generation algorithm code can be found in the Socket Protocol Description document.

Host Security for Compute Resources

Host level security can vary widely from one site to another with everything from pure on-your-honor based clusters to complete encrypted VLAN based network security and government approved per job scrubbing procedures being used. The following documentation describes some best practices in use throughout the industry.

Minimal Host Security Enforcement

For minimal host security, no additional configuration is required.

Medium Host Security Enforcement

• Login Access
  • PAM — Enable/disable access by modifying /etc/security/access.conf.
• Processes
  • Kill all processes associated with job user (dedicated).
  • Kill all processes associated with job session (dedicated/shared). Use ps -ju <USER> or ps -js <SESSID>.
• IPC (Inter-Process Communication)
  • Remove shared memory, semaphores, and message queues (use ipcs/ipcrm).
  • Remove named pipes.
• Network/Global File System Access
  • Explicitly unmount user home and global file systems.
• Local Temporary File Systems
  • Where possible, mount local file systems read-only.
  • Clear /tmp, /scratch and other publicly available local file systems.
  • Remove user files with shred; shred is a Linux command that first overwrites files completely before removing them, preventing remnant data from surviving on the hard drive.

Strict Host Security Enforcement

• VLAN creation
• Host rebuild
  • U.S Dept of Energy Disk/File Sanitization (SCRUB)
  • U.S Dept of Defense Scrubbing Software (DOD-5520)

Moab Access Portal Security Overview

The Moab Access Portal (MAP) security model is composed of several different components. First, users will use a Web browser to log in and interact with the Web server running MAP. This communication can be encrypted using industry standard SSL to protect usernames/passwords and other sensitive information that may be accessed by the user. (Instructions on how to set up SSL connections with popular Web servers and servlet engines are readily available on the Internet. A guide for setting up SSL with Apache is available in the MAP documentation.)

When a user logs in via their Web browser, the JSP interface passes this request to a back-end Java infrastructure that then uses an encrypted SSH connection to authenticate the user's credentials at the cluster head node. After the credentials are authenticated and the SSH connection established, further communication between MAP and the cluster head node occurs over the encrypted SSH connection. These three components provide an end-to-end security solution for Web-based job submission and workload management.