Appendix E: Security
Moab provides role and host based authorization, encryption*, and DES, HMAC, and MD5 based authentication. The following sections describe these features in more detail.
- E.1 Authorization
- E.1.1 Role Based Authorization Security Configuration
- E.1.2 Host Based Authorization (Administrative Hosts)
- E.2 Authentication
- E.2.1 Mauth Authentication
- E.2.2 Munge Authentication
- E.2.3 Server Response Control
- E.2.4 Interface Development Notes
- E.3 Host Security
- E.3.1 Minimal Host Security Enforcement
- E.3.2 Medium Host Security Enforcement
- E.3.3 Strict Host Security Enforcement
- E.4 Access Portal Security
E.1 Authorization
E.1.1 Role Based Authorization Security Configuration
Moab provides access control mechanisms to limit how the scheduling environment is managed. The primary means of accomplishing this is through limiting the users and hosts that are trusted and have access to privileged commands and data.
With regard to users, Moab breaks access into three distinct levels.
E.1.1.1 Level 1 Moab Admin (Administrator Access)
Level 1 Moab administrators have global access to information and unlimited control over scheduling operations. By default, they are allowed to control scheduler configuration, policies, jobs, reservations, and all scheduling functions. They are also granted access to all available statistics and state information. Level 1 administrators are specified using the ADMINCFG[1] parameter.
E.1.1.2 Level 2 Moab Admin (Operator Access)
Level 2 Moab administrators are specified using the ADMINCFG[2] parameter. By default, the users listed under this parameter are allowed to change all job attributes and are granted access to all informational Moab commands.
E.1.1.3 Level 3 Moab Admin (Help Desk Access)
Level 3 administrators are specified via the ADMINCFG[3] parameter. By default, they are allowed access to all informational Moab commands. They cannot change scheduler or job attributes.
E.1.1.4 Configuring Role Based Access
Moab allows site specific tuning of exactly which functions are available to each administrator level. Moab also provides two additional administrator levels (ADMINCFG[4] and ADMINCFG[5]) that may be used for site specific needs.
To configure Moab role based access, use the ADMINCFG parameter.
ADMINCFG[1] USERS=root,john SERVICES=ALL NAME=admin ADMINCFG[3] USERS=joe,mary SERVICES=mdiag,mrsvctl,mcredctl NAME=power ADMINCFG[5] USERS=joy,blake SERVICES=NONE NAME=users ...
 |
A NONE in services will still allow users to run showq and checkjob on their own jobs. |
To determine the role of system users and what commands they can run, use the mcredctl -q role user:<USERID> command.
Using the SERVICES attribute of the ADMINCFG parameter, access to an arbitrary selection of services can be enabled on a per administrator-level basis. Possible services include the following:
| Service | Description |
|---|---|
| changeparam | Change any scheduling policy or parameter (This command is deprecated. Use mschedctl -m instead). |
| checkjob | View detailed information for any job. |
| checknode | View detailed information for any node. |
| mbal | Perform real-time load-balancing of interactive commands. |
| mcredctl | View and modify credential attributes. |
| mdiag | Provide diagnostic reports for resources, workload, and scheduling. |
| mjobctl | Modify, control, and view jobs. Allows subcommand specification using subcommands cancel (you can also use canceljob), checkpoint, diagnose, modify, query, requeue, resume, signal, submit, suspend, adjusthold, and adjustprio. |
| mnodectl | Modify, control, and view nodes. |
| mrmctl | Modify, control, and view resource managers. |
| mrsvctl | Modify, control, and view reservations. |
| mschedctl | Modify, control, and view scheduler behavior. |
| mshow | View existing configuration and predicted resource availability. |
| showstats | View all scheduler and credential statistics. |
| releaseres | Release reservations (This command is deprecated. Use mrsvctl -r instead). |
| resetstats | Clear/reset all scheduler statistics. |
| runjob | Immediately execute any job (see mjobctl -x). |
| setqos | Set QoS on any job (This command is deprecated. Use mjobctl -m instead). |
| setres | Create a reservation (This command is deprecated. Use mrsvctl -c instead). |
| setspri | Set system priority on any job (This command is deprecated. Use mjobctl -p instead). |
| showconfig | Show all scheduler configuration parameters (This command is deprecated. Use mschedctl -l instead). |
| showres | Show detailed information for any reservation. |
E.1.1.5 Account and Class/Queue Admins
While the ADMINCFG parameter allows organizations to provide controlled access to scheduling objects, it does not allow for distribution along organizational boundaries. For example, a site may set up a level 3 administrator to be able to view statistics, diagnose jobs, and modify job priorities; it does not provide a way to differentiate one type of job from another. If a site adminsitrator wanted to allow control based on the queue or account associated with a job, they would best accomplish this using the credential MANAGERS feature.
A credential manager allows a user to be trusted to administer workload and policies for an associated subgroup of jobs. For example, in the configuration below, a number of queue and account managers are configured.
CLASSCFG[orion] MANAGERS=johns CLASSCFG[xray] MANAGERS=steve2 CLASSCFG[gamma] MANAGERS=steve2,jpw ACCOUNTCFG[bio] MANAGERS=charles
By default, the specified managers can do anything to a job that the actual job owner could do, including viewing cumulative and per job statistics, seeing job details, modifying job priorities and holds, cancelling and preempting jobs, and otherwise adjusting policies and constraints within the associated credential.
E.1.2 Host Based Authorization (Administrative Hosts)
If specified, the ADMINHOSTS parameter allows a site to specify a subset of trusted hosts. All administrative commands (level 1-3) will be rejected unless they are received from one of the hosts listed.
E.2 Authentication (Interface Security)
Moab supports password-challenge, DES, HMAC, and MD5 based authentication. Authentication protocols may be specified on a per interface basis allowing independent realms of trust with per realm secret keys and even per realm authentication protocols.
E.2.1 Mauth Authentication
Mauth is a tool provided with Moab that provides client authentication services. With mauth enabled, each client request is packaged with the client ID, a timestamp, and an encrypted key of the entire request generated using the shared secret key.
This tool is enabled by providing a secret key. A random key is selected when the Moab configure script is run and may be regenerated at any time by rerunning configure and rebuilding Moab. If desired, this random key may be overridden by specifying a new key in the protected .moab.key file as in the example below:
> vi /opt/moab/.moab.key (insert key) > cat /opt/moab/.moab.key XXXXXXXX # secure file by setting owner read-only permissions > chmod 400 /opt/moab/.moab.key # verify file is owned by root and permissions allow only root to read file > ls -l /opt/moab/.moab.key -r-------- 1 root root 15 2007-04-05 03:47 /opt/moab/.moab.key
|
If .moab.key is used, this protected file will need to be on each host that is authorized to run Moab client commands. |
|
By default, this file will be owned by the user root and its contents will be read by the mauth tool which provides client authorization services. If desired, the ownership of this file can be changed so long as this file is readable by the Moab server and the mauth tool. This can be accomplished if the Moab primary administrator, the owner of mauth, and the owner of .moab.key are the same. |
|
By default, it is up to the individual cluster administrators to determine whether to use the .moab.key file. For sites with source code, the use of .moab.key can be mandated by using configure --with-keyfile. |
|
By default, mauth is located in the install bin directory. If an alternate name or alternate file location is desired, this can be specified by setting the AUTHCMD attribute of the CLIENTCFG parameter within the moab.cfg file as in the following example. |
CLIENTCFG AUTHCMD=/opt/sbin/mauth
E.2.1.1 Configuring Peer-Specific Secret Keys
Peer-specific secret keys can be specified using the CLIENTCFG parameter. This key information must be kept secret and consequently can only be specified in the moab-private.cfg file. With regard to security, there are two key attributes that can be set. Other resource managers or clients such as Gold or a SLURM/Wiki interface can also use the attributes to configure their authentication algorithms. The default, unless otherwise stated, is always DES. These attributes are listed in the table below:
| Attribute | Format | Default | Description | Example |
|---|---|---|---|---|
| AUTH | one of ADMIN1, ADMIN2, or ADMIN3 | --- | Specifies the level of control/information available to requests coming from this source/peer. | CLIENTCFG[RM:clusterB] AUTH=admin1 KEY=14335443 |
| AUTHTYPE | one of DES, HMAC, HMAC64, or MD5. | DES | Specifies the encryption algorithm to use when generating the message checksum. | CLIENTCFG[AM:bio3] AUTHTYPE=HMAC64 |
| HOST | STRING | --- | Specifies the hostname of the remote peer. Peer requests coming from this host will be authenticated using the specified mechanism. This parameter is optional. | CLIENTCFG[RM:clusterA] HOST=orx.pb13.com KEY=banana6 |
| KEY | STRING | --- | Specifies the shared secret key to be used to generate the message checksum. | CLIENTCFG[RM:clusterA] KEY=banana6 |
The CLIENTCFG parameter takes a string index indicating which peer service will use the specified attributes. In most cases, this string is simply the defined name of the peer service. However, for the special cases of resource and allocation managers, the peer name should be prepended with the prefix RM: or AM: respectively, as in CLIENTCFG[AM:bank] or CLIENTCFG[RM:devcluster].
|
The first character of any secret key can be viewed by trusted administrators using specific diagnostic commands to analyze Moab interfaces. If needed, increase the length of the secret keys to maintain the desired security level. |
E.2.2 Munge Authentication
Moab also integrates with MUNGE, an open source authentication service created by Lawrence Livermore National Laboratory (http://home.gna.org/munge/). MUNGE works with Moab to authenticate user credentials being passed between the Moab client and the Moab server or from Moab server to Moab server.
To set up MUNGE in a cluster or grid, download and install MUNGE on every node in the cluster or grid by following the installation steps found at http://home.gna.org/munge/. The MUNGE secret key must reside on each node in the cluster or grid. Before starting the Moab daemon, the MUNGE daemon must be running on all nodes.
To enable Moab to use MUNGE for authentication purposes, specify the MUNGE executable path in the moab.cfg file using CLIENTCFG and AUTHCMD as in the following example. The MUNGE executable path must reside in each client's moab.cfg file as well.
CLIENTCFG AUTHCMD=/usr/bin/munge
|
Moab requires that the MUNGE and UNMUNGE executable names be "munge" and "unmunge" respectively. It also assumes that the UNMUNGE executable resides in the same directory as the MUNGE executable. |
E.2.2.1 Configuring Munge Command Options
Moab also integrates with MUNGE command line options. For example, to set up Moab to use a specific socket that was created when the MUNGE daemon was started, use CLIENTCFG and AUTHCMDOPTIONS to specify the newly created socket. The AUTHCMDOPTIONS command, like AUTHCMD, must also reside in the client's moab.cfg file.
CLIENTCFG AUTHCMD=/usr/bin/munge CLIENTCFG AUTHCMDOPTIONS="-S /var/run/munge/munge.socket.2"
E.2.3 Server Response Control
If a request is received that is corrupt or cannot be authenticated, Moab will report some limited information to the client indicating the source of the failure, such as "bad key," "malformed header," and so forth. In the case of highly secure environments, or to minimize the impact of sniffing or denial of service attacks, Moab can be configured to simply drop invalid requests. This is accomplished by adding the DROPBADREQUEST attribute to the CLIENTCFG parameter in the moab-private.cfg file as in the following example:
CLIENTCFG[DEFAULT] DROPBADREQUEST=TRUE
E.2.4 Interface Development Notes
Sample checksum generation algorithm code can be found in the Socket Protocol Description document.
E.3 Host Security for Compute Resources
Host level security can vary widely from one site to another with everything from pure on-your-honor based clusters to complete encrypted VLAN based network security and government approved per job scrubbing procedures being used. The following documentation describes some best practices in use throughout the industry.
E.3.1 Minimal Host Security Enforcement
For minimal host security, no additional configuration is required.
E.3.2 Medium Host Security Enforcement
- Login Access
- PAM — Enable/disable access by modifying /etc/security/access.conf.
- Processes
- Kill all processes associated with job user (dedicated).
- Kill all processes associated with job session (dedicated/shared). Use ps -ju
or ps -js <SESSID>.
- IPC (Inter-Process Communication)
- Remove shared memory, semaphores, and message queues (use ipcs/ipcrm).
- Remove named pipes.
- Network/Global Filesystem Access
- Explicitly unmount user home and global file systems.
- Local Temporary Filesystems
- Where possible, mount local file systems read-only.
- Clear /tmp, /scratch and other publicly available local file systems.
- Remove user files with shred; shred is a Linux command that first overwrites files completely before removing them, preventing remnant data from surviving on the hard drive.
E.3.3 Strict Host Security Enforcement
- VLAN creation
- Host rebuild
E.4 Moab Access Portal Security Overview
The Moab Access Portal (MAP) security model is composed of several different components. First, users will use a Web browser to log in and interact with the Web server running MAP. This communication can be encrypted using industry standard SSL to protect usernames/passwords and other sensitive information that may be accessed by the user. (Instructions on how to set up SSL connections with popular Web servers and servlet engines are readily available on the Internet. A guide for setting up SSL with Apache is available in the MAP documentation.)
When a user logs in via their Web browser, the JSP interface passes this request to a back-end Java infrastructure that then uses an encrypted SSH connection to authenticate the user's credentials at the cluster/grid head node. After the credentials are authenticated and the SSH connection established, further communication between MAP and the cluster/grid head node occurs over the encrypted SSH connection. These three components provide an end-to-end security solution for Web-based job submission and workload management.
