(Click to open topic with navigation)
Moab provides access control mechanisms to limit how the scheduling environment is managed. The primary means of accomplishing this is through limiting the users and hosts that are trusted and have access to privileged commands and data.
With regard to users, Moab breaks access into three distinct levels.
Level 1 Moab Admin (Administrator Access)
Level 1 Moab administrators have global access to information and unlimited control over scheduling operations. By default, they are allowed to control scheduler configuration, policies, jobs, reservations, and all scheduling functions. They are also granted access to all available statistics and state information. Level 1 administrators are specified using the ADMINCFG parameter.
Level 2 Moab Admin (Operator Access)
Level 2 Moab administrators are specified using the ADMINCFG parameter. By default, the users listed under this parameter are allowed to change all job attributes and are granted access to all informational Moab commands.
Level 3 Moab Admin (Help Desk Access)
Level 3 administrators are specified via the ADMINCFG parameter. By default, they are allowed access to all informational Moab commands. They cannot change scheduler or job attributes.
Configuring Role Based Access
Moab allows site specific tuning of exactly which functions are available to each administrator level. Moab also provides two additional administrator levels (ADMINCFG and ADMINCFG) that may be used for site specific needs.
ADMINCFG is different from other administrator levels because, when set, all commands are authorized at that level by default. Furthermore, if a service is set at level 5, all other services are disallowed.
To configure Moab role based access, use the ADMINCFG parameter.
ADMINCFG USERS=root,john SERVICES=ALL NAME=admin ADMINCFG USERS=joe,mary SERVICES=mdiag,mrsvctl,mcredctl NAME=power ADMINCFG USERS=joy,blake SERVICES=NONE NAME=users ...
To determine the role of system users and what commands they can run, use the mcredctl -q role user:<USERID> command.
Using the SERVICES attribute of the ADMINCFG parameter, access to an arbitrary selection of services can be enabled on a per administrator-level basis. Possible services include the following:
|changeparam||Change any scheduling policy or parameter (This command is deprecated. Use mschedctl -m instead).|
|checkjob||View detailed information for any job.|
|checknode||View detailed information for any node.|
|mbal||Perform real-time load-balancing of interactive commands.|
|mcredctl||View and modify credential attributes.|
|mdiag||Provide diagnostic reports for resources, workload, and scheduling.|
|mjobctl||Modify, control, and view jobs.
|mnodectl||Modify, control, and view nodes.|
|mrmctl||Modify, control, and view resource managers.|
|mrsvctl||Modify, control, and view reservations.|
|mschedctl||Modify, control, and view scheduler behavior.|
|mshow||View existing configuration and predicted resource availability.|
|showstats||View all scheduler and credential statistics.|
|releaseres||Release all reservations (This command is deprecated. Use mrsvctl -r instead).|
|runjob||Immediately execute any job (see mjobctl -x).|
|setqos||Set QoS on any job (This command is deprecated. Use mjobctl -m instead).|
|setres||Create any reservation (This command is deprecated. Use mrsvctl -c instead).|
|setspri||Set system priority on any job (This command is deprecated. Use mjobctl -p instead).|
|showconfig||Show all scheduler configuration parameters (This command is deprecated. Use mschedctl -l instead).|
|showres||Show detailed information for any reservation.|
|showstate||Show detailed information for all jobs, including their locations, and display job error messages, if any.|
Account and Class/Queue Admins
While the ADMINCFG parameter allows organizations to provide controlled access to scheduling objects, it does not allow for distribution along organizational boundaries. For example, a site may set up a level 3 administrator to be able to view statistics, diagnose jobs, and modify job priorities; it does not provide a way to differentiate one type of job from another. If a site administrator wanted to allow control based on the queue or account associated with a job, they would best accomplish this using the credential MANAGERS attribute.
A credential manager allows a user to be trusted to administer workload and policies for an associated subgroup of jobs. For example, in the configuration below, a number of queue and account managers are configured.
CLASSCFG[orion] MANAGERS=johns CLASSCFG[xray] MANAGERS=steve2 CLASSCFG[gamma] MANAGERS=steve2,jpw ACCOUNTCFG[bio] MANAGERS=charles
By default, the specified managers can do anything to a job that the actual job owner could do. By default, this would include the ability to view cumulative and per job statistics, see job details, modify job priorities and holds, cancel and preempt jobs, and otherwise adjust policies and constraints within the associated credential.
Moab supports password-challenge, DES, HMAC, and MD5 based authentication. Authentication protocols may be specified on a per interface basis allowing independent realms of trust with per realm secret keys and even per realm authentication protocols.
Mauth is a tool provided with Moab that provides client authentication services. With mauth enabled, each client request is packaged with the client ID, a timestamp, and an encrypted key of the entire request generated using the shared secret key.
This tool is enabled by providing a secret key. A random key is selected when the Moab ./configure script is run and may be regenerated at any time by rerunning ./configure and rebuilding Moab. If desired, this random key may be overridden by specifying a new key in the protected .moab.key file as in the example below:
Moab must be shut down before setting a new secret key. Use the service moab stop or mschedctl -k commands to shut down Moab.
> vi /opt/moab/etc/.moab.key (insert key) > cat /opt/moab/etc/.moab.key XXXXXXXX # secure file by setting owner read-only permissions > chmod 400 /opt/moab/etc/.moab.key # verify file is owned by root and permissions allow only root to read file > ls -l /opt/moab/etc/.moab.key -r-------- 1 root root 15 2007-04-05 03:47 /opt/moab/etc/.moab.key
All directories in the path containing .moab.key must be owned by the root or primary Moab user It must not be writable by "other" in its permissions.
If .moab.key is used, this protected file will need to be on each host that is authorized to run Moab client commands.
By default, this file will be owned by the user root and its contents will be read by the mauth tool which provides client authorization services. If desired, the ownership of this file can be changed so long as this file is readable by the Moab server and the mauth tool. This can be accomplished if the Moab primary administrator, the owner of mauth, and the owner of .moab.key are the same.
By default, it is up to the individual cluster administrators to determine whether to use the .moab.key file. For sites with source code, the use of .moab.key can be mandated by using ./configure --with-keyfile.
By default, mauth is located in the install bin directory. If an alternate name or alternate file location is desired, this can be specified by setting the AUTHCMD attribute of the CLIENTCFG parameter within the moab.cfg file as in the following example.
Peer-specific secret keys can be specified using the CLIENTCFG parameter. This key information must be kept secret and consequently can only be specified in the moab-private.cfg file. With regard to security, there are two key attributes that can be set. (Other resource managers or clients such as Moab Accounting Manager or a SLURM/Wiki interface can also use the attributes to configure their authentication algorithms. The default, unless otherwise stated, is always DES. These attributes are listed in the table below:
|Format||one of ADMIN1, ADMIN2, or ADMIN3|
|Description||Specifies the level of control/information available to requests coming from this source/peer.|
CLIENTCFG[RM:clusterB] AUTH=admin1 KEY=14335443
|Format||one of DES, HMAC, HMAC64, or MD5.|
|Description||Specifies the encryption algorithm to use when generating the message checksum.|
|Description||Specifies the hostname of the remote peer. Peer requests coming from this host will be authenticated using the specified mechanism. This parameter is optional.|
CLIENTCFG[RM:clusterA] HOST=orx.pb13.com KEY=banana6
|Description||Specifies the shared secret key to be used to generate the message checksum.|
The CLIENTCFG parameter takes a string index indicating which peer service will use the specified attributes. In most cases, this string is simply the defined name of the peer service. However, for the special cases of resource and accounting managers, the peer name should be prepended with the prefix RM: or AM: respectively, as in CLIENTCFG[AM:mam] or CLIENTCFG[RM:devcluster].
The first character of any secret key can be viewed by trusted administrators using specific diagnostic commands to analyze Moab interfaces. If needed, increase the length of the secret keys to maintain the desired security level.
Moab also integrates with MUNGE, an open source authentication service created by Lawrence Livermore National Laboratory (http://home.gna.org/munge/). MUNGE works with Moab to authenticate user credentials being passed between the Moab client and the Moab server or from Moab server to Moab server.
To set up MUNGE in a cluster or grid, download and install MUNGE on every node in the cluster or grid by following the installation steps found at http://home.gna.org/munge/. The MUNGE secret key must reside on each node in the cluster or grid. Before starting the Moab daemon, the MUNGE daemon must be running on all nodes.
To enable Moab to use MUNGE for authentication purposes, specify the MUNGE executable path in the moab.cfg file using CLIENTCFG and AUTHCMD as in the following example. The MUNGE executable path must reside in each client's moab.cfg file as well.
Moab requires that the MUNGE and UNMUNGE executable names be "munge" and "unmunge" respectively. It also assumes that the UNMUNGE executable resides in the same directory as the MUNGE executable.
Moab also integrates with MUNGE command line options. For example, to set up Moab to use a specific socket that was created when the MUNGE daemon was started, use CLIENTCFG and AUTHCMDOPTIONS to specify the newly created socket. The AUTHCMDOPTIONS attribute, like AUTHCMD, must also reside in the client's moab.cfg file.
CLIENTCFG AUTHCMD=/usr/bin/munge CLIENTCFG AUTHCMDOPTIONS="-S /var/run/munge/munge.socket.2"
If a request is received that is corrupt or cannot be authenticated, Moab will report some limited information to the client indicating the source of the failure, such as "bad key," "malformed header," and so forth. In the case of highly secure environments, or to minimize the impact of sniffing or denial of service attacks, Moab can be configured to simply drop invalid requests. This is accomplished by adding the DROPBADREQUEST attribute to the CLIENTCFG parameter in the moab-private.cfg file as in the following example:
The SERVERCSALGO parameter lets you choose the algorithm used for message digests and message authentication codes.
If you are using Moab Web Services, then you must set the MWS configuration parameter moab.messageDigestAlgorithm to match the value of SERVERCSALGO. See moab.messageDigestAlgorithm for more information.
Sample checksum generation algorithm code can be found in the Socket Protocol Description document.
Host level security can vary widely from one site to another with everything from pure on-your-honor based clusters to complete encrypted VLAN based network security and government approved per job scrubbing procedures being used. The following documentation describes some best practices in use throughout the industry.
For minimal host security, no additional configuration is required.