TORQUE Resource Manager > Configuring Data Management > File stage-in/stage-out

File stage-in/stage-out

File staging requirements are specified using the stagein and stageout directives of the qsub command. Stagein requests occur before the job starts execution, while stageout requests happen after a job completes.

On completion of the job, all staged-in and staged-out files are removed from the execution system. The file_list is in the form local_file@hostname:remote_file[,...] regardless of the direction of the copy. The name local_file is the name of the file on the system where the job executed. It may be an absolute path or relative to the home directory of the user. The name remote_file is the destination name on the host specified by hostname. The name may be absolute or relative to the user's home directory on the destination host. The use of wildcards in the file name is not recommended.

The file names map to a remote copy program (rcp/scp/cp, depending on configuration) called on the execution system in the following manner:

For stagein: rcp/scp hostname:remote_file local_file

For stageout: rcp/scp local_file hostname:remote_file

Examples

# stage /home/john/input_source.txt from node13.fsc to /home/john/input_destination.txt on master compute node

> qsub -l nodes=1,walltime=100 -W [email protected]:/home/john/input_destination.txt

# stage /home/bill/output_source.txt on master compute node to /tmp/output_destination.txt on node15.fsc

> qsub -l nodes=1,walltime=100 -W stageout=/tmp/[email protected]:/home/bill/output_destination.txt

$ fortune >xxx;echo cat xxx|qsub -W stagein=xxx@`hostname`:xxx

199.myhost.mydomain

$ cat STDIN*199

Anyone who has had a bull by the tail knows five or six more things

than someone who hasn't.

-- Mark Twain