template

bgqmap template is a tool aimed to ease the creation of a jobs file to be used with bgqmap submit.

Features:

• find your current loaded EasyBuild modules and adds them to the output as pre-command
• find your current conda environment and add them to the output as pre-command
• the job parameters passed through command line are added to the generated jobs_file

Generating a file

By default, the output is printed to the standard output. You can provide a file with the -f flag or redirect the output to a file (> file.txt).

Using wildcards

bgqmap template accepts two types of wildcards:

• user wildcards: indicated with {{...}} They can contain:
  • list of , separated items: the wildcard is replaced by each item
  • file name: he wildcard is replaced by each line in the file
• glob wildcards: if any of *, **, ? and [x-y] is found, it is assumed to be a glob wildcard and therefore expanded using the python glob module.

How it works

The expansion of wildcards is a two step process. First user wildcards are expanded and glob wildcards are expanded in a second phase. For the latter, any set of characters surrounded by blanks is analysed. If it contains one or more of the mentioned wildcards, a glob search is performed.

Note

Use \ before glob wildcards to avoid their expansion

Named groups

bgqmap template contains a special feature that allows the user to replace the values of any of the wildcards in different parts of the command.

To use this feature, the wildcard needs to be named using {{?<name>:<value>}} and it can be replaced anywhere using {{?=<name>}}.

The name can be anything, but a glob wildcard character. It cannot start with a number.

Tip

We recommend to limit the names to characters in a-z, A-Z and 0-9.

The value can be anything in a user wildcard or a glob wildcard.

Note

Even if it is possible to use a glob wildcard in a user wildcard (e.g. {{a,*.txt}}) we do not recommend this use for named groups as the result might differ from the expected.

Warning

As mentioned, in a user group you can place anything that is is a user or glob wildcard. Thus, {{?group:*}}.txt recognize the glob wildcard and and will do a glob search for all .txt files. On the other hand, {{?group:*.txt}} assumes it is a user wildcard (as it is not only a glob wildcard) and will try to open a file named *.txt which most likely will not exits and will fail.

Usage

bgqmap template "<command>" -m <memory> -c <cores> -f <output file>

---

Usage: bgqmap template [OPTIONS] CMD

Create a file template for execution with bgqmap.

Conda environment and Easy build modules, if not provided are taken from the corresponding environment variables

The commands accepts ‘{{…}}’ and ‘*’, ‘?’, [x-y] (from glob module) as wildcards. If you want to use the value resulting of the expansion of that wildcard, name it ‘{{?name:…}}’ and use it anywhere ‘{{?=name}}’ and as many times as you want.

First, items between ‘{{…}}’ are expanded: If there only one element, it is assumed to be a file path, and the wildcard is replaced by any line in that file which is not empty or commented. If there are more ‘,’ separated elements, it is assumed to be a list, and the wildcard replaced by each of the list members. If the inner value corresponds to one of the glob module wildcards, its expansion is postponed.

In a second phase, ‘*’ wildcards are substituted as in glob.glob. Wildcards with are not in a named group are expanded first, and the latter ones are expanded in a final iterative process.

Options:

-f, --file PATH
	File to write the template to
-c, --cores INTEGER
	Number of cores to use
-m, --memory TEXT
	Max memory
-t, --wall_time TEXT
	Wall time
--conda-env TEXT
	Conda environment. Default: current environment
--module PATH	Easy build modules. Default: current loaded modules
-h, --help	Show this message and exit.

Examples

Easybuild modules and conda environments are recognized:

$ bgqmap template "sleep 5"
# module load anaconda3/4.4.0
# source activate test_bgqmap
sleep 5

Job parameters can also be added:

$ bgqmap template "sleep 5" -c 1 -m 1G
## cores=1, memory=1G
sleep 5

Using user wildcards with lists:

$ bgqmap template "sleep {{5,10}}"
sleep 5
sleep 10

Using user wildcards with files:

$ bgqmap template "sleep {{examples/input/sleep_times.txt}}"
sleep 5
sleep 10

Using glob wildcards:

$ bgqmap template "mypgrog --input examples/input/*.txt"
mypgrog --input examples/input/sleep_times.txt
mypgrog --input examples/input/empty.txt

Using named wildcards:

$ bgqmap template "myprog --input examples/input/{{?f_name:*}}.txt --variable {{?v_name:a,b}} --output {{?=f_name}}_{{?=v_name}}"
myprog --input examples/input/sleep_times.txt --variable a --output sleep_times_a
myprog --input examples/input/empty.txt --variable a --output empty_a
myprog --input examples/input/sleep_times.txt --variable b --output sleep_times_b
myprog --input examples/input/empty.txt --variable b --output empty_b