Corpus Research: required_files Usage in IUC (AI Generated)#

If you are adding <required_files> to a tool and want real exemplars to copy, this page surveys how the galaxyproject/tools-iuc collection uses the element in practice — the patterns that work, a couple that don’t, and a recommended shape for a new tool. It backs the Pulsar compatibility checklist and complements Galaxy’s reference docs for the required_files tool element.

What the element is for#

<required_files> declares which files from a tool’s own directory must travel with the job when it runs somewhere other than the Galaxy server — most importantly remote execution via Pulsar, where the tool directory is not on a shared filesystem and only the declared files get staged to the remote host. Any wrapped script you reach for through $__tool_directory__ needs to be listed here, or it silently goes missing on a Pulsar node while working fine locally.

Adoption across IUC is small but growing, and the exact set turns over fast enough that pinning a list here would be misleading within weeks. To see who declares it today, search the collection directly:

As a snapshot: on 2026-07-27, 23 tool wrappers declared the element. This is nowhere near universal — the vast majority of IUC tools that call $__tool_directory__/script.py still do not declare required_files and rely on the legacy whole-directory copy. So there is not yet a deep well of precedent, but the exemplars discussed below give you a clean set to model from.

One thing worth knowing up front: as of that snapshot every wrapper that declares the element uses only the <include path="..."/> form. There is no use of <exclude>, glob patterns, or bulk directory= inclusion anywhere in the collection. In practice required_files is used purely as an explicit allow-list of individual files — so that is the idiom to follow.

Links to tool XMLs below are pinned to a fixed commit so the line numbers keep pointing at what the text describes; follow the search link above for the current state of any given wrapper.

Ways tools use it#

The exemplars fall into a handful of shapes. Find the one closest to your tool and copy it.

A single helper or entry script — the common case#

By far the most frequent pattern: one wrapper script (Python or R) lives beside the XML and is both declared and invoked through $__tool_directory__. add_input_name_as_column is the canonical shape:

<required_files>
    <include path="add_input_name_as_column.py"/>
</required_files>

and the command simply runs python '$__tool_directory__/add_input_name_as_column.py'. Other one-script examples you can copy directly:

A script in a subdirectory#

Same as above, but the script lives under a scripts/ (or similar) subdirectory. Just write the relative path — including the subdirectory — into path, and reference the same relative path under $__tool_directory__ in the command. hyphy_infer_stasis_clusters:

<required_files>
    <include path="scripts/infer_stasis_clusters.py"/>
</required_files>

run with python3 '$__tool_directory__/scripts/infer_stasis_clusters.py'. The sibling hyphy_strike_ambigs does the same with a HyPhy batch-language file, scripts/strike-ambigs.bf — a reminder that the declared file need not be Python or R.

Several co-dependent scripts#

When the entry script imports or sources its siblings, or the pipeline shells out to more than one file, declare them all. virAnnot_otu is the clearest example:

<required_files>
    <include path="otu.py" />
    <include path="seek_otu.R" />
    <include path="rps2tree_html.py" />
</required_files>

Only otu.py is named on the command line; it is handed the tool directory (-tp '$__tool_directory__/') so it can locate its two helpers at runtime. Those helpers never appear on the command line, so without the explicit includes they would be missing on a Pulsar node. Same idea, different languages:

  • hgvsparser declares two R files, parseHGVS.R and buildHGVS.R, each source()d from $__tool_directory__.

  • ena_upload declares extract_tables.py and dump_yaml.py, called at different stages of the command.

A library sourced by a generated config script#

Sometimes the declared file is a library of functions rather than the entry point: the actual program is written into a <configfile> at runtime, and that generated script sources the library from $__tool_directory__. Both ggplot2_barplot and ggplot2_boxplot do this:

<required_files>
    <include path="utils.r" />
</required_files>

with the command Rscript -e 'source("${__tool_directory__}/utils.r")' -e 'source("${run_script}")' — the shared library is sourced first, then the generated run_script configfile. The key point: required_files and configfiles are complementary. The stable helper code is declared as a required file; the per-job program stays a <configfile>.

Copying files into the working directory#

Occasionally the declared files are not run in place from $__tool_directory__ at all — the command copies them into the job working directory first. This comes up with Python, where the caller and its imported modules must sit in the same directory (a soft link does not satisfy the import). table_compute:

<required_files>
    <include path="scripts/safety.py" />
    <include path="scripts/table_compute.py" />
</required_files>

cp '$userconf' ./userconfig.py &&
cp '$__tool_directory__/scripts/safety.py' ./safety.py &&
cp '$__tool_directory__/scripts/table_compute.py' ./table_compute.py &&
python ./table_compute.py

A generated userconfig.py has to be importable from the same directory as the script, so everything is copied together. Even though the scripts are copied rather than executed in place, they still must be listed in required_files so they exist on the remote host to be copied.

Patterns worth internalizing#

  • Include-only allow-list. Every tool uses <include path="..."/> and nothing else. Treat required_files as “the explicit list of tool-dir files this job needs,” never as “start from everything and subtract.”

  • One include per $__tool_directory__ reference. The healthy idiom is 1:1 — every file the command reaches for under $__tool_directory__/... has a matching <include> with the same relative path. Both $__tool_directory__/x and ${__tool_directory__}/x spellings work and are used interchangeably.

  • Relative paths mirror the on-disk layout. Paths are always relative to the tool directory (find_str.py, scripts/table_compute.py); subdirectories go straight into the path attribute.

  • It’s usually code. Most declared files are executable wrappers and helpers — .py, .R/.r, one HyPhy .bf. Small static data shipped alongside the wrapper is fair game too: getitd declares anno/amplicon.txt and anno/amplicon_kayser.tsv. What does not belong here is bulk reference data or .loc files — those go through a data table, not the tool directory.

Anti-patterns to avoid#

  • A declared file that is never used. upsetr included utils.r, but nothing in its command or run-script configfile ever referenced it (the command is just Rscript '${run_script}'). It looks copy-pasted from the ggplot2 wrappers, which genuinely source utils.r. Staging it is dead weight and a maintenance trap — drop includes you don’t use. (The wrapper has since dropped the block entirely, hence the pinned link.)

  • The entry script left off the list. extract_genomic_dna declares only extract_genomic_dna_utils.py, but the command actually runs python '$__tool_directory__/extract_genomic_dna.py' — the main entry script is not declared. Under Pulsar, only the utils file would stage and the tool would fail to find its own entry point. This is the sharpest cautionary example: the whole value of the element is defeated if the invoked script is omitted.

  • Inconsistent path spellings. Some tools write $__tool_directory__/foo and others ${__tool_directory__}/foo. Both work; pick one for readability within a wrapper.