ARMer: A Python Library for Adaptive Resource Allocation in High-Throughput Workflow

High-throughput modeling requires allocation of different types of computing resources (e.g., GPU/CPU) for various computational sub-tasks in high-performance computing (HPC) clusters. To enhance efficiency of resource consumption, here we developed an adaptive resource allocation strategy to dynamically request computing resources based on the specific need of a certain modeling sub-task in the workflow. We implemented the strategy as a new Python library, i.e., adaptive resource manager (ARMer). As a proof of concept, we employed ARMer for allocating computing resources during the high-throughput enzyme modeling of fluoroacetate dehalogenase using EnzyHTP. The workflow involves four sequential sub-tasks, including: mutant generation, molecular dynamics simulation, quantum mechanical calculation, and data analysis. Compared to fixed resource allocation where both CPU and GPU are on-call for use during the entire workflow, the use of ARMer in the workflow can save up to 87% CPU hours and 14% GPU hours. In addition, ARMer allows parallel submission of multiple computational jobs in a job array and provides customized environment settings for each software used in the workflow.