Real-world scientific applications often encompass end-to-end data processing pipelines composed of a large number of interconnected computational tasks of various granularity. HyperLoom is an open source platform for defining and executing such pipelines in distributed environments and providing a Python interface for defining tasks. HyperLoom is a self-contained system that does not use an external scheduler for the actual execution of the task. We have successfully employed HyperLoom for executing chemogenomics pipelines used in pharmaceutic industry for novel drug discovery.

HyperLoom features:

• in-memory data processing reducing filesystem load,
• direct worker-to-worker data transfer reducing server overhead,
• support for execution of third party applications,
• data-location aware scheduling algorithm reducing inter-node network traffic.
• C++ core with a Python client enabling high performance through a simple API,
• high scalability and native HPC support,
• BSD license.