pydra.engine.workers module
Execution workers.
- class pydra.engine.workers.ConcurrentFuturesWorker(n_procs=None)[source]
Bases:
Worker
A worker to execute in parallel using Python’s concurrent futures.
- async exec_as_coro(runnable, rerun=False, environment=None)[source]
Run a task (coroutine wrapper).
- plugin_name = 'cf'
- class pydra.engine.workers.DaskWorker(**kwargs)[source]
Bases:
Worker
A worker to execute in parallel using Dask.distributed. This is an experimental implementation with limited testing.
- plugin_name = 'dask'
- class pydra.engine.workers.DistributedWorker(loop=None, max_jobs=None)[source]
Bases:
Worker
Base Worker for distributed execution.
- async fetch_finished(futures)[source]
Awaits asyncio’s
asyncio.Task
until one is finished.Limits number of submissions based on py:attr:DistributedWorker.max_jobs.
- Parameters:
futures (set of asyncio awaitables) – Task execution coroutines or asyncio
asyncio.Task
- Returns:
pending – Pending asyncio
asyncio.Task
.- Return type:
- max_jobs
Maximum number of concurrently running jobs.
- class pydra.engine.workers.PsijLocalWorker(**kwargs)[source]
Bases:
PsijWorker
A worker to execute tasks using PSI/J on the local machine.
- plugin_name = 'psij-local'
- subtype = 'local'
- class pydra.engine.workers.PsijSlurmWorker(**kwargs)[source]
Bases:
PsijWorker
A worker to execute tasks using PSI/J using SLURM.
- plugin_name = 'psij-slurm'
- subtype = 'slurm'
- class pydra.engine.workers.PsijWorker(**kwargs)[source]
Bases:
Worker
A worker to execute tasks using PSI/J.
- async exec_psij(runnable, rerun=False)[source]
Run a task (coroutine wrapper).
- Raises:
Exception – If stderr is not empty.
- Return type:
None
- make_job(spec, attributes)[source]
Create a PSI/J job.
- Parameters:
spec (psij.JobSpec) – PSI/J job specification.
attributes (any) – Job attributes.
- Returns:
PSI/J job.
- Return type:
psij.Job
- class pydra.engine.workers.SGEWorker(loop=None, max_jobs=None, poll_delay=1, qsub_args=None, write_output_files=True, max_job_array_length=50, indirect_submit_host=None, max_threads=None, poll_for_result_file=True, default_threads_per_task=1, polls_before_checking_evicted=60, collect_jobs_delay=30, default_qsub_args='', max_mem_free=None)[source]
Bases:
DistributedWorker
A worker to execute tasks on SLURM systems.
- plugin_name = 'sge'
- class pydra.engine.workers.SerialWorker(**kwargs)[source]
Bases:
Worker
A worker to execute linearly.
- async fetch_finished(futures)[source]
Awaits asyncio’s
asyncio.Task
until one is finished.- Parameters:
futures (set of asyncio awaitables) – Task execution coroutines or asyncio
asyncio.Task
- Returns:
pending – Pending asyncio
asyncio.Task
.- Return type:
- plugin_name = 'serial'
- class pydra.engine.workers.SlurmWorker(loop=None, max_jobs=None, poll_delay=1, sbatch_args=None)[source]
Bases:
DistributedWorker
A worker to execute tasks on SLURM systems.
- plugin_name = 'slurm'
- class pydra.engine.workers.Worker(loop=None)[source]
Bases:
object
A base class for execution of tasks.
- async fetch_finished(futures)[source]
Awaits asyncio’s
asyncio.Task
until one is finished.- Parameters:
futures (set of asyncio awaitables) – Task execution coroutines or asyncio
asyncio.Task
- Returns:
pending – Pending asyncio
asyncio.Task
.- Return type: