LFX Platform

Know more about LFX Platform

LFX Insights
Curated Collections

Asynchronous & Batch Processing

Libraries for asynchronous background tasks and large-scale scheduled jobs, addressing both web-scale usage (job queues) and big data batch processing.

The Linux Foundation

by The Linux Foundation

57 projects ・ Updated 14 Mar 2025

Project
Contributors
Organizations
Software value
Laravel Framework
24,206
4,643
$11M
Ray
9,084
1,538
$54M
Celery
9,071
2,322
$4.1M
Sidekiq
5,652
1,811
$832K
Apache Beam
4,806
643
$95M
XXL-JOB
4,803
265
$1.5M
Apache DolphinScheduler
4,492
244
$13M
Rundeck
4,047
706
$22M
Prefect
3,948
799
$36M
Hangfire
2,687
415
$2.8M
Nextflow
2,465
498
$17M
Apache Hadoop
2,342
274
$188M
Resque
2,054
777
$282K
BullMQ
1,994
595
$3.1M
RQ
1,989
543
$706K
Agenda
1,567
406
$2.1M
Temporal
1,547
381
$27M
Quartz Scheduler
1,414
228
$1.6M
Spring Batch
1,380
204
$7M
node-cron
1,171
296
$153K
Windmill
959
251
$29M
Node Schedule
928
241
$86K
Enqueue
868
230
$2.5M
Schedule
855
193
$96K
Oban
739
274
$598K
Machinery
710
175
$784K
Sidekiq-Cron
659
237
$133K
OCA Queue
646
99
$500K
SQLMesh
636
117
$9.6M
Action Scheduler
635
109
$409K
Scalding
576
120
$2.6M
Taskcluster
572
108
$19M
Apache Gobblin
480
41
$12M
gocron
448
134
$273K
cron-utils
444
80
$526K
HTCondor
396
54
$30M
cron-parser
383
121
$175K
Parallel
360
144
$64K
Ganga
330
55
$3.6M
rufus-scheduler
301
112
$204K
HPCC Systems Platform
291
11
$90M
workerpool
288
93
$213K
node-resque
270
95
$220K
ProActive Workflows & Scheduling
245
11
$8.9M
Apache Tez
241
33
$8.5M
Tron
206
31
$1.6M
limits
161
59
$318K
Worker Farm
127
55
$31K
Queue
108
33
$39K
Better Queue
95
26
$74K
fastq
93
35
$48K
Pegasus WMS
82
13
$43M
Tinypool
75
36
$115K
Are We There Yet
54
21
$9.5K
libEnsemble
53
12
$1.3M
batch
44
21
$25K
ActiveJob::TrafficControl
0
0
$0
Looking for a project that’s not listed?