Open source job dependency tools

I was looking for open source Alternatives for job dependency management.

Few things i found

Taskforest is a simple but expressive open-source job scheduler that allows you to chain jobs/tasks and create time dependencies. It uses text config files to specify task dependencies.

http://www.taskforest.com/


schedulix is the Open Source Enterprise Job Scheduling System, which meets the complex requirements of modern IT process automation.

http://www.schedulix.org/

Some other tools popular in Hadoop world

https://github.com/mesos/chronos

Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows.
https://azkaban.github.io/


Luigi

Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
https://github.com/spotify/luigi

Luigi allows you to run batch types of jobs with complex scheduling via Python code.
By default it supports running Hadoop , MySQL , Scalding , Spark etc jobs.
You can see the list of available configurations here
https://luigi.readthedocs.org/en/latest/configuration.html


No comments:

Post a Comment

Please share your views and comments below.

Thank You.