I was looking for open source Alternatives for job dependency management.
Few things i found
Taskforest is a simple but expressive open-source job scheduler that allows you to chain jobs/tasks and create time dependencies. It uses text config files to specify task dependencies.
http://www.taskforest.com/
schedulix is the Open Source Enterprise Job Scheduling System, which meets the complex requirements of modern IT process automation.
http://www.schedulix.org/
Some other tools popular in Hadoop world
https://github.com/mesos/chronos
Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows.
https://azkaban.github.io/
Luigi
Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
https://github.com/spotify/luigi
Luigi allows you to run batch types of jobs with complex scheduling via Python code.
By default it supports running Hadoop , MySQL , Scalding , Spark etc jobs.
You can see the list of available configurations here
https://luigi.readthedocs.org/en/latest/configuration.html
Few things i found
Taskforest is a simple but expressive open-source job scheduler that allows you to chain jobs/tasks and create time dependencies. It uses text config files to specify task dependencies.
http://www.taskforest.com/
schedulix is the Open Source Enterprise Job Scheduling System, which meets the complex requirements of modern IT process automation.
http://www.schedulix.org/
Some other tools popular in Hadoop world
https://github.com/mesos/chronos
Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows.
https://azkaban.github.io/
Luigi
Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
https://github.com/spotify/luigi
Luigi allows you to run batch types of jobs with complex scheduling via Python code.
By default it supports running Hadoop , MySQL , Scalding , Spark etc jobs.
You can see the list of available configurations here
https://luigi.readthedocs.org/en/latest/configuration.html
No comments:
Post a Comment
Please share your views and comments below.
Thank You.