The FlowJob task scheduling framework supports delay, timing, and corn tasks; task distribution supports random, polling, consistent hash, broadcast, failover, MapReduce and other modes; supports task scheduling, DAG workflow, and supports execution in the workflow rule judgment.

flowjobIt is mainly used to build a unified task scheduling platform, which is convenient for all business parties to access and use. When the project was designed, related issues such as scalability, stability, and scalability were considered, and it can be used as the company’s task scheduling middleware.

Features

Flowjob is mainly divided into the following parts:

• Broker: The central node is responsible for task scheduling.
• Worker: The working node is mainly responsible for the specific execution of the task.
• Console: Through the Api provided by Broker, perform some control operations such as task creation/update.
• Registry: Registration center, currently using DB as the registration center. The corresponding interface is provided, which can be flexibly packaged based on other components such as zk, nacos, etc.
• Datasource: The database is used to persist running data

scheduling type

• Fixed speed: After the job is created, after each schedule is issued, the job schedule is triggered again after a fixed interval of time.
• Fixed delay: After the job is created, after each job delivery is completed (successful or failed), after a fixed time interval, the job scheduling is triggered again.
• CRON: Specify the time point when the job triggers the scheduling through the CRON expression.

load policy

• Random: Randomly send jobs to a certain worker for execution.
• Round Robin: Assign tasks to workers one by one.
• Least frequently used: Send the job to the worker that receives the least number of jobs within a time window.
• Least recently used: Send the job to the worker who has not accepted a worker for the longest time within a time window.
• Consistent hash: Jobs with the same parameters will always be sent to the same machine.
• Designated node: Let the job be assigned to a worker for execution.

Node filter method

1. Executor: The task will only be sent to the worker that contains the executor corresponding to the task.
2. Label: The task will only be delivered to the worker with the specified label.
3. Capacity: worker-based task queue/CPU/memory usage filtering (TODO)

Task trigger method

• API: trigger task execution by specifying api.
• Scheduling: Broker automatically assembles data and schedules corresponding tasks.

task type

A task can be a single task/workflow. A single task can be understood as a workflow task with only one node.

• Normal: corresponding to an executor, the execution ends and the task ends.
• Broadcasting: At the point in time of delivery, a task is delivered to each node that can be delivered, so the subtask execution is completed before the current task is completed.
• Map: It is divided into two steps: split and map. When splitting, the tasks are split, and the map executes each split task.
• MapReduce: Compared with Map, there are more Reduce processes, which can summarize the execution results of all Map tasks.

Server deployment

Environmental requirements

Step1: Database configuration

Currently usingflywaydbDynamically perform data initialization operations

Step2: Service packaging

Modify configuration as needed

In the root directory of the project, execute the following command to package and compile, through-PThe parameter specifies the environment, such as the development environment is-P dev

mvn clean package -Dmaven.test.skip=true -P dev

Worker deployment

For web applications (host applications) that need to use workers, you can refer todemo.

Step1: Add dependencies

forMavenproject

<dependency>
    <groupId>io.github.limbo-world</groupId>
    <artifactId>flowjob-worker-spring-boot-starter</artifactId>
    <version>1.0.1</version>
</dependency>

Step2: Modify the configuration

If you have any suggestions for this project or want to join us, you can use the following methods: Welcome to submit issues for correction.