System Design: Scheduler

gistfile1.txt

design a scheduler:


 - schedule for a given time   input->(fn, time)
 - cron like schedule ( every night at 12:00 AM)



Functionality:
- configure 

schedule looks like this:


POST /schedules -d {'config'}
-header: serviceName
-header: x-auth: digest (service_id+nonce)
-header: nonce
{
job_detail: {
    job_name:
    job_priority: (1-10)
    job_function: 
    job_type: cron/fixed-schedule
    cron: cron_expression (* * * * *) (upto second's granularity)
    fixed-schedule: time_in_the_future (unix timestamp in GMT) 
    retry_count: 3
    backoff: (2)^attempt 
}
target:
{
    type: webhook/event/rpc,
    value: "webhook endpoint"
}
}

Scheduler invocation:
webhook flow 
serviceb.gp/api/v1/notify_users
- curl -X post webhook -d {}
 - x-auth: 
 - x-service_id:

event flow:


scheduler -> api gateway ->  aws SNS ->    lambda (function url)



allowed_services=sid1:$rateLimit,sid2




- statically conifugre when app starts up 
- library (programmatic way to register schedules) (import scheduler.schedule(fn,....))
- wrap this in a service (http rest api call) POST /schedules -d {'config'}    GET /schedules/:id
    invocation: webhook (application logic), scheduler emits event, rpc (grpc)


Visibility into the scheduler system:
    Control Center (UI) - rbac (auth)

    Queues, schedules, crons (filter by date, which is going to execute soon)

    way to manually execute (in case of failures)

    dlq:


APM:
 - log aggregation, tracing, monitoring of request count , cpu , mem usage



distributed scheduler:

s1, s2, s3  :
    - idempotency: 

handling many crons
handing many crons with priorities