Trying to set up an Azure worker role with an asynchronous workflow for a polling service. The problem is that the role starts well and after some time becomes unresponsive. I don't know exactly what's the issue as in intellitrace there's no explicit exception. Do you see any problem here which may lead to this unstable situation ? After Mark an…

PollingService.fs

module PollingService
    
    open System.Diagnostics

    let poll interval work =
        let sw = Stopwatch()
        let rec loop() =
            async {
                sw.Restart()
                work()
                sw.Stop()
                let elapsed = int sw.ElapsedMilliseconds
                if elapsed < interval then
                    do! Async.Sleep(interval - elapsed)
                return! loop()
            }
        loop()

WorkerRole.fs

type WorkerRole() =
    inherit RoleEntryPoint() 

    let log message (kind : string) = Trace.TraceInformation(message, kind)
    let cts = new CancellationTokenSource()
    let interval = 2000
    let mutable onStopCalled = false
    let mutable returnedFromRunMethod = false;

    override wr.Run() =

        log "ExternalUserImport entry point called" "Information"

        try
            let polling = PollingService.poll interval (fun () ->
                    log "Doing asynchronous work" "Information"
                )

            Async.Start(polling, cts.Token)

        with
            | ex -> let mutable err = ex.Message
                    if (ex.InnerException <> null) then
                        err <- err + "Inner Exception : " + ex.InnerException.Message
                        
                    Trace.TraceError(err)
        
        while (true) do
            if (onStopCalled = true) then
                Trace.TraceInformation("onStopCalled WorkerRoleB")
                returnedFromRunMethod <- true
                ()

            Thread.Sleep(1000)


    override wr.OnStart() = 

        log "On start called" "Information"

        // Set the maximum number of concurrent connections 
        ServicePointManager.DefaultConnectionLimit <- Environment.ProcessorCount

        Diagnostics.configureDiagnostics()

        base.OnStart()

       
    override wr.OnStop() =
        
        log "On stop called" "Information"

        cts.Cancel()
        cts.Dispose()
        //cts <- new CancellationTokenSource()

        onStopCalled <- true
        while(returnedFromRunMethod = false) do
            Thread.Sleep(1000)
        
        base.OnStop()

As far as I can tell, the Run method starts an async workflow, but never awaits its result. The async workflow runs indefinitely.

This all happens in a while(true) loop, so as soon as it's kicked off one async workflow, it loops around and starts another one. Since this is happening in a tight loop, it seems to me that it would soon run out of threads on the thread pool, since each async workflow runs on its own thread.

I may be wrong, but it looks like thread starvation to me. AFAICT, there are two nested, infinite loops.

I think the problem maybe that the polling is starting on a background thread using Async.Start, and then this frees the main thread to just continue the loop, which will in turn start another polling thread. So I guess eventually your ending up starving the worker of threads?

Arrghh, just beat me to it. Good that we came to the same conclusion thou :)

@ploeh, @colinbull thanks for your help, effectively it's a nice catch. I'll refactor removing "while(true) do" loop and will post the response

For the not working role please look at revision 2

In

while (true) do
            if (onStopCalled = true) then
                Trace.TraceInformation("onStopCalled WorkerRoleB")
                returnedFromRunMethod <- true
                ()

            Thread.Sleep(1000)

The () at the end of the if expression doesn't break the loop.