My mongod has suddenly exited ...

gistfile1.md

Based on the log message, we have been through this line:

https://github.com/mongodb/mongo/blob/master/src/mongo/util/net/listen.cpp#L292

Context :

            SockAddr from;
            int s = accept(*it, from.raw(), &from.addressSize);
            if ( s < 0 ) {
                int x = errno; // so no global issues
                if ( x == ECONNABORTED || x == EBADF ) {
                    log() << "Listener on port " << _port << " aborted" << endl;
                    return;
                }

Okay, return ... that doesn't sound so bad. Right? Wrong!

That's the impl of initAndListen() for the class Listener : https://github.com/mongodb/mongo/blob/master/src/mongo/util/net/listen.h#L44

        void initAndListen(); // never returns unless error (start a thread)

Okay, so the thread exits? That's not so bad, right?
Wrong! And also, wrong!

How'd we get here ...

Like any good C program, we start with main() -- in this case, from

https://github.com/mongodb/mongo/blob/master/src/mongo/db/db.cpp#L1103

here's the end of main() :

int main(int argc, char* argv[]) {
    ...
    StartupTest::runTests();
    initAndListen(cmdLine.port);
    dbexit(EXIT_CLEAN);
    return 0;
}

initAndListen() call goes up to line 542

    void initAndListen(int listenPort) {
        try { 
            _initAndListen(listenPort); 
        }
        catch ( DBException &e ) {
            log() << "exception in initAndListen: " << e.toString() << ", terminating" << endl;
            dbexit( EXIT_UNCAUGHT );
        }
        catch ( std::exception &e ) {
            log() << "exception in initAndListen std::exception: " << e.what() << ", terminating" << endl;
            dbexit( EXIT_UNCAUGHT );
        }
        catch ( int& n ) {
            log() << "exception in initAndListen int: " << n << ", terminating" << endl;
            dbexit( EXIT_UNCAUGHT );
        }
        catch(...) {
            log() << "exception in initAndListen, terminating" << endl;
            dbexit( EXIT_UNCAUGHT );
        }
    }

delegates to line 436:

    void _initAndListen(int listenPort ) {

        Client::initThread("initandlisten");

        Database::_openAllFiles = false;

        Logstream::get().addGlobalTee( new RamLog("global") );

        ...

        Module::initAll();

        if ( scriptingEnabled ) {
            ScriptEngine::setup();
            globalScriptEngine->setCheckInterruptCallback( jsInterruptCallback );
            globalScriptEngine->setGetInterruptSpecCallback( jsGetInterruptSpecCallback );
        }

        repairDatabasesAndCheckVersion();

        ...
 
        if( !noauth ) { 
            // open admin db in case we need to use it later. TODO this is not the right way to 
            // resolve this. 
            writelock lk;
            Client::Context c("admin",dbpath,false);
        }

        listen(listenPort);

        // listen() will return when exit code closes its socket.
        exitCleanly(EXIT_NET_ERROR);
    }

at which point the listen() call brings us up to line 226 :

    void listen(int port) {
        //testTheDb();
        MessageServer::Options options;
        options.port = port;
        options.ipList = cmdLine.bind_ip;

        MessageServer * server = createServer( options , new MyMessageHandler() );
        server->setAsTimeTracker();

        startReplication();
        if ( !noHttpInterface )
            boost::thread web( boost::bind(&webServerThread, new RestAdminAccess() /* takes ownership */));

#if(TESTEXHAUST)
        boost::thread thr(testExhaust);
#endif
        server->run();
    }

What's that server being run()? It's a MessageServer, says message_server.h :

    MessageServer * createServer( const MessageServer::Options& opts , MessageHandler * handler );

Implementation :

https://github.com/mongodb/mongo/blob/master/src/mongo/util/net/message_server_port.cpp#L113

#include "listen.h"

...

    class PortMessageServer : public MessageServer , public Listener {
...
        void run() {
            initAndListen();
        }

        virtual bool useUnixSockets() const { return true; }
    };


    MessageServer * createServer( const MessageServer::Options& opts , MessageHandler * handler ) {
        return new PortMessageServer( opts , handler );
    }

Which is where we came in. Our journey is complete.

To sum up :

• Get ECONNABORTED or EBADF from socket accept() ==>
• Return from initAndListen() ==>
• Return from PortMessageServer::run() ==>
• Return from listen() in db.cpp ==>
• Return from _initAndListen() in db.cpp ==>
• Return from initAndListen() in db.cpp ==>
• Call dbexit(EXIT_CLEAN); and then return from main() ==>
• Game over, dude!