Skip to content

Instantly share code, notes, and snippets.

@ataft
Forked from bermi/README.md
Last active May 31, 2023 16:54
Show Gist options
  • Save ataft/42b7cc3d4055cf6b5a62a5f1f57da4b2 to your computer and use it in GitHub Desktop.
Save ataft/42b7cc3d4055cf6b5a62a5f1f57da4b2 to your computer and use it in GitHub Desktop.
DuckDB cluster mode issues

Issue resolved with 'READ_ONLY' access_mode, as per: duckdb/duckdb#7743 (comment)


This gist showcases how DuckDB on nodejs fails to with "Connection was never established or has been closed already" when running in cluster mode.

Calling (using an in :memory: database)

node cluster.js

outputs the correct result

Forked child
Forked child
Forked child
Forked child
Child done
Child done
Child done
Child done
42
42
42
42

If we use a file database

node cluster.js some-db.duckdb

it'll only run the query once, all other processes in the cluster will fail

Forked child
Forked child
Forked child
Forked child
Child done
Child done
Child done
Child done
42

[Error: Connection Error: Connection was never established or has been closed already] {
  errno: -1,
  code: 'DUCKDB_NODEJS_ERROR',
  errorType: 'Connection'
}
const cluster = require('cluster');
if (cluster.isMaster) {
// Fork workers.
for (var i = 0; i < 4; i++) {
cluster.fork();
}
cluster.on('error', function(err) {
console.error(err);
});
} else {
console.log('Forked child');
require('./index')(process.argv[2] || ':memory:')
console.log('Child done')
}
const duckdb = require('duckdb')
module.exports = (dbPath = ':memory:') => {
console.log('dbPath', dbPath);
const accessMode = (dbPath===':memory:'?'AUTOMATIC':'READ_ONLY');
console.log('accessMode', accessMode);
const db = new duckdb.Database(dbPath, {access_mode:accessMode});
db.all('SELECT 42 AS fortytwo', function(err, res) {
if (err) {
throw err;
}
console.log(res[0].fortytwo)
});
}
if (require.main === module) {
module.exports(process.argv[2] || ':memory:')
}
{
"name": "duckdbcluster",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "MIT",
"dependencies": {
"duckdb": "^0.8.0"
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment