I'm using websockets & protobuf in my vscode extension. I want to replace the message abstractions & inconsistent code with a nicely defined server/client interface using grpc.
After working on some POC code and running the extension for the first time, I was faced with the following error:
Activating extension 'richardwillis.vscode-gradle' failed:
Failed to load gRPC binary module because it was not installed for the current system
Expected directory: electron-v6.1-darwin-x64-unknown
Found: [node-v72-darwin-x64-unknown]
This problem can often be fixed by running "npm rebuild" on the current system
Original error: Cannot find module '/Users/richardwillis/Projects/badsyntax/vscode-gradle/node_modules/grpc/src/node/extension_binary/electron-v6.1-darwin-x64-unknown/grpc_node.node' Require stack: - /Users/richardwillis/Projects/badsyntax/vscode-gradle/node_modules/grpc/src/grpc_extension.js - /Users/richardwillis/Projects/badsyntax/vscode-gradle/node_modules/grpc/src/client_interceptors.js - /Users/richardwillis/Projects/badsyntax/vscode-gradle/node_modules/grpc/src/client.js - /Users/richardwillis/Projects/badsyntax/vscode-gradle/node_modules/grpc/index.js - /Users/richardwillis/Projects/badsyntax/vscode-gradle/out/java-gradle-tasks/src/main/proto/service_grpc_pb.js -
This is because the grpc node library uses native binary modules, and when I run npm install on my host machine to install the extension dependencies, the installed grpc binaries do not match the platform & version of runtime vscode.
(It's also important to note that vscode does not run npm install when you install the extension from the marketplace. The publisher has to provide the node_modules when publishing.)
The error message above suggests:
This problem can often be fixed by running "npm rebuild" on the current system
When developing locally, the following command fixes the error:
npm rebuild grpc --target=v6.0.1 --runtime=electron --update-binary --fallback-to-build
This will regenerate the binaries to match vscode runtime (electron). You'll note we have to specify an electron runtime version. This complicates things when publishing, as different versions of vscode will be using different electron versions. So for example when a new vscode version is released with a new electron runtime version, the extension breaks. This means we can't rebuild grpc at build time, we have to do it at runtime.
This is the approach the IBM folks settled on with DependencyManager.ts, which installs missing dependecies on extension activate, is somewhat complicated and chunky, and has a few issues, for example: IBM-Blockchain/blockchain-vscode-extension#1621
The issue above exists because no pre-compiled binaries can be downloaded for the matching environment (this fix is awaiting release from google), and there are build issues when grpc attempts to build the binaries from source (which is why it's bad to run npm install on users' machines), as summarized here: IBM-Blockchain/blockchain-vscode-extension#1124 (comment)
Yep. I want to use grpc in my extension! I don't like the hacky approach the IBM folk have taken, but there doesn't seem to be any better option.
What can do be done about it? How can vscode better support this scenario?
This issues seems very relevant, but sadly has not been updated in a bit: microsoft/vscode#658
The solution is to use a pure JavaScript implementation of the gRPC client: @grpc/grpc-js
This is easier said than done though, as it's not particularly easy to generate the correct Typescript type definitions.
The package recommends using @grpc/proto-loader to load the proto files and @grpc/grpc-js to construct the client at runtime:
const packageDefinition = protoLoader.loadSync(protoFileName, options);
const packageObject = grpcLibrary.loadPackageDefinition(packageDefinition);When consuming this code in TypeScript, you'll get type definition errors, and you'll get around this by using Typescript's any, but this is not ideal and goes against the point of using Typescript.
In order to use the correct type definitions for your client and server stubs, you'll need to generate code at build time and generate types from that code.
Using the proto compiler, you'll need to use --grpc_out with --js_out to generate the service stubs and the gRPC client code. The problem is the compiler assumes you're using grpc-node and will add require('grpc') statements into the generated code.
To prevent this, you can do --grpc-out=generate_package_definition, which prevents the compiler from adding any require statements and client constructor code. This allow you to dynmically create the client at runtime, using @grpc/grpc-js's loadPackageDefinition, but now we're back to the same problem of not having a typed client. Also, the Typescript package I use to generate types (ts-protoc-gen) does not play well with "package definitions", and will generate types for the client and not the package definition, which means I cannot import the package defintion from typescript.
Eventually I found an approach that works. It's only slightly hacky.
As @grpc/grpc-js is designed to be a "drop-in" replacement for grpc-node, I generate the server stubs & client code as I would do for grpc-node, using ts-protoc-gen to generate the types, and do a simple find/replace to replace grpc require/import statements with @grpc/grpc-js.
It's important to note that, at the time of writing, @grpc/grpc-js does not have feature parity with grpc-node. See here for feature comparisons: https://github.com/grpc/grpc-node/blob/master/PACKAGE-COMPARISON.md
Here's an example of how I do this:
#!/usr/bin/env bash
OUT_DIR="./out"
TS_OUT_DIR="./src"
IN_DIR="java-gradle-tasks/src/main/proto"
PROTOC="$(npm bin)/grpc_tools_node_protoc"
PROTOC_GEN_TS="$(npm bin)/protoc-gen-ts"
mkdir -p "$OUT_DIR"
$PROTOC \
-I=./ \
--plugin=protoc-gen-ts=$PROTOC_GEN_TS \
--js_out=import_style=commonjs:$OUT_DIR \
--grpc_out=:$OUT_DIR \
--ts_out=service=grpc-node:$TS_OUT_DIR \
"$IN_DIR"/service.proto
sed -i "" -e \
"s/require('grpc')/require('@grpc\/grpc-js')/g" \
"$OUT_DIR/$IN_DIR/"*
sed -i "" -e \
"s/from \"grpc\"/from \"@grpc\/grpc-js\"/g" \
"$TS_OUT_DIR/$IN_DIR/"*In my typescript code, I construct the gRPC client like so:
import * as grpc from '@grpc/grpc-js'
import { GradleTasksClient as GrpcClient } from './java-gradle-tasks/src/main/proto/service_grpc_pb';
const grpcClient = new GrpcClient(
'localhost:1234',
grpc.credentials.createInsecure()
);
I've added a bunch of examples on how to use TypeScript with gRPC here: https://github.com/badsyntax/grpc-js-types