I’ve recently been experimenting with various ways to construct Linux VM images, but for these images to be practical, they need to interact with the outside world. At a minimum, they need to communicate with the host machine.

vsock is a technology specifically designed with VMs in mind. It eliminates the need for a TCP/IP stack or network virtualization to enable communication with or between VMs. At the API level, it behaves like a standard socket but utilizes a specialized addressing scheme.

In the experiment below, we’ll explore using vsock as the transport mechanism for a gRPC service running on a VM. We’ll build this project with Bazel for easy reproducibility. Check out this post if you need an intro to Bazel.

Table of contents

Motivation

There are many use cases for efficient communication between a VM and its host (or between multiple VMs). One simple reason is to create a hermetic environment within the VM and issue commands via RPC from the host. This is the primary driver for using gRPC in this example, but you can easily generalize the approach shown here to build far more complex systems.

GitHub repo

The complete repository is hosted here and serves as the source of truth for this experiment. While there may be minor inconsistencies between the code blocks below and the repository, please rely on GitHub as the definitive source.

Code breakdown

Let’s break down the code step by step:

External dependencies

Here are the external dependencies listed as Bazel modules:

bazel_dep(name = "rules_proto", version = "7.1.0")
bazel_dep(name = "rules_cc", version = "0.2.14")
bazel_dep(name = "protobuf", version = "33.1", repo_name = "com_google_protobuf")
bazel_dep(name = "grpc", version = "1.76.0.bcr.1")

This is largely self-explanatory. The protobuf repository is used for C++ proto-generation rules, and grpc provides the monorepo for Bazel rules to generate gRPC code for the C++ family of languages.

gRPC library generation

The following Bazel targets generate the necessary C++ Protobuf and gRPC libraries:

load("@rules_proto//proto:defs.bzl", "proto_library")
load("@com_google_protobuf//bazel:cc_proto_library.bzl", "cc_proto_library")
load("@grpc//bazel:cc_grpc_library.bzl", "cc_grpc_library")

proto_library(
    name = "vsock_service_proto",
    srcs = ["vsock_service.proto"],
)

cc_proto_library(
    name = "vsock_service_cc_proto",
    deps = [
        ":vsock_service_proto",
    ],
    visibility = [
        "//server:__subpackages__",
        "//client:__subpackages__",
    ],
)

cc_grpc_library(
    name = "vsock_service_cc_grpc",
    grpc_only = True,
    srcs = [
        ":vsock_service_proto",
    ],
    deps = [
        ":vsock_service_cc_proto",
    ],
    visibility = [
        "//server:__subpackages__",
        "//client:__subpackages__",
    ],
)

The protocol definition is straightforward:

syntax = "proto3";

package popovicu_vsock;

service VsockService {
  rpc Addition(AdditionRequest) returns (AdditionResponse) {}
}

message AdditionRequest {
  int32 a = 1;
  int32 b = 2;
}

message AdditionResponse {
  int32 c = 1;
}

It simply exposes a service capable of adding two integers.

Server implementation

The BUILD file is straightforward:

load("@rules_cc//cc:defs.bzl", "cc_binary")

cc_binary(
    name = "server",
    srcs = [
        "server.cc",
    ],
    deps = [
        "@grpc//:grpc++",
        "//proto:vsock_service_cc_grpc",
        "//proto:vsock_service_cc_proto",
    ],
    linkstatic = True,
    linkopts = [
        "-static",
    ],
)

We want a statically linked binary to run on the VM. This choice simplifies deployment, allowing us to drop a single file onto the VM.

The code is largely self-explanatory:

#include <iostream>
#include <memory>
#include <string>

#include <grpc++/grpc++.h>
#include "proto/vsock_service.grpc.pb.h"

using grpc::Server;
using grpc::ServerBuilder;
using grpc::ServerContext;
using grpc::Status;
using popovicu_vsock::VsockService;
using popovicu_vsock::AdditionRequest;
using popovicu_vsock::AdditionResponse;

// Service implementation
class VsockServiceImpl final : public VsockService::Service {
  Status Addition(ServerContext* context, const AdditionRequest* request,
                  AdditionResponse* response) override {
    int32_t result = request->a() + request->b();
    response->set_c(result);
    std::cout << "Addition: " << request->a() << " + " << request->b()
              << " = " << result << std::endl;
    return Status::OK;
  }
};

void RunServer() {
  // Server running on VM (guest)
  // vsock:-1:9999 means listen on port 9999, accept connections from any CID
  // CID -1 (VMADDR_CID_ANY) allows the host to connect to this VM server
  std::string server_address("vsock:3:9999");
  VsockServiceImpl service;

  ServerBuilder builder;
  builder.AddListeningPort(server_address, grpc::InsecureServerCredentials());
  builder.RegisterService(&service);

  std::unique_ptr<Server> server(builder.BuildAndStart());
  std::cout << "Server listening on " << server_address << std::endl;

  server->Wait();
}

int main() {
  RunServer();
  return 0;
}

The only part requiring explanation is the server_address. The vsock: prefix indicates that we’re using vsock as the transport layer. gRPC supports various transports, including TCP/IP and Unix sockets.

The number 3 is the CID, or Context ID. This functions similarly to an IP address. Certain CIDs have special meanings. For instance, CID 2 represents the VM host itself; if the VM needs to connect to a vsock socket on the host, it targets CID 2. CID 1 is reserved for the loopback address. Generally, VMs are assigned CIDs starting from 3.

The 9999 is simply the port number, functioning just as it does in TCP/IP.

Client implementation

The BUILD file is, again, quite simple:

load("@rules_cc//cc:defs.bzl", "cc_binary")

cc_binary(
    name = "client",
    srcs = [
        "client.cc",
    ],
    deps = [
        "@grpc//:grpc++",
        "//proto:vsock_service_cc_grpc",
        "//proto:vsock_service_cc_proto",
    ],
    linkstatic = True,
    linkopts = [
        "-static",
    ],
)

And the C++ code:

#include <iostream>
#include <memory>
#include <string>

#include <grpc++/grpc++.h>
#include "proto/vsock_service.grpc.pb.h"

using grpc::Channel;
using grpc::ClientContext;
using grpc::Status;
using popovicu_vsock::VsockService;
using popovicu_vsock::AdditionRequest;
using popovicu_vsock::AdditionResponse;

class VsockClient {
 public:
  VsockClient(std::shared_ptr<Channel> channel)
      : stub_(VsockService::NewStub(channel)) {}

  int32_t Add(int32_t a, int32_t b) {
    AdditionRequest request;
    request.set_a(a);
    request.set_b(b);

    AdditionResponse response;
    ClientContext context;

    Status status = stub_->Addition(&context, request, &response);

    if (status.ok()) {
      return response.c();
    } else {
      std::cout << "RPC failed: " << status.error_code() << ": "
                << status.error_message() << std::endl;
      return -1;
    }
  }

 private:
  std::unique_ptr<VsockService::Stub> stub_;
};

int main() {
  // Client running on host, connecting to VM server
  // vsock:3:9999 means connect to CID 3 (guest VM) on port 9999
  // CID 3 is an example - adjust based on your VM's actual CID
  std::string server_address("vsock:3:9999");

  VsockClient client(
      grpc::CreateChannel(server_address, grpc::InsecureChannelCredentials()));

  int32_t a = 5;
  int32_t b = 7;
  int32_t result = client.Add(a, b);

  std::cout << "Addition result: " << a << " + " << b << " = " << result
            << std::endl;

  return 0;
}

Running it all together

Bazel shines here. You only need a working C++ compiler on your host system. Bazel automatically fetches and builds everything else on the fly, including the Protobuf compiler.

To get the statically linked server binary:

bazel build //server

Similarly, for the client:

bazel build //client

To create a VM image, I used debootstrap on an ext4 image, as described in this post on X:

This is a quick, albeit hacky, solution for creating a runnable Debian instance.

Next, I copied the newly built server binary to /opt within the image.

Now, the VM can be booted straight into the server binary as soon as the kernel runs:

 qemu-system-x86_64 -m 1G -kernel /tmp/linux/linux-6.17.2/arch/x86/boot/bzImage \
  -nographic \
  -append "console=ttyS0 init=/opt/server root=/dev/vda rw" \
  --enable-kvm \
  -smp 8 \
  -drive file=./debian.qcow2,format=qcow2,if=virtio -device vhost-vsock-pci,guest-cid=3

As shown in the last line, a virtual device is attached to the QEMU VM acting as vsock networking hardware, configured with CID 3.

The QEMU output shows:

[    1.581192] Run /opt/server as init process
[    1.889382] random: crng init done
Server listening on vsock:3:9999

To send an RPC to the server from the host, I ran the client binary:

bazel run //client

The output confirmed the result:

Addition result: 5 + 7 = 12

Correspondingly, the server output displayed:

Addition: 5 + 7 = 12

We have successfully invoked an RPC from the host to the VM!

Under the hood

I haven’t delved into the low-level system API for vsocks, as frameworks typically abstract this away. However, vsocks closely resemble TCP/IP sockets. Once created, they are used in the same way, though the creation API differs. Information on this is readily available online.

Conclusion

I believed it was more valuable to focus on a high-level RPC system over vsock rather than raw sockets. With gRPC, you can invoke a structured RPC on a server running inside the VM. This opens the door to running interesting applications in sealed, isolated environments, allowing you to easily combine different OSes (e.g., a Debian host and an Arch guest) or any platform supporting vsock. Additionally, gRPC allows you to write clients and servers in many different languages and technologies. This is achieved without network virtualization, resulting in increased efficiency.

I hope this was fun and useful to you as well.

Please consider following me on X and LinkedIn for further updates.