AnuDB is a lightweight, serverless document database designed for C++ applications, offering efficient storage of JSON documents through MessagePack serialization. It provides a serverless, schema-less solution for applications requiring flexible data management with robust query capabilities.

Since AnuDB written on top of RocksDB, it ensures atomicity, durability and consistency of your documents. You can adjust memory/CPU usage of AnuDB based on RocksDB options mentioned in StorageEngine.cpp and StorageEngine.h.Based on these configurations, you can get your desired performance results tailored to your specific platform requirements.

Ready to try AnuDB for your project? Follow these simple steps to get started quickly:

1. Install AnuDB
   See the installation guide to set up AnuDB on your machine.

2. Explore Example Code
   Check out our usage examples to understand how to interact with AnuDB.

3. Start Building!
   Dive right in by using AnuDB in your application and start storing your data with lightning-fast speed.

4. Docker Support
   AnuDB supports Docker! Run AnuDB in a containerized environment with ease. See our Docker setup guide to get started.

5. MQTT Support
   AnuDB integrates seamlessly with MQTT, enabling real-time data communication between devices and servers. If you're working on IoT or real-time messaging systems, AnuDB's MQTT support will make your job easier. Check out our MQTT integration guide to get started with MQTT and AnuDB.

6. Benchmark

• Embedded & Serverless: Run directly within your application with no separate server process required
• Embedded Platform Support: Specifically designed for both standard Linux/Windows systems and embedded platforms
• Faster Writes – RocksDB (LSM-tree) backend is optimized for high-speed inserts/updates.
• JSON Document Storage: Store and query complex JSON documents
• RocksDB Backend: Built on RocksDB for high performance and durability
• Transactional Properties: Inherits atomic operations and configurable durability from RocksDB
• Compression Options: Support for ZSTD compression to reduce storage footprint
• Flexible Querying: Support for equality, comparison, logical operators and sorting
• Indexing: Create indexes to speed up queries on frequently accessed fields
• Update Operations: Rich set of document modification operations
• Import/Export: Easy JSON import and export for data migration. Exported JSON file can be also used to migrate data to Postgresql/SQL sever/MySQL/Oracle DB
• C++11 Compatible: Designed to support a wide range of embedded devices efficiently
• Windows/Linux Support: Designed for Windows/Linux environments and embedded Linux platforms
• MQTT Interface: Connect and operate via MQTT protocol from various platforms
• High Concurrency: Supports 32 concurrent nng worker threads(configurable using CONCURRENT_THREADS) for handling MQTT requests
• TLS Security: Secure communications using mbedTLS for encrypted MQTT connections
• Cloud MQTT Support: Compatible with major cloud MQTT brokers
• ACID Compliance: AnuDB inherits atomicity, consistency, isolation, and durability from RocksDB

AnuDB now supports interaction via MQTT protocol, allowing you to connect and operate the database from various platforms without direct C++ integration. The implementation uses nlohmann::json for JSON handling. AnuDB subscribe to MQTT anudb/request topic then if request comes with request id on mentioned topic, then it does that operation then send reponse back on anudb/response/requestid, we can get all responses on anudb/response/+ topic, below diagram to illustrate it.

In below demo, showing AnuDBMqttBridge and mosquitto broker server started, then using client.bash script, ran all supported MQTT commands

MQTT supported commands

To run AnuDB using Docker:

docker pull kanadaanu/anudb-mqtt:v1

You need an MQTT broker (e.g., EMQX, Mosquitto) running before starting AnuDB.

Use one of the following commands based on your setup:

# With TLS (e.g., EMQX public broker, mosquitto broker, etc)
docker run -v <disk folder>:/data kanadaanu/anudb-mqtt:v1 \
  --broker_url tls+mqtt-tcp://<broker_url>:8883 \
  --database_name AnuDB \
  --tls_cacert <> \
  --tls_cert <>
   ...

# Without TLS
docker run -v <disk folder>:/data kanadaanu/anudb-mqtt:v1 \
  --broker_url mqtt-tcp://<broker_url>:1883 \
  --database_name AnuDB

• The /data folder is where you should mount a disk volume to persist the database.
• If you're using a TLS connection, place the required certificate files (e.g., CA cert) inside the mounted /data directory so that the binary can access them.

• C++11 >= compatible compiler
• CMake 3.10 or higher
• ZSTD development libraries (optional, for compression support)
• Mosquitto MQTT broker (for MQTT interface, supported version v3.1.1 but 5.0.0 version can be supported with simple API modification)
• mbedTLS libraries (for TLS support)

There is no need to install any other third party libraries. Below commands will generate AnuDB.exe bin which executes all operations supported by AnuDB, also it will generate liblibanu.a static library, you can use it in your application with required header files.

# Clone the repository
git clone https://github.com/hash-anu/AnuDB.git
cd AnuDB/third_party/nanomq/
git submodule update --init --recursive
cd ../..
mkdir build
cd build
cmake ..
make

or
# Configure with ZSTD compression support
cmake -DZSTD_INCLUDE_DIR=/path/to/zstd/include -DZSTD_LIB_DIR=/path/to/zstd/lib ..
make

For embedded platforms, you may need to use a cross-compiler toolchain:

# Configure with cross-compiler
cmake .. \
  -DCMAKE_C_COMPILER=arm-linux-gnueabihf-gcc \
  -DCMAKE_CXX_COMPILER=arm-linux-gnueabihf-g++ \
  -DCMAKE_C_FLAGS="-march=armv7-a -mfpu=neon -mfloat-abi=hard -DNIOSTATS_CONTEXT=1" \
  -DCMAKE_CXX_FLAGS="-march=armv7-a -mfpu=neon -mfloat-abi=hard -DNIOSTATS_CONTEXT=1 -Wno-error" \
  -DWITH_ALL_TESTS=OFF \
  -DWITH_TESTS=OFF \
  -DWITH_TOOLS=OFF \
  -DWITH_GFLAGS=OFF \
  -DPORTABLE=ON \
  -DBUILD_TESTS=OFF

Examples of AnuDB database have been added to the examples folder. API Overview

#include "Database.h"
#include <iostream>

int main() {
    anudb::Database db("./my_database");
    anudb::Status status = db.open();

    if (!status.ok()) {
        std::cerr << "Failed to open database: " << status.message() << std::endl;
        return 1;
    }

    // Create a collection
    status = db.createCollection("users");

    // Get the collection
    anudb::Collection* users = db.getCollection("users");

    // Create a document
    nlohmann::json userData = {
        {"name", "Hash"},
        {"email", "[email protected]"},
        {"age", 33}
    };
    anudb::Document doc("user001", userData);

    // Insert the document
    status = users->createDocument(doc);

    anudb::Document doc1;
    status = users->readDocument("user001", doc1);
    std::cout << doc1.data().dump(4) << std::endl;
    // Close the database
    db.close();

    return 0;
}

Sample for normal write -> read operation

Sample for get list of collections

Sample for Export then Import operation

AnuDB implements a high-performance concurrent worker architecture to handle MQTT requests efficiently:

• 32 Concurrent Worker Threads: The MQTT bridge spawns 32 nng (nanomsg-next-generation) worker threads to process incoming requests in parallel
• Load Balancing: Requests are automatically distributed across all worker threads for optimal performance
• Thread Safety: All database operations are thread-safe, enabling true concurrent processing
• Asynchronous Processing: Each worker operates independently, preventing slow operations from blocking the entire system
• Scalable Design: The worker architecture scales efficiently on multi-core systems

                           ┌─────────────────┐
                           │  MQTT Broker    │
                           │  (Mosquitto)    │
                           └────────┬────────┘
                                    │
                                    ▼
                           ┌─────────────────┐
                           │  MQTT Bridge    │
                           │   Connector     │
                           └────────┬────────┘
                                    │
                                    ▼
                    ┌───────────────────────────────┐
                    │     Message Dispatcher        │
                    └───┬───────┬───────┬───────┬───┘
          ┌─────────────┘       │       │       └─────────────┐
          │                     │       │                     │
┌─────────▼───────┐   ┌─────────▼───┐   ▲           ┌─────────▼───────┐
│  nng Workers    │   │  nng Workers │   │           │  nng Workers    │
│ (Threads 1-8)   │   │ (Threads 9-16)│   │           │ (Threads 25-32) │
└─────────────────┘   └─────────────┘    │           └─────────────────┘
                                     ┌───┴───────┐
                                     │nng Workers│
                                     │(17-24)    │
                                     └───────────┘
                                          │
                                          ▼
                                  ┌─────────────────┐
                                  │    AnuDB        │
                                  │   Core Engine   │
                                  └─────────────────┘

1. Install and start Mosquitto MQTT broker:

   # On Ubuntu/Debian
   sudo apt-get install mosquitto mosquitto-clients
   sudo systemctl start mosquitto
   
   # On Windows
   # Download and install from https://mosquitto.org/download/
   # Start the service via Windows Services

2. Run the AnuDB MQTT service (after building the project):

   # Move to build/mqtt folder
   # Windows
   # Start the AnuDB MQTT service using AnuDBMqttBridge (TLS example)
   .\AnuDBMqttBridge.exe --broker_url tls+mqtt-tcp://<mqtt broker URL>:8883 --database_name <Your DB name> --tls_cacert <ca-cert path>
   
   # Start the AnuDB MQTT service using AnuDBMqttBridge (with out TLS example)
   .\AnuDBMqttBridge.exe --broker_url mqtt-tcp://<mqtt broker URL>:1883 --database_name <Your DB name>
   
   # Ubuntu/Debian
   ./AnuDBMqttBridge --broker_url mqtt-tcp://<mqtt broker URL>:1883 --database_name <Your DB name>
   
   Note: Use localhost in <mqtt broker URL> if you have installed mosquitto broker on your local host.
   

   Usage of AnuDBMqttBridge:

   Usage: AnuDBMqttBridge.exe --broker_url <url> --database_name <name> [--username <user>] [--password <pass>] [--tls_cacert <path>] [--tls_cert <path>] [--tls_key <path>] [--tls_pass <pass>]
   

We provide client scripts for both Linux (client.bash) and Windows (client.ps1) environments to interact with AnuDB via MQTT:

# Basic usage
./client.bash

# Run load test
./client.bash --load-test --threads 16 --operations 1000

The following commands are supported through the MQTT interface:

Note: To update a document, use the create_document command with an existing document ID. This will overwrite the previous document with the new content.

The find_documents command supports the following query operators:

The provided client scripts offer several features:

• Interactive demo mode showcasing all MQTT operations
• Arbitrary command execution for testing specific commands
• Load testing with configurable threads and operations
• Detailed performance reporting
• Comprehensive help with command-line options

AnuDB leverages RocksDB's storage engine and inherits the following transactional characteristics:

• Atomic Operations: Individual write operations (create, update, delete) are atomic
• Write-Ahead Logging: Configurable WAL for crash recovery
• Point-in-time Snapshots: Read consistency during queries

AnuDB supports various query operations using a JSON-based query language:

#include "Database.h"
#include <iostream>

anudb::Database db("./product_db");
db.open();

// Create a collection
db.createCollection("products");
anudb::Collection* products = db.getCollection("products");

// Create a document
nlohmann::json productData = {
    {"name", "Laptop"},
    {"price", 1299.99},
    {"category", "Electronics"},
    {"specs", {
        {"processor", "i9"},
        {"ram", "32GB"}
    }}
};

anudb::Document doc("prod001", productData);
products->createDocument(doc);

// Find all electronics
nlohmann::json query = {
    {"$eq", {
        {"category", "Electronics"}
    }}
};

std::vector<std::string> results = products->findDocument(query);

// Price range query
nlohmann::json priceRangeQuery = {
    {"$and", {
        {{"$gt", {{"price", 1000.0}}}},
        {{"$lt", {{"price", 2000.0}}}}
    }}
};

results = products->findDocument(priceRangeQuery);

// Update document fields
nlohmann::json updateOp = {
    {"$set", {
        {"price", 1399.99},
        {"specs.ram", "64GB"}
    }}
};

products->updateDocument("prod001", updateOp);

// Add tags to a document
nlohmann::json pushOp = {
    {"$push", {
        {"tags", "sale"}
    }}
};

products->updateDocument("prod001", pushOp);

• Create indexes on fields on which findDoc operations are needed
• Use specific queries rather than broad ones for better performance
• Export/import operations for large collections can be intensive; plan accordingly
• On embedded platforms, consider using the ZSTD compression to reduce storage requirements
• Adjust memory budget and cache size based on your device capabilities
• For MQTT operations, leverage the 32 concurrent worker threads for optimal throughput
• Consider adjusting worker thread count for your specific hardware capabilities
• For high-volume MQTT usage, ensure your broker is properly configured for the expected load

• Configurable memory limits to work on constrained devices
• ZSTD compression reduces storage footprint and I/O operations
• Low CPU overhead suitable for embedded processors
• Minimal dependencies for smaller deployment size
• Supports cross-compilation for various architectures
• MQTT interface enables lightweight clients on resource-constrained devices
• TLS support can be conditionally compiled to reduce binary size when not needed

• Embedded-only; no client-server architecture (except via MQTT interface)
• No built-in replication or sharding