CUDA-L2 is a system that combines large language models (LLMs) and reinforcement learning (RL) to automatically optimize Half-precision General Matrix Multiply (HGEMM) CUDA kernels. CUDA-L2 systematically outperforms major matmul baselines to date, from the widely-used torch.matmul to state-of-the-art NVIDIA closed-source libraries (cuBLAS, cuBLASLt-heuristic, cuBLASLt-AutoTuning). Paper
- [Dec 2, 2025] Released A100 optimized HGEMM kernels across 1,000 configurations.
Q: Do A100 kernels apply to other machines like RTX 3090 or H100?
A: Ideally, kernels trained on A100 should only be used on A100 if you are targeting speedup. They might have speedup on other machines, but it's not guaranteed. We will progressively release kernels trained on different machines.
Q: What if I need matrix dimensions (M, N, K) not found in your configurations?
A: 1. You can find the nearest neighbor configuration (larger than yours) and pad with zeros. 2. Feel free to post your dimensions on GitHub issues. We are happy to release kernels for your configuration.
- Python: Ensure you have a working Python environment.
- PyTorch: This project requires PyTorch version 2.6.0 or higher.
This project depends on NVIDIA CUTLASS. You must clone specific tag v4.2.1 into a directory named cutlass:
git clone -b v4.2.1 https://github.com/NVIDIA/cutlass.git cutlass
⚠️ Warning: Please ensure you download the correct CUTLASS version (v4.2.1) and set theCUTLASS_DIRenvironment variable correctly. Incorrect CUTLASS setup may cause the project to fail silently or produce no results.
Before building or running the project, you must configure the following environment variables:
CUTLASS_DIR: Points to the directory where you cloned CUTLASS.TORCH_CUDA_ARCH_LIST: Specifies the target GPU architecture (e.g., "8.0" for NVIDIA Ampere / A100 / RTX 30 series).
Run the following commands:
export CUTLASS_DIR=/path/to/your/cutlass
export TORCH_CUDA_ARCH_LIST="8.0"To run the evaluation, use the eval_one_file.sh script. Below is an example command for offline mode:
./eval_one_file.sh --mnk 64_4096_64 --warmup_seconds 5 --benchmark_seconds 10 --base_dir ./results --gpu_device_id 7 --mode offlineFor server mode, you need to specify --target_qps:
./eval_one_file.sh --mnk 64_4096_64 --warmup_seconds 5 --benchmark_seconds 10 --base_dir ./results --gpu_device_id 7 --mode server --target_qps 100| Argument | Description |
|---|---|
--mnk |
Specifies the problem size (e.g., 64_4096_64). |
--warmup_seconds |
Duration of warmup in seconds before timing. |
--benchmark_seconds |
Duration of benchmarking in seconds. |
--base_dir |
Directory to save the compile and output results. |
--gpu_device_id |
The ID of the GPU to use (e.g., 7). |
--mode |
Execution mode. Options are: • offline: Runs the evaluation in offline/batch processing mode.• server: Runs the evaluation in server mode (simulating request-based scenarios). |
--target_qps |
Target Queries Per Second (QPS) for server mode. Required if mode is server. |
If you have any questions, please open a GitHub issue or reach out to us at [email protected].