inference/sglang/sgl-kernel

SGL Kernel

Kernel Library for SGLang

Installation

For CUDA 11.8:

pip3 install sgl-kernel -i https://docs.sglang.ai/whl/cu118

For CUDA 12.1 or CUDA 12.4:

pip3 install sgl-kernel

Developer Guide

Development Environment Setup

Use Docker to set up the development environment. See Docker setup guide.

Create and enter development container:

docker run -itd --shm-size 32g --gpus all -v $HOME/.cache:/root/.cache --ipc=host --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
docker exec -it sglang_zhyncs /bin/zsh

Project Structure

Dependencies

Third-party libraries:

• CCCL
• CUTLASS
• FlashInfer

Kernel Development

Steps to add a new kernel:

1. Implement the kernel in csrc
2. Expose the interface in include/sgl_kernel_ops.h
3. Create torch extension in csrc/torch_extension.cc
4. Update setup.py to include new CUDA source
5. Expose Python interface in python

Build & Install

Development build:

make build

Note:

The sgl-kernel is rapidly evolving. If you experience a compilation failure, try using make rebuild.

Testing & Benchmarking

1. Add pytest tests in tests/
2. Add benchmarks using triton benchmark in benchmark/
3. Run test suite

Release new version

Update version in pyproject.toml and version.py