199 lines
5.5 KiB
Markdown
199 lines
5.5 KiB
Markdown
# Kubernetes Deployment for MPC System
|
|
|
|
This directory contains Kubernetes manifests for deploying the MPC system with dynamic party pool service discovery.
|
|
|
|
## Architecture Overview
|
|
|
|
The Kubernetes deployment implements a **Party Pool** architecture where:
|
|
|
|
- **Server parties are dynamically discovered** via Kubernetes service discovery
|
|
- **Session coordinator** automatically selects available parties from the pool
|
|
- **Parties can be scaled** up/down without code changes (just scale the deployment)
|
|
- **No hardcoded party IDs** - each pod gets a unique name as its party ID
|
|
|
|
## Prerequisites
|
|
|
|
- Kubernetes cluster (v1.24+)
|
|
- kubectl configured to access your cluster
|
|
- Docker images built for all services
|
|
- PostgreSQL, Redis, and RabbitMQ deployed (see infrastructure/)
|
|
|
|
## Quick Start
|
|
|
|
### 1. Create namespace
|
|
|
|
```bash
|
|
kubectl apply -f namespace.yaml
|
|
```
|
|
|
|
### 2. Create secrets
|
|
|
|
Copy the example secrets file and fill in your actual values:
|
|
|
|
```bash
|
|
cp secrets-example.yaml secrets.yaml
|
|
# Edit secrets.yaml with your base64-encoded secrets
|
|
# Generate base64: echo -n "your-secret" | base64
|
|
kubectl apply -f secrets.yaml
|
|
```
|
|
|
|
### 3. Create ConfigMap
|
|
|
|
```bash
|
|
kubectl apply -f configmap.yaml
|
|
```
|
|
|
|
### 4. Deploy Session Coordinator
|
|
|
|
```bash
|
|
kubectl apply -f session-coordinator-deployment.yaml
|
|
```
|
|
|
|
The session coordinator requires RBAC permissions to discover party pods.
|
|
|
|
### 5. Deploy Server Party Pool
|
|
|
|
```bash
|
|
kubectl apply -f server-party-deployment.yaml
|
|
```
|
|
|
|
This creates a deployment with 3 replicas by default. Each pod gets a unique name (e.g., `mpc-server-party-0`, `mpc-server-party-1`, etc.) which serves as its party ID.
|
|
|
|
### 6. Deploy other services
|
|
|
|
```bash
|
|
kubectl apply -f message-router-deployment.yaml
|
|
kubectl apply -f account-service-deployment.yaml
|
|
kubectl apply -f server-party-api-deployment.yaml
|
|
```
|
|
|
|
## Scaling Server Parties
|
|
|
|
To scale the party pool, simply adjust the replica count:
|
|
|
|
```bash
|
|
# Scale up to 5 parties
|
|
kubectl scale deployment mpc-server-party -n mpc-system --replicas=5
|
|
|
|
# Scale down to 2 parties
|
|
kubectl scale deployment mpc-server-party -n mpc-system --replicas=2
|
|
```
|
|
|
|
The session coordinator will automatically discover new parties within 30 seconds (configurable via `MPC_PARTY_DISCOVERY_INTERVAL`).
|
|
|
|
## Service Discovery Configuration
|
|
|
|
The session coordinator uses environment variables to configure party discovery:
|
|
|
|
- `K8S_NAMESPACE`: Namespace to search for parties (auto-detected from pod metadata)
|
|
- `MPC_PARTY_SERVICE_NAME`: Service name to discover (`mpc-server-party`)
|
|
- `MPC_PARTY_LABEL_SELECTOR`: Label selector (`app=mpc-server-party`)
|
|
- `MPC_PARTY_GRPC_PORT`: gRPC port for parties (`50051`)
|
|
- `MPC_PARTY_DISCOVERY_INTERVAL`: Refresh interval (`30s`)
|
|
|
|
## RBAC Permissions
|
|
|
|
The session coordinator requires the following Kubernetes permissions:
|
|
|
|
- `pods`: get, list, watch (to discover party pods)
|
|
- `services`: get, list, watch (to discover services)
|
|
|
|
These permissions are granted via the `mpc-session-coordinator-role` Role and RoleBinding.
|
|
|
|
## Health Checks
|
|
|
|
All services expose a `/health` endpoint on their HTTP port (8080) for:
|
|
|
|
- Liveness probes: Detects if the service is alive
|
|
- Readiness probes: Detects if the service is ready to accept traffic
|
|
|
|
## Monitoring Party Pool
|
|
|
|
Check available parties:
|
|
|
|
```bash
|
|
# View all party pods
|
|
kubectl get pods -n mpc-system -l app=mpc-server-party
|
|
|
|
# Check party pod logs
|
|
kubectl logs -n mpc-system -l app=mpc-server-party --tail=50
|
|
|
|
# Check session coordinator logs for party discovery
|
|
kubectl logs -n mpc-system -l app=mpc-session-coordinator | grep "party"
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Session coordinator can't discover parties
|
|
|
|
1. Check RBAC permissions:
|
|
```bash
|
|
kubectl get role,rolebinding -n mpc-system
|
|
```
|
|
|
|
2. Check if service account is correctly assigned:
|
|
```bash
|
|
kubectl get pod -n mpc-system -l app=mpc-session-coordinator -o yaml | grep serviceAccount
|
|
```
|
|
|
|
3. Check coordinator logs:
|
|
```bash
|
|
kubectl logs -n mpc-system -l app=mpc-session-coordinator
|
|
```
|
|
|
|
### Parties not showing as ready
|
|
|
|
1. Check party pod status:
|
|
```bash
|
|
kubectl get pods -n mpc-system -l app=mpc-server-party
|
|
```
|
|
|
|
2. Check readiness probe:
|
|
```bash
|
|
kubectl describe pod -n mpc-system <party-pod-name>
|
|
```
|
|
|
|
3. Check party logs:
|
|
```bash
|
|
kubectl logs -n mpc-system <party-pod-name>
|
|
```
|
|
|
|
## Migration from Docker Compose
|
|
|
|
Key differences from docker-compose deployment:
|
|
|
|
1. **No hardcoded party IDs**: In docker-compose, parties had static IDs (`server-party-1`, `server-party-2`, `server-party-3`). In K8s, pod names are used as party IDs.
|
|
|
|
2. **Dynamic scaling**: Can scale parties up/down without restarting other services.
|
|
|
|
3. **Service discovery**: Automatic discovery via Kubernetes API instead of DNS.
|
|
|
|
4. **DeviceInfo optional**: `DeviceInfo` is now optional in the protocol layer, aligning with international MPC standards.
|
|
|
|
## Advanced Configuration
|
|
|
|
### Custom party selection strategy
|
|
|
|
The default selection strategy is "first N available parties". To implement custom strategies (e.g., load-based, geo-aware), modify the `SelectParties()` method in `services/session-coordinator/infrastructure/k8s/party_discovery.go`.
|
|
|
|
### Party affinity
|
|
|
|
To ensure parties run on different nodes for fault tolerance:
|
|
|
|
```yaml
|
|
spec:
|
|
template:
|
|
spec:
|
|
affinity:
|
|
podAntiAffinity:
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
|
- weight: 100
|
|
podAffinityTerm:
|
|
labelSelector:
|
|
matchLabels:
|
|
app: mpc-server-party
|
|
topologyKey: kubernetes.io/hostname
|
|
```
|
|
|
|
Add this to `server-party-deployment.yaml` under `spec.template`.
|