From Notebook to Production API: Containerised Model Serving

The data scientist's Jupyter notebook works. The model produces correct outputs. The demo impresses stakeholders. Then someone asks: "Can we put this in production?"

This is where most enterprise AI initiatives stall — not because the model fails, but because the gap between a working notebook and a reliable production API is larger than anyone estimated. According to Gartner's 2025 AI deployment survey, only 54 percent of AI projects move from pilot to production. The technical barrier is not model quality. It is model serving.

The production requirements gap

A notebook runs on a single machine, processes one request at a time, has no error handling, no authentication, no monitoring, and no recovery mechanism. Production requires all of these simultaneously.

Concurrency. Production APIs handle multiple requests simultaneously. A model serving 100 concurrent users needs request queuing, batch processing, and resource management that a notebook script does not provide.

Reliability. Production systems need health checks, graceful degradation, automatic restart on failure, and defined behaviour when the model is overloaded. Notebook crashes silently. Production crashes page the on-call engineer.

Latency consistency. A notebook takes as long as it takes. Production requires predictable latency — not just average latency, but p95 and p99. Users and downstream systems depend on consistent response times.

Security. Authentication, rate limiting, input validation, output sanitisation. A model that accepts arbitrary inputs without validation is a security liability in production.

Observability. In production, you need to know how the model performs in real time: request volume, latency distribution, error rates, resource utilisation, and output quality metrics.

The model serving stack

Three layers bridge the gap from notebook to production.

Layer 1: Containerisation. Package the model, its dependencies, and the inference code in a Docker container with the NVIDIA Container Toolkit for GPU access. This creates a portable, reproducible unit that runs identically in development, staging, and production. The container eliminates "it works on my machine" — the most common deployment failure mode.

Layer 2: Serving framework. A model serving framework handles the HTTP API, request batching, model loading, and GPU memory management. The three production-grade options in 2026 are:

vLLM — optimised for LLM inference with PagedAttention for efficient GPU memory management. Handles continuous batching, tensor parallelism for multi-GPU setups, and quantised model support. The default choice for most LLM serving deployments.

Text Generation Inference (TGI) by Hugging Face — tight integration with the Hugging Face model hub. Simpler setup than vLLM for standard models, with built-in monitoring and OpenAI-compatible API endpoints.

NVIDIA Triton Inference Server — the enterprise-grade option for multi-model, multi-framework deployments. Supports ensemble models, dynamic batching, and model versioning. More complex to configure, but the most capable for organisations running diverse model types.

Layer 3: Orchestration. Kubernetes manages container lifecycle, scaling, and resource allocation. Load balancers distribute requests across model replicas. Auto-scaling rules add or remove replicas based on demand. Health checks detect and replace failed containers automatically.

For Mittelstand companies, Kubernetes is often overkill for initial deployments. A single container running behind a reverse proxy (NGINX, Traefik) with Docker Compose handles 1 to 3 model deployments. Scale to Kubernetes when you operate more than 5 models or need auto-scaling.

The deployment patterns

Blue-green deployment. Run the new model version alongside the old. Switch traffic to the new version after validation. If problems emerge, switch back instantly. This requires double the infrastructure during the transition but eliminates downtime and enables instant rollback.

Canary deployment. Route a small percentage of traffic (5 to 10 percent) to the new model version. Monitor performance metrics against the existing version. Gradually increase traffic if metrics hold. Roll back if they degrade. This is the safest pattern for production model updates.

Shadow deployment. Route all traffic to both the old and new model versions. Use the old model's outputs for production responses. Compare the new model's outputs offline. This detects performance differences without any user impact but doubles inference cost during testing.

The performance optimisation checklist

Before declaring a model production-ready, validate these:

Quantisation. INT8 or INT4 quantisation reduces memory footprint and increases throughput by 2 to 4x with minimal accuracy loss for most inference tasks. vLLM and TGI both support GPTQ and AWQ quantisation natively. Always measure accuracy on your specific task before and after quantisation — some tasks are more sensitive than others.

Batching. Continuous batching — processing new requests as slots become available rather than waiting for fixed batch windows — increases GPU utilisation from 30 to 40 percent (typical without batching) to 70 to 85 percent. Both vLLM and TGI support this natively.

KV-cache management. For transformer models, key-value cache management determines how many concurrent sequences a GPU can serve. vLLM's PagedAttention algorithm manages KV-cache like virtual memory — a significant efficiency improvement over naive caching strategies.

Warm-up. Load the model and run inference on representative inputs before accepting production traffic. Cold-start inference — the first request after model loading — is 2 to 5x slower than steady-state and should never reach users.

Book a fit call to plan your production model serving architecture. We design the deployment pattern, serving framework, and scaling strategy matched to your team's capability and operational requirements. Book your fit call →

References: Gartner, "AI in the Enterprise: Deployment Survey," 2025 (54% pilot-to-production rate); vLLM Project, "Efficient Memory Management for Large Language Model Serving with PagedAttention," 2023; Hugging Face, "Text Generation Inference Documentation," 2026; NVIDIA, "Triton Inference Server Architecture Guide," 2026.