Phase 5: Guardrails & Defensive Controls · 60 min · Python · NeMo Guardrails
NeMo Guardrails & Colang
Colang doesn't make the model safer — it makes the conversation's shape enforceable.
Real-world application: NeMo Guardrails is the most-cited self-hosted, open-source guardrail runtime in production LLM stacks. Being able to explain what Colang rails actually enforce — and where they stop and a real access-control layer has to start — signals you've deployed it, not just skimmed the README.
What you will learn
- Explain NeMo Guardrails' self-hosted runtime architecture and how it wraps an LLM call with input, dialog, and output rails
- Read and write basic Colang flows for topical restriction and conversational safety
- Distinguish what a topical rail can enforce (conversation shape and scope) from what it cannot (it is not a data access control or tool authorization layer)
- Map a hand-built DSL interpreter's concepts onto real NeMo Guardrails/Colang syntax
The Problem
The layered model from Lesson 1 tells you where guardrails need to run. It doesn't tell you how to actually build the "restrict what this bot will talk about" layer without writing a pile of brittle if/else string matching that breaks the first time someone phrases a request slightly differently.
NVIDIA's NeMo Guardrails is the most widely deployed answer to that specific problem. It's a self-hosted, open-source runtime that sits between your application and the LLM, and its programming model — Colang — lets you define conversational rails declaratively: "if the user is asking about X, respond with Y and stop" rather than writing that logic imperatively for every case. It's not a magic safety layer; it's a dialog management system that happens to be extremely good at the specific job of keeping a conversation inside a defined scope.
The reason this matters for a support bot, an internal tool, or any narrow-purpose assistant: a banking chatbot that will cheerfully answer cooking questions, give medical advice, or debate politics isn't just off-brand — it's outside the scope your legal, compliance, and support teams actually reviewed and approved. Topical rails are how you keep the deployed system inside the box it was scoped for, enforced by code rather than by hoping the system prompt holds.
How NeMo guardrails actually works
NeMo Guardrails wraps your LLM calls with three groups of rails, configured in a config.yml:
- Input rails run on the user's message before it reaches the model — this is where you'd wire in jailbreak detection or PII stripping.
- Dialog rails — the Colang flows this lesson focuses on — shape what the conversation is allowed to be about and how it responds to specific intents.
- Output rails run on the generated response before it's returned — fact-checking, moderation, or output validation.
The runtime itself is self-hosted: you deploy LLMRails alongside your application, point it at a model provider (OpenAI, a local model, whatever you're already using), and it intercepts the conversation. This matters architecturally — you're not sending your traffic to a third-party API the way you would with a hosted guardrail service (that's Lesson 3's territory); you own the runtime and the rails logic.
Colang's core primitive is the canonical form — a named intent defined by a handful of example utterances:
define user express off topic cooking
"how do i bake a cake"
"what's a good pasta recipe"
"give me a recipe for lasagna"
At runtime, NeMo Guardrails doesn't string-match the user's message against these examples. In the default Colang 1.0 dialog flow it makes an LLM call to decide the canonical form: it retrieves the example utterances most similar to the message (by embedding) and puts them in the prompt as few-shot examples, and the model names the user intent. That is what makes topical rails robust to paraphrasing: "what's a decent recipe for lasagna" maps to express off topic cooking even though it's not one of the literal example strings. If you would rather not pay for that call, rails.dialog.user_messages.embeddings_only: True skips the LLM and takes the nearest example by embedding similarity, with a threshold and a fallback intent for messages that match nothing well.
A topical rail is not an access-control layer
Colang flows answer one question: "is this conversation turn in scope?" They have nothing to do with who is allowed to see which data, and nothing to do with whether a proposed tool call is safe to execute. A topical rail that successfully blocks every cooking question does precisely nothing to stop a tool-call injection or a cross-tenant retrieval leak. Treat Colang rails as one layer of the four from Lesson 1 — specifically, they're your dialog-level input/scope control — not a replacement for retrieval scope filtering or tool-call authorization.
Writing flows: canonical forms, bot responses, and stop
A flow ties a canonical form to a response and an action:
define bot refuse off topic
"I'm sorry, I can only help with questions about your accounts, cards, and transactions."
define flow block cooking topic
user express off topic cooking
bot refuse off topic
stop
stop is the operative keyword: it halts the pipeline for that turn. The bot's canned response becomes the final answer, and the response-generation step never runs. That is what makes topical rails cheap and predictable at scale: an off-topic request never gets a full answer composed by your main model. (It is not zero calls: by default the intent-classification step above is itself an LLM call, which you can configure to use a small, cheap model, or remove with embeddings_only.) Compare that to relying on a system-prompt instruction like "don't discuss cooking" — the model still has to generate a full completion, it can still be argued out of the constraint mid-conversation, and you're paying for every attempt.
If no stop flow matches a given turn, the conversation falls through to normal dialog handling and the core model generates a response as usual — there's no special "allow" keyword needed; the absence of a match is the allow path.
Why does a Colang flow that uses stop cost less at scale than relying on a system-prompt instruction like "don't discuss cooking" to enforce the same restriction?
When a stop flow matches, the rail's canned response becomes the final answer and the response-generation step never runs. The turn is not free: by default NeMo already made one LLM call to classify the message into a canonical form, and only embeddings_only removes that. But the expensive part, a full answer from your main model, is skipped. A system-prompt restriction still requires that full completion for every off-topic attempt, and the model can still be argued out of the rule mid-conversation. So the rail is both cheaper and more predictable: the refusal text is fixed, not composed afresh each time.
A team defines a Colang rail that blocks any message matching the canonical form express off topic cooking and responds with a refusal, using stop. A user asks: "My grandmother used to make a lasagna with a special sauce — can you help me recreate her recipe for a family reunion?" The rail's example utterances don't literally contain this phrasing. What determines whether this message gets blocked, and why is that different from a keyword blocklist?
NeMo Guardrails' canonical-form matching is semantic, not literal. It indexes the example utterances under define user ...; at runtime it retrieves the examples closest to the incoming message by embedding and asks the LLM which canonical form the message expresses (or, with embeddings_only, takes the nearest example directly). This is why Colang rails hold up better than a hand-rolled keyword blocklist: "recreate her recipe for a family reunion" is close in meaning to "cooking questions" though it shares almost no vocabulary with the examples. A blocklist keyed on "recipe" or "cook" would miss endless paraphrases; a rail matched on meaning catches the intent instead of the words.
Build
Build it
Before diving into the full interpreter below, here's a minimal taste of canonical-form matching using nothing but token overlap — the same idea the real DSL implements, compressed to a single scoring function.
import re
def tokenize(text):
return set(re.findall(r"[a-z0-9']+", text.lower()))
CANONICAL_FORMS = {
"off_topic_cooking": ["how do i bake a cake", "what's a good pasta recipe"],
"ask_account_balance": ["what's my account balance", "show me my balance"],
}
def match(message, threshold=0.28):
msg_tokens = tokenize(message)
best_form, best_score = None, 0.0
for form, examples in CANONICAL_FORMS.items():
for example in examples:
ex_tokens = tokenize(example)
overlap = len(msg_tokens & ex_tokens)
union = len(msg_tokens | ex_tokens)
score = overlap / union if union else 0.0
if score > best_score:
best_form, best_score = form, score
return (best_form, round(best_score, 2)) if best_score >= threshold else (None, round(best_score, 2))
for msg in ["what's a decent recipe for lasagna", "how much money do i have in checking", "what's the weather"]:
form, score = match(msg)
print(f"{msg!r:55} -> form={form} score={score}")
The code artifact lesson_05_02_colang_rails.py builds a small Colang-style DSL interpreter from scratch — a parser for define user / define bot / define flow blocks, canonical-form matching via token-overlap similarity (a deliberately simple stand-in for NeMo Guardrails' LLM-plus-retrieval matching), and an engine that executes flows, including the stop action that short-circuits the pipeline before the mocked core LLM ever runs. It ends with COLANG_MAPPING_NOTES, a detailed mapping from this teaching DSL onto real NeMo Guardrails/Colang syntax, project layout, and the LLMRails / RailsConfig API you'd actually use in production — including an honest note about Colang 2.0's different grammar in newer releases.
"""
lesson_05_02_colang_rails.py
A minimal, hand-built interpreter for a Colang-style topical rail DSL.
NeMo Guardrails itself is a heavy dependency (it pulls in an embedding
model for canonical-form matching, an LLM client, and a fair amount of
runtime machinery) so this lesson builds a small, honest stand-in: a
parser + interpreter for a DSL that mirrors the *shape* of real Colang
(`define user ...`, `define bot ...`, `define flow ...`, `stop`) closely
enough that the concepts transfer directly. See COLANG_MAPPING_NOTES at
the bottom of this file for exactly where our engine is a simplification
and what the real NeMo Guardrails / Colang equivalent looks like.
Usage:
python lesson_05_02_colang_rails.py
Requirements:
None to run the interpreter — standard library only.
Optional: pip install nemoguardrails (only used to detect whether the
real package is available; this script does not require it to run)
"""
from __future__ import annotations
import re
from dataclasses import dataclass, field
try:
import nemoguardrails # noqa: F401
NEMOGUARDRAILS_AVAILABLE = True
except ImportError:
NEMOGUARDRAILS_AVAILABLE = False
# pip install nemoguardrails
# (not required for this script — see COLANG_MAPPING_NOTES for how this
# DSL maps onto a real RailsConfig + LLMRails deployment)
# ── The DSL ───────────────────────────────────────────────────────────────
#
# Three block types, mirroring real Colang 1.0 syntax:
# define user <canonical form name> -- example utterances that mean this
# define bot <canonical form name> -- candidate response text(s)
# define flow <flow name> -- user form -> bot form -> action
#
# "stop" halts the pipeline right there (the rail's response IS the final
# answer — no response is generated). "continue" means "this input is
# in scope, hand it to the underlying model as normal."
TOPICAL_RAILS = """
define user express off topic cooking
"how do i bake a cake"
"what's a good pasta recipe"
"give me a recipe for lasagna"
"how long do i roast a chicken"
define user express off topic medical
"what medication should i take for a headache"
"can you diagnose my symptoms"
"is this rash dangerous"
define user ask account balance
"what's my account balance"
"how much money do i have in checking"
"show me my balance"
define user ask about fees
"why was i charged an overdraft fee"
"what are your wire transfer fees"
define bot refuse off topic
"I'm sorry, I can only help with questions about your accounts, cards, and transactions."
define flow block cooking topic
user express off topic cooking
bot refuse off topic
stop
define flow block medical topic
user express off topic medical
bot refuse off topic
stop
define flow allow account balance
user ask account balance
continue
define flow allow fee question
user ask about fees
continue
"""
# ── Parser ────────────────────────────────────────────────────────────────
@dataclass
class Flow:
name: str
user_form: str | None = None
bot_form: str | None = None
action: str = "continue" # "stop" | "continue"
@dataclass
class ParsedRails:
user_forms: dict[str, list[str]] = field(default_factory=dict)
bot_forms: dict[str, list[str]] = field(default_factory=dict)
flows: list[Flow] = field(default_factory=list)
def parse_colang_lite(source: str) -> ParsedRails:
rails = ParsedRails()
lines = [ln.rstrip() for ln in source.strip().splitlines()]
i = 0
while i < len(lines):
line = lines[i].strip()
if not line:
i += 1
continue
m_user = re.match(r'define user (.+)', line)
m_bot = re.match(r'define bot (.+)', line)
m_flow = re.match(r'define flow (.+)', line)
if m_user:
name = m_user.group(1).strip()
examples = []
i += 1
while i < len(lines) and lines[i].strip().startswith('"'):
examples.append(lines[i].strip().strip('"').lower())
i += 1
rails.user_forms[name] = examples
continue
if m_bot:
name = m_bot.group(1).strip()
responses = []
i += 1
while i < len(lines) and lines[i].strip().startswith('"'):
responses.append(lines[i].strip().strip('"'))
i += 1
rails.bot_forms[name] = responses
continue
if m_flow:
name = m_flow.group(1).strip()
flow = Flow(name=name)
i += 1
while i < len(lines) and lines[i].strip() and not lines[i].strip().startswith("define"):
step = lines[i].strip()
if step.startswith("user "):
flow.user_form = step[len("user "):].strip()
elif step.startswith("bot "):
flow.bot_form = step[len("bot "):].strip()
elif step == "stop":
flow.action = "stop"
elif step == "continue":
flow.action = "continue"
i += 1
rails.flows.append(flow)
continue
i += 1
return rails
# ── Canonical-form matching ─────────────────────────────────────────────
#
# Real NeMo Guardrails (Colang 1.0) maps a message to a canonical form with
# an LLM call: it retrieves the example utterances most similar to the
# message (by embedding) and asks the model to name the user intent, with
# those examples as few-shot context. Setting embeddings_only skips the LLM
# and takes the nearest example directly. We don't want an embedding model
# or an LLM in a teaching example, so this uses token-overlap (Jaccard)
# similarity: it matches on shared words, so unlike NeMo it misses
# paraphrases. What it keeps is the shape: "pick the canonical form whose
# examples are closest to what the user said".
def _tokenize(text: str) -> set[str]:
return set(re.findall(r"[a-z0-9']+", text.lower()))
def match_user_form(message: str, user_forms: dict[str, list[str]], threshold: float = 0.28) -> tuple[str | None, float]:
msg_tokens = _tokenize(message)
best_form, best_score = None, 0.0
for form_name, examples in user_forms.items():
for example in examples:
ex_tokens = _tokenize(example)
if not msg_tokens or not ex_tokens:
continue
overlap = len(msg_tokens & ex_tokens)
union = len(msg_tokens | ex_tokens)
score = overlap / union if union else 0.0
if score > best_score:
best_form, best_score = form_name, score
if best_score >= threshold:
return best_form, best_score
return None, best_score
# ── Engine ───────────────────────────────────────────────────────────────
@dataclass
class RailDecision:
message: str
matched_form: str | None
match_score: float
matched_flow: str | None
action: str # "stop" | "continue"
bot_message: str | None
final_response: str
def mock_core_llm(message: str) -> str:
"""Stand-in for the underlying LLM the rails wrap. Real deployments
call an actual model here; this mock keeps the demo self-contained."""
lowered = message.lower()
if "balance" in lowered:
return "Your checking account balance is $2,431.10."
if "fee" in lowered or "overdraft" in lowered:
return "Overdraft fees are $0 on this account tier; wire transfers are $15 domestic / $45 international."
return "I can help with account, card, and transaction questions — could you rephrase your question?"
class ColangLiteEngine:
def __init__(self, source: str):
self.rails = parse_colang_lite(source)
def process(self, message: str) -> RailDecision:
matched_form, score = match_user_form(message, self.rails.user_forms)
matched_flow = None
if matched_form is not None:
matched_flow = next(
(f for f in self.rails.flows if f.user_form == matched_form), None
)
if matched_flow is not None and matched_flow.action == "stop":
bot_message = None
if matched_flow.bot_form and self.rails.bot_forms.get(matched_flow.bot_form):
bot_message = self.rails.bot_forms[matched_flow.bot_form][0]
return RailDecision(
message=message, matched_form=matched_form, match_score=score,
matched_flow=matched_flow.name, action="stop",
bot_message=bot_message, final_response=bot_message or "[blocked, no bot response defined]",
)
# No matching "stop" flow -> in scope. Hand off to the core model.
flow_name = matched_flow.name if matched_flow else None
core_response = mock_core_llm(message)
return RailDecision(
message=message, matched_form=matched_form, match_score=score,
matched_flow=flow_name, action="continue",
bot_message=None, final_response=core_response,
)
# ── Demo ─────────────────────────────────────────────────────────────────
TEST_MESSAGES = [
"What's my account balance right now?",
"How do I bake a cake for my daughter's birthday?",
"Can you diagnose my symptoms, I have a headache and a fever?",
"Why was I charged an overdraft fee last week?",
"What's the weather like today?", # out of scope, but no rail defined for it
]
def run_demo() -> None:
engine = ColangLiteEngine(TOPICAL_RAILS)
print(f"nemoguardrails package installed: {NEMOGUARDRAILS_AVAILABLE}")
print("(this demo runs identically either way — it never imports the real package)\n")
for msg in TEST_MESSAGES:
decision = engine.process(msg)
print("=" * 70)
print(f"User: {msg}")
print(f" matched_form={decision.matched_form} score={decision.match_score:.2f}")
print(f" matched_flow={decision.matched_flow} action={decision.action.upper()}")
print(f" Response: {decision.final_response}")
# ── Mapping to real NeMo Guardrails / Colang ────────────────────────────
COLANG_MAPPING_NOTES = """
How this DSL maps onto a real NeMo Guardrails deployment
==========================================================
1. Canonical form matching (define user ...)
Ours: token-overlap (Jaccard) similarity against example utterances.
Real: NeMo Guardrails (Colang 1.0) retrieves the example utterances
closest to the message by embedding and asks the LLM to name the
canonical form, with those examples as few-shot context. Set
rails.dialog.user_messages.embeddings_only to skip the LLM and
take the nearest example directly. Same question — "which
canonical form does this utterance match" — answered by meaning
rather than by shared words.
2. Flow definitions (define flow ...)
Ours and real Colang (1.0, the syntax most current tutorials use) are
nearly identical in shape:
define flow block cooking topic
user express off topic cooking
bot refuse off topic
stop
`stop` is a real Colang action that halts further processing for that
turn — the rail's own response becomes the final answer and the
response-generation step never runs (the intent-classification call
has already happened). This is exactly the mechanism this
lesson's `action = "stop"` models.
3. What we call "continue" doesn't exist as a keyword in real Colang.
In a real deployment, if no rail's `stop` flow matches, the turn simply
falls through to the dialog rails and then core LLM generation as
normal — there's nothing to opt into. We made it explicit here so the
demo's control flow is easy to read.
4. Project layout for a real deployment:
my_rails_project/
config.yml # models, rails: { input: [...], dialog: [...], output: [...] }
rails/
topical.co # the .co Colang files, one or many
actions.py # custom Python actions callable from flows
config.yml wires rails into named groups (input rails run before the
LLM sees the message, output rails run after it responds, dialog rails
like the ones above shape the conversation itself), then:
from nemoguardrails import LLMRails, RailsConfig
config = RailsConfig.from_path("my_rails_project")
rails = LLMRails(config)
response = rails.generate(messages=[{"role": "user", "content": "..."}])
5. Colang 2.0. Newer NeMo Guardrails releases introduced Colang 2.0 with a
different grammar (`flow main`, `user said "..."`, event-based flow
composition) aimed at more expressive multi-flow orchestration. Most
existing tutorials and this lesson use the classic 1.0 syntax shown
above because it's still the most widely deployed version — check the
NeMo Guardrails docs for which version a given project targets before
copying syntax verbatim.
6. What a topical rail is NOT a substitute for: it restricts *what topics
the conversation can be about* — it is not a data access control layer
(that's retrieval scope filtering, lesson 01) and it is not a tool
authorization layer (that's lesson 04). A topical rail that blocks
"cooking" questions does nothing to stop an authorized-looking tool call
from executing with attacker-controlled arguments.
"""
if __name__ == "__main__":
print("COLANG-STYLE TOPICAL RAIL DEMO")
run_demo()
print("\n" + "=" * 70)
print(COLANG_MAPPING_NOTES)
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy