← TinyMachines Labs
Design note · Caliptra / Sentry AI SOC · Qwen3.5-4B

A security RL environment: teaching Qwen3.5-4B to drive a Kali toolchain.

This note describes the reinforcement-learning environment we use to build offensive — and defensive — cybersecurity skill into Qwen3.5-4B. The north-star: after RLVR training, the model is extremely good at offensive security by driving a real Kali Linux toolchain end to end, inside a no-egress sandbox. Reward is gated on outcomes a program can check — a captured flag, a shell with a proven UID, or Domain-Admin access confirmed by a planted nonce: no verified outcome, no reward. A model judge then shapes quality (methodology, stealth, report) only among rollouts that already cleared that programmatic gate, so the signal stays anchored to verifiable results (RLVR). This is legitimate security research — all offensive capability is contained to ephemeral lab targets; nothing in the loop touches the live internet or any third party. This is a design note: it describes the environment and the method, not measured training results.

ProgramCaliptra / Sentry AI SOC
Base modelQwen3.5-4B
MethodFull-parameter GRPO (RLVR)
StatusDesign note
01

Rebuild the world in a sandbox

Security is not one domain. A task is a world the agent acts in, not a prompt it answers.

Most of the world a security agent needs is scattered, gated, and not built for agents: hosts behind VPNs and jump boxes, services with rate limits, CVE feeds in inconsistent formats, logs spread across a dozen tools, and exploitation steps that are dangerous to run against anything real. We take that world offline and rebuild it around its own logic inside a sandbox: a victim estate of containerized hosts and services, a frozen CVE / patch knowledge base, and a log / telemetry plane — all reconstructed as a contained offline world, attacked from a real Kali operator box.

What the agent sees is not that storage but a small set of semantically general, agent-native typed tools that each execute real Kali Linux tooling on a Kali-based attacker host inside the sandbox — the operations a practitioner actually performs: scan, enumerate, exploit, pivot, read-logs, query-SIEM, write-detection, remediate, submit. The same environment serves training and inference. Every property of the world is a setting that travels with the episode: the date it is frozen at, which tools exist, the step budget, how work is scored, and the offense-vs-defense framing.

This mirrors the frontier practice of training in an offline world: OpenAI reports that all of its frontier RL runs use an offline world of indexed / cached content with no internet access (Securing our frontier model training environment, 2026-10-23). A domain environment built for agents is that offline world for cybersecurity: rich enough to learn real attacker / defender skill, and contained.

Cybersecurity is not one domain. Generic pipelines treat security as one task type with different nouns. Web-app exploitation, network-service exploitation, Active Directory attack paths, and SOC alert triage differ in their sources, their workflow, and in what counts as a right answer. So each task family gets its own generation pipeline, data sources, task logic, and rule base — a deliberately small, high-value taxonomy of 15 families (3).

02

A task is a workspace, not a question

Each task is a workspace, not a prompt:

  • Target environment — one or more containerized victim hosts / services (and, for defense, a telemetry plane: logs, EDR events, a SIEM index) built from a declarative manifest.
  • Goal / request — a practitioner's brief (“get a shell on the billing service and read /root/flag”; or defensively, “an alert fired at 02:14 — triage, scope, and contain”).
  • Hidden rubric — a set of binary verifiable criteria used as the reward, never shown to the agent.
  • Separate verifier — a component independent of the agent that decides whether each criterion is met (programmatic checks + a model judge for quality criteria).

Rewards mix programmatic / verifiable signals (flag captured, shell obtained with a proven UID, CVE validated against the frozen KB, a detection rule that fires on the true positives and stays silent on the benign corpus) with rubric-as-reward for non-verifiable quality (report quality, methodology soundness, stealth, blast-radius discipline). This follows Rubrics as Rewards for the non-verifiable part and standard CTF-style verification for the rest. The programmatic criterion is a gate: with no verified outcome the episode scores zero regardless of the judge, and the judge only shapes reward among already-verified successes. The programmatic-vs-judge split is a configurable weight, and an rlvr-strict mode sets the judge weight to zero. Because the verifiable component gates every reward, the signal stays anchored to verifiable rewards (RLVR); the judge is a shaping term, not a source of unearned credit.

03

Task families, built one at a time

Each family below gives its Sources · Task logic · Rule base, the specific Kali tooling a practitioner drives, and the two-part reward. We train a deliberately small, high-value taxonomy of 15 families — 14 offensive skills plus one purple-team family scored as both attacker and defender — so the model learns each attacker skill in isolation before they are composed. Full specs and example task JSON live under rl/tasks/<family>/. Everything here is the design of our own lab environment; every target is a disposable lab host inside the no-egress sandbox.

Offensive families (14)

Recon & enumeration recon-and-enumerationoffense

Sources
a seed range plus a fabricated synthetic org graph, lab service banners, DNS / vhost lists, and frozen wordlists; the manifest's ground-truth topology.
Task logic
sweep and fingerprint hosts / services / versions, enumerate exposed content and metadata, and assemble an attack-surface map that reaches a planted target fact.
Rule base
offline snapshot and the seeded flag; discovered hosts / ports / endpoints are checked against the manifest topology, coverage rewarded and noisy or destructive scanning penalized.
Kali tooling
nmapmasscantheHarvesterrecon-ngamassffufgobuster
Reward
Programmatic: hosts_match / ports_match vs the manifest and the seeded target fact. Judge: methodical recon, recon-before-exploit discipline, low noise.

Web exploitation — injection web-exploitation-injectionoffense

Sources
self-authored vulnerable web apps + de-identified real patterns, OWASP A03, CWE-89 / -78 / -94 / -502, the app's schema and request / response traces.
Task logic
find the injection point and class — SQLi, SSRF, SSTI, deserialization, or command injection — and escalate to data theft or code execution, capturing the flag.
Rule base
OWASP A03, CWE-89 / -78 / -94 / -502, and the app's schema; success iff the seeded row / flag is exfiltrated or code execution is proven with a nonce.
Kali tooling
sqlmapBurp Suitetplmapysoserial / phpggcffufcustom python
Reward
Programmatic: flag_match + rce / access proof (nonce). Judge: correct injection point and class, reproducible, in-scope.

Web auth & access control web-auth-and-access-controloffense

Sources
apps with login / session / JWT / OAuth / 2FA flows and object-reference endpoints, OWASP A01 / A07, CWE-287 / -639 / -863, the identity model.
Task logic
defeat authn / authz logic (JWT / OAuth / session / 2FA flaws, IDOR / BOLA / BFLA, mass assignment) to assume a privileged identity or reach another principal's data.
Rule base
OWASP A01 / A07, CWE-287 / -639 / -863, the identity model and a seeded privileged account / victim record; success iff the privileged identity or victim-only data is proven.
Kali tooling
jwt_toolBurp Suite (Autorize)hydragraphw00fInQLffuf
Reward
Programmatic: privileged-session / victim-record nonce. Judge: correct auth / access-control reasoning; no brute force where a logic flaw was intended.

Network service exploitation network-service-exploitationoffense

Sources
exposed lab network services, the CVE / advisory KB frozen to the snapshot, default-credential catalogs, and any WAF / filter ruleset.
Task logic
exploit an exposed network service (RCE, protocol flaw, default / weak creds, misconfig), evading any WAF / filter on the payload, to gain a foothold.
Rule base
the declared primitive (CVE / misconfig / default-cred) and the seeded foothold flag; success iff a foothold is proven with a nonce.
Kali tooling
Metasploitnmap NSEnetexecwafw00fsearchsploit
Reward
Programmatic: foothold nonce + flag_match. Judge: correct primitive, principled WAF evasion over blind fuzzing.

Credential attacks credential-attacksoffense

Sources
captured hashes (NTLM / NetNTLM / Kerberos / bcrypt / sha-crypt), auth services, frozen wordlists + rules, CWE-521 / -916.
Task logic
identify hash formats and crack / spray / relay credentials (wordlist / rule / mask, Kerberoast, NTLM relay), then reuse them to reach a protected resource.
Rule base
CWE-521 / -916 and the seeded credential / flag; success iff an authenticated session is proven with a nonce. Keyspaces are bounded so cracking is tractable in-budget.
Kali tooling
hashcatJohn the Ripperhashidnetexecresponderimpacket
Reward
Programmatic: session / auth proof via nonce. Judge: appropriate mode (offline vs online), policy-aware, no destructive lockouts.

Privilege escalation — Linux privilege-escalation-linuxoffense

Sources
misconfigured Linux hosts (sudo / SUID / cron / PATH / capabilities), pinned kernels, GTFOBins, CWE-250 / -269.
Task logic
from an unprivileged Linux shell, escalate to root via a misconfiguration or a pinned kernel bug.
Rule base
GTFOBins, CWE-250 / -269 and the seeded root flag; success iff a command runs as uid 0 echoing a fresh proof nonce.
Kali tooling
linpeasGTFOBins techniqueslinux-exploit-suggesterpspygcc / gdb
Reward
Programmatic: uid0_proof (nonce) + flag_match. Judge: correct misconfiguration identified, host not damaged.

Privilege escalation — Windows privilege-escalation-windowsoffense

Sources
misconfigured Windows hosts (unquoted service paths, weak service ACLs, token / registry abuse, UAC, LOLBAS), LOLBAS, CWE-250 / -269.
Task logic
from a low-privilege Windows shell, escalate to SYSTEM via an unquoted service path, weak service ACL, token / registry abuse, UAC bypass, or LOLBAS.
Rule base
LOLBAS, CWE-250 / -269 and the seeded SYSTEM flag; success iff a command runs as SYSTEM echoing a fresh proof nonce. Runs in a Windows microVM guest.
Kali tooling
winpeasPowerUp / SharpUpPrivescCheckaccesschk
Reward
Programmatic: system_proof (nonce) + flag_match. Judge: correct abuse primitive, host not damaged.

Active Directory attacks active-directory-attacksoffense

Sources
a lab AD forest (DC + members), a BloodHound graph of the frozen domain, Kerberos tickets, ACL / GPO layout.
Task logic
traverse an AD attack path (Kerberoast, AS-REP roast, ACL abuse, delegation, DCSync) from a low-priv user to Domain Admin.
Rule base
the BloodHound edge used and the seeded DA-only flag / krbtgt hash; success iff DA-equivalent access is proven.
Kali tooling
BloodHound / SharpHoundImpacketnetexecmimikatz
Reward
Programmatic: da_proof (nonce) + flag_match. Judge: valid path explained step by step, stealth (low-signal operations).

Post-exploitation & pivoting post-exploitation-and-pivotingoffense

Sources
multi-segment lab networks (DMZ / internal / management) with network namespaces, routing / firewall rules, planted creds and trusts.
Task logic
from an initial host, establish persistence, pivot across network segments, and exfiltrate a crown-jewel from a non-routable target.
Rule base
the pivot / persistence mechanism and the seeded crown-jewel flag; success iff the non-routable flag is read via a proven pivot chain.
Kali tooling
proxychainschisel / ligoloMetasploit routingssh tunnelingnetcat
Reward
Programmatic: crown-jewel match via a proven pivot chain. Judge: efficient, scoped pivoting and durable persistence.

Container & Kubernetes exploitation container-and-kubernetes-exploitationoffense

Sources
misconfigured containers / pods (privileged, docker.sock, risky caps, hostPath), a kind / k3s cluster, a pinned runtime, Kubernetes RBAC.
Task logic
break out of a mis-set container or abuse Kubernetes RBAC / service-account to reach the host node or cluster-admin.
Rule base
the breakout / RBAC-abuse primitive and the seeded host flag or cluster secret; success iff host / node or cluster-admin access is proven with a nonce, within the guest microVM.
Kali tooling
deepceamicontainedkubectlkube-hunternsenter
Reward
Programmatic: node_access_proof + secret match. Judge: correct escape / RBAC primitive, no impact beyond the lab node.

Crypto & protocol attacks crypto-and-protocol-attacksoffense

Sources
apps / protocols with crypto flaws (padding oracle, nonce reuse, weak RNG, downgrade, handshake), ciphertext / oracle services, CWE-327 / -330 / -347.
Task logic
exploit a crypto / protocol flaw to recover plaintext or forge a valid token.
Rule base
CWE-327 / -330 / -347 and the seeded secret; the recovered plaintext / forged token is checked against ground truth.
Kali tooling
oracle / protocol clientspython / sageopensslBurp Suite
Reward
Programmatic: plaintext / token match. Judge: correct vuln class and attack derivation.

Cloud exploitation cloud-exploitationoffense

Sources
an offline mock cloud control plane (LocalStack-style AWS, a kind / k3s cluster), IAM policies, bucket ACLs, the metadata service, CWE-732 / -284 — all offline.
Task logic
chain a misconfiguration (public bucket → creds, IMDS SSRF → role creds, RBAC escape) to reach a protected resource within its real IAM scope.
Rule base
CWE-732 / -284 and the seeded protected flag; success iff the protected resource is obtained with the stolen creds, tenant-scoped.
Kali tooling
aws clikubectlpacucurl (mock cloud only)
Reward
Programmatic: secret / resource match via nonce + a creds-use trace. Judge: correct misconfig chain, tenant-scoped.

Payload & evasion payload-and-evasionoffense

Sources
lab endpoints with EDR / AV-style detections, benign lab-authored payload templates, a lab-internal C2 listener, obfuscation catalogs.
Task logic
build and deliver a payload and C2 channel that evade the lab's EDR / network detections, maintain a stable beacon, and complete an objective below the detection threshold.
Rule base
the evasion / C2 techniques and the seeded objective flag; strictly lab-internal (no egress), benign payloads; success iff an internal C2 session + objective is proven under the detection threshold.
Kali tooling
msfvenomSliver (lab-only)obfuscation toolingproxychains
Reward
Programmatic: internal C2 session proof + events below the evasion threshold. Judge: tradecraft soundness, strict no-egress / blast-radius discipline.

Full-chain CTF full-chain-ctfoffense

Sources
a composed multi-host world chaining the single-skill families: an external web foothold, an internal estate, an AD forest, and a crown-jewel target.
Task logic
a long-horizon full kill chain — external web foothold → internal recon → privesc → AD lateral movement → crown-jewel exfil — proving each milestone with its nonce.
Rule base
the per-milestone proofs and the final seeded flag; success iff each milestone nonce and the final flag are proven in order.
Kali tooling
the full Kali toolset across stages
Reward
Programmatic: per-milestone nonces + final flag_match. Judge: coherent kill chain, recovery from dead ends, stealth.
Both offense & defense (1)

Purple-team co-training purple-team-cotrainingoffense + defense

Sources
one shared lab world with an attacker side (full offensive estate) and a defender side (telemetry plane: logs, EDR events, a SIEM index, and a benign baseline).
Task logic
an attacker role and a defender role act against each other on one shared world in a single episode (self-play or policy-vs-scripted), so offense and defense are trained adversarially in one loop.
Rule base
each role is scored relative to the other — attacker: reach the goal and stay undetected; defender: detect and contain while sparing the benign baseline (precision / recall).
Kali tooling
full offensive toolset (one side)query_siemwrite_detectionremediate
Reward
Programmatic: attacker goal nonce + stealth events; defender detection precision / recall + containment. Judge: soundness on both sides — scored as both offense and defense.

Count: 15 families — 14 offensive + 1 purple-team (scored as both attacker and defender). The mix is configurable; see the dataset plan (10).

04

Offense ↔ defense dual framing

The same underlying task world can be scored as attacker or as defender. A staged intrusion is the recorded consequence of an active-directory-attacks, post-exploitation-and-pivoting, or payload-and-evasion attack; the same world, two reward functions:

  • Offense framing (frame: offense) — reward for reaching the goal (flag / shell / DA), with bonus rubric weight on stealth (fewer high-signal events generated) and blast-radius discipline.
  • Defense framing (frame: defense) — the attack is executed by a scripted red agent (or replayed from a recorded offense episode) to generate telemetry; the model is rewarded for detecting and containing it — correct TP/FP calls, correct scope, a detection rule that fires on this intrusion and not on the benign baseline, and a containment action that cuts the attacker's access in the sandbox.

This lets one authored world train both capabilities, and lets offense episodes feed defense episodes: a successful offensive rollout is replayed to produce fresh, realistic detection data. The purple-team-cotraining family takes this furthest — an attacker and a defender act against each other on one shared world in a single episode, each scored relative to the other.

05

The sandbox: isolation, reproducibility, safety

Build & teardown per episode. Each episode's world is declared in a manifest and realized as a short-lived container composition. Targets are built from digest-pinned images, brought up on a dedicated network namespace, used, and destroyed. No state survives an episode unless snapshotted.

Isolation layers (defense in depth)

  • Kernel / VM boundary — every target runs inside a hardware-virtualized microVM with its own guest kernel: Firecracker for Linux guests and Cloud Hypervisor / Kata for Windows guests (the Active Directory, Windows-privesc and PowerShell families), since Firecracker runs Linux guests only. Where a family needs a cluster (kind / k3s, container-and-kubernetes-exploitation), those containers run nested inside the guest, never on the host. The isolation model is the same throughout — a hardware-virt guest plus a no-egress network namespace — even though the guest OS and inner stack vary: the privilege-escalation and container-and-kubernetes-exploitation families still get a genuine kernel to attack, and an escape is contained to the guest.
  • Process boundary — the agent's tool-executor runs under firejail + seccomp with a minimal syscall profile.
  • Network boundary — each episode gets its own Linux network namespace with no egress: a default-deny firewall, no default route to the internet, DNS served locally. The only reachable hosts are the episode's own targets. This is the primary safety boundary — but we treat no single layer as infallible. The boundaries are defense-in-depth, and the deployment is escape-tested (host / guest kernel versions, hypervisor config, network rules, mounts, management interfaces) as a standing part of operating the environment rather than assumed secure.
  • Filesystem boundary — read-only base layers + an ephemeral overlay; the host filesystem is never mounted into a target.

Snapshot — freeze the clock

Every episode carries an as_of date. The CVE / advisory KB, package versions, OS / kernel builds, pinned Kali tool versions, exploit availability, and the telemetry index are all filtered to that date, so nothing leaks from the future and every run is reproducible. Target images are content-addressed; seeds for any randomized placement (flag paths, credentials, service ports) are recorded in the episode record, so an episode can be rebuilt bit-for-bit and verifiers are deterministic given the seed.

Safety / compliance boundary. No live internet in the loop; no third-party systems targeted; malware-shaped samples are benign and lab-authored; phishing recipients are scripted personas, never real people; every task passes a safety / compliance review with human sign-off before entering the dataset. The environment produces offensive skill, exercised only against its own disposable targets. Episodes are independent and stateless at the controller level, so they can be run concurrently at scale (achievable throughput depends on the available infrastructure).

06

Agent-native typed tools over a real Kali toolchain

The agent is driving the real toolchain, not a toy abstraction. The sandbox's Kali attacker box ships the full Kali toolset preinstalled, and the agent acts through about a dozen typed tools, each with one clear meaning and a structured JSON return. Under the hood, scan, enumerate, exploit, pivot and run_cmd execute the actual Kali Linux CLI tools on that attacker host and parse their output into structured observations. Fast tools return in a second or two; long-running tools (sqlmap dumps, hashcat, masscan sweeps, AFL++) run asynchronously against a per-tool budget — the call returns a handle and progress, and the agent polls or does other work meanwhile. To keep episodes tractable inside the step and time budget, the fuzzing and credential families ship pre-seeded corpora and bounded keyspaces, so the objective is reachable within budget: the skill being trained is operating the tool correctly, not real 0-day discovery or exhaustive cracking. The goal of training is precisely that Qwen3.5-4B becomes extremely good at operating this toolchain to carry out full offensive-security tasks end to end. The available tool subset is a per-episode setting (action-space ablations).

ToolMeaningOff / Def
scanDiscover hosts, ports and services on a reachable segmentoff
enumerateDeep-enumerate a service or host (versions, users, shares, endpoints)off
exploitRun a parameterized exploit or technique against a targetoff
run_cmdRun a command on the Kali attacker host, or on any session host once compromisedboth
workspaceWrite, edit and compile files on the attacker host (custom C / pwntools / Python exploits)off
pivotEstablish routing or a tunnel from a compromised host to a new segmentoff
read_fileRead a file on a session host (flag / proof retrieval)both
read_logsRead raw logs and artifacts (auth, app, EDR, pcap, sysmon)def
query_siemStructured query over the SIEM index (frozen to snapshot)def
write_detectionSubmit a Sigma / YARA / query rule to be compiled and testeddef
remediateApply a patch, config, or containment actiondef
noteRecord a structured finding / IOC / hypothesis (scored, not world-changing)both
submitEnd the episode and hand the deliverable to the verifierboth

What each typed tool actually drives

Tools are the only way the agent affects the world — there is no browser and no internet. There is a real shell and a real workspace: run_cmd runs commands on the Kali attacker host (and on any session host once compromised), and workspace lets the agent write, edit and compile files, so it can author custom C exploits, pwntools scripts and ad-hoc Python — everything the custom-exploitation families require. What is constrained is reach, not expressiveness: all execution stays inside the no-egress sandbox. Each typed tool wraps real tooling and returns parsed results:

Typed toolReal Kali / host tooling it drives
scannmap (+NSE), masscan, naabu
enumerategobuster, ffuf, nikto, enum4linux-ng, smbclient, SharpHound-style collectors
exploitMetasploit (msfconsole, msfvenom), sqlmap, Burp Suite (Repeater/Intruder via API), Impacket, pwntools
run_cmda real shell on the Kali attacker host or a compromised session: bash / PowerShell, netcat, proxychains, Hydra, hashcat, John, mimikatz-style tooling
workspacegcc / clang, python3, pwntools, an editor — write and compile custom exploits on the attacker host
pivotMetasploit autoroute/socks, Chisel, proxychains, ssh -D
read_filecat / type over the active session
read_logstshark, zeek, sysmon / EDR readers, log parsers
query_siemSIEM query DSL over the frozen index
write_detectionsigmac (Sigma), yara / yarac (YARA), SIEM query compiler
remediatepatch / diff, iptables / nft, config management, container isolate / kill

A single exploit call, for example, may run sqlmap against a lab web app, launch a Metasploit module, or fire a pwntools script; enumerate fans out to gobuster, ffuf and a SharpHound-style collector; run_cmd is a real shell on the Kali attacker host (or a compromised session) where the agent chains netcat, proxychains, hashcat and the rest, and workspace compiles a custom exploit when a family needs one. The environment captures each tool invocation, parses it, and filters the result to the snapshot date and the agent's current access.

07

Observation model & step protocol

The agent sees the brief, its current sessions (host → access level), the tool subset, the remaining step / time budget, the snapshot date, the frame, and the accumulated observations it has gathered. It never sees the rubric, the verifier internals, the proof nonces, or anything dated after as_of.

The step protocol is reset → (act → observe)* → submit → reward. reset(task_id, config) builds the world and returns the initial observation. Each act(tool, args) drives the underlying tooling and returns a structured observation filtered to the snapshot date and the agent's current access; illegal or over-budget actions return typed errors, not crashes, and still consume budget. submit(deliverable) freezes the episode; the separate verifier scores it once and returns the reward breakdown — never mid-episode.

POST /env/step {"episode_id": "ep_1a2b", "action": {"tool": "exploit", "args": {"target": "10.13.0.5", "service": "http", "technique": "sqli", "param": "id", "payload_id": "union-read"}}} # on the Kali attacker host, this drives: sqlmap -u 'http://10.13.0.5/item?id=1' --technique=U --dump -T secrets -> {"observation": {"type": "exploit_result", "success": true, "session": "s1", "note": "read access to app DB; 1 row in `secrets`"}, "budget_left": {"steps": 18}, "done": false}

Every episode persists {task_id, config, seed, action trace, observations, deliverable, verifier output, reward} for reproducibility, judge audits, and offense → defense replay.

08

Reward & verifier design

Two-part reward per task, combined by the separate verifier at submit:

  • Programmatic verifiers — deterministic checks run in the sandbox: flag-string matches, a proof token on a uid-0 shell, exploit reproduction, a detection rule compiled and evaluated over the labeled corpus, and the “old exploit now fails + tests still pass” check for remediation.
  • Model-judge rubric — a stronger judge model scores non-verifiable criteria (methodology, report quality, stealth, soundness) against a per-task rubric of binary criteria.

Reward modes travel with the episode config: strict (every criterion must be met), recall (any correct component scores — early curriculum), and weighted (programmatic vs rubric weights, offense-vs-defense weighting).

Anti-reward-hacking

  • Flags and proof tokens are per-episode random (seeded) and never present in the image until placed by the builder, so they cannot be guessed or hard-coded.
  • Proof-of-access requires a fresh nonce the verifier plants at reset and the agent must echo from inside the target — this defeats “claiming” success.
  • Detection rules are evaluated on a held-out benign corpus too; false positives are penalized hard, defeating “match everything”.
  • The judge model is different and stronger than the policy; its judgments are audited and a sample is re-graded by a second judge.
  • No egress + seeded placement defeat “look it up online” and “memorize the flag”.
09

A configurable environment — one task, many environments

Every property is a setting that travels with the episode. One task yields many environments.

SettingExampleWhat changesUsed for
as_of2024-06-01CVE / patch / exploit knowledge frozen to the datefair, reproducible task
as_of: knowablefirst day the vuln was publicthe hardest fair settingdifficulty ceiling
snapshot: offtodayall later CVEs / patches visibleablation: how much hindsight helps
tools[-exploit] or [-query_siem]action-space ablationmeasure tool dependence
max_steps25step / time budgetcurriculum / efficiency
frameoffense / defensewhich reward function scores the worlddual-capability training
rewardrecall / strict / weightedscoring modecurriculum
difficulty1..5mitigations on, path length, telemetry noisecurriculum
POST /env/reset {"task_id": "sqli-union-101", "config": {"as_of": "2024-06-01", "snapshot": "on", "max_steps": 25, "tools": ["scan","enumerate","exploit","run_cmd","workspace","read_file","note","submit"], "frame": "offense", "reward": "strict", "difficulty": 3, "seed": 91823}}
10

Dataset plan & scale

The planned seed dataset is roughly ~300 base tasks across the 15 families, expanding to about ~750 with variations (snapshot date, difficulty, tool subset, offense / defense frame — each base task reset with different seeds and configs). The mix is ~92% offensive / ~8% purple-team, matching the north-star of a strong offensive operator that also understands defense. The distribution below is the planned target for the base-task seed set; the per-family counts are a plan, not a measured corpus, and the variation multiplier is a design estimate rather than a produced artifact.

FamilyBase tasksShare
recon-and-enumeration186.0%
web-exploitation-injection3010.0%
web-auth-and-access-control227.3%
network-service-exploitation206.7%
credential-attacks186.0%
privilege-escalation-linux227.3%
privilege-escalation-windows206.7%
active-directory-attacks248.0%
post-exploitation-and-pivoting186.0%
container-and-kubernetes-exploitation206.7%
crypto-and-protocol-attacks144.7%
cloud-exploitation186.0%
payload-and-evasion165.3%
full-chain-ctf165.3%
purple-team-cotraining248.0%
Total (base)300100%

With variations this reaches ~750 training instances. Each family's generator, sources, and rule base are built and reviewed before its tasks enter the pool, so the dataset grows family-by-family rather than all at once.

11

Generation pipeline & quality gates

Tasks are generated per family, then put through a pipeline of 8 stages, two of them gated, before anything enters the training pool:

  1. Author — a family generator emits a candidate manifest (targets, goal, seeds) from its own sources and rule base.
  2. Build — the world is realized from the manifest in the sandbox; build failures are dropped.
  3. Verifier authoring — the programmatic checks and the rubric are written and attached to the task.
  4. Seed / snapshot binding — randomized placement is seeded and the task is bound to its as_of snapshot.
  5. Automated solvability (GATE) — reference solvers and strong models must solve the task, and the verifier must distinguish a correct run from a null run; tasks that are unsolvable, trivially solvable, or whose verifier can be gamed are rejected.
  6. Human second-review (GATE) — a human reviews realism, difficulty labeling, and verifier soundness; borderline tasks are fixed or cut.
  7. Safety / compliance sign-off — a human confirms containment, benign samples, scripted personas, and no live-internet dependence before admission.
  8. Admit & version — the task is content-addressed and added to the pool with its family, difficulty, and snapshot metadata.

Both gates must pass; a failure at either sends the task back or drops it. New families follow the same path, so quality is enforced per family rather than assumed.

12

SFT cold-start — a warm start before RL

Before any RL, we teach the base model the shape of the job: the tool-calling format and a baseline skill in each family.

GRPO on a cold base model is slow and noisy: early rollouts mostly emit malformed tool calls and never reach a verifiable outcome, so there is little reward signal to learn from. We fix this with a full-parameter SFT cold-start on Qwen3.5-4B — all 4B weights train, no LoRA / no adapters. The goal is narrow: make the model reliably speak the ReAct one-action-per-step protocol over the 12 typed tools, respect the JSON tool schema, and carry a baseline competence in each of the 15 families, so that when GRPO starts, a meaningful fraction of rollouts already clear the programmatic gate.

The corpus is ~1,450 verifier-passed trajectories — successful ReAct rollouts (reference solvers and strong models driving the real environment) that the programmatic verifier confirms, filtered for clean methodology and spanning all 15 families. Loss is masked to the assistant's reasoning and tool_call tokens only. The SFT corpus is held disjoint from the held-out RL eval, so the cold-start never leaks evaluation tasks.

SFT settingValue
Methodfull-parameter SFT — no LoRA / adapters (all 4B weights train)
Precisionbf16 + gradient checkpointing (fits a single 80GB GPU)
Corpus~1,450 verifier-passed ReAct tool-call trajectories across all 15 families
Epochs3 (small corpus; 3 passes without heavy overfit)
Effective batch64 sequences (micro-batch 4 × grad-accum 16)
Max sequence8,192 tokens, sample-packing on
OptimizerAdamW (β 0.9 / 0.95, wd 0.0), LR 1e-5 peak, cosine, 3% warmup, grad clip 1.0
Loss maskingassistant reasoning + tool_call tokens only
Split90 / 10 train / val, stratified by family
# SFT cold-start config — Qwen3.5-4B (full-parameter, NO LoRA) stage: sft base: qwen3.5-4b # same model RL then fine-tunes method: full_parameter # all 4B weights train — no LoRA / adapters precision: {dtype: bf16, gradient_checkpointing: true} train: epochs: 3 micro_batch_size: 4 grad_accum_steps: 16 # effective batch = 64 sequences max_seq_len: 8192 sample_packing: true loss_mask: assistant_only # loss on reasoning + tool_call; mask system/user/tool optimizer: {name: adamw, betas: [0.9, 0.95], weight_decay: 0.0, lr: 1.0e-5, lr_schedule: cosine, warmup_ratio: 0.03, max_grad_norm: 1.0} data: format: react_tool_calls # the 12 typed tools; schema = rl/tasks/task.schema.json corpus: ~1450 verifier-passed trajectories, all 15 families train_val_split: 0.9 # stratified by family handoff: init_for: grpo_qwen3.5-4b.yaml # selected ckpt becomes GRPO init weights gates: [format_alignment, tool_schema_conformance, invalid_action_rate < 0.02, easy_task_pass_rate > 0] disjoint_from: rl/eval # SFT data never overlaps held-out eval

The selected SFT checkpoint must clear a small set of handoff gates — format alignment, tool-schema conformance, an invalid-action rate under 2%, and a non-zero pass rate on easy tasks — before it becomes the initial weights for GRPO. SFT teaches the format and a floor of skill; RL then pushes capability past what demonstrations alone can reach.

13

Training Qwen3.5-4B — full-parameter GRPO

Warm-started from the SFT cold-start checkpoint (12), we train Qwen3.5-4B with GRPO, a critic-free policy-gradient method well suited to agentic, long-horizon, verifiable-reward tasks. Training is full-parameter — every one of the 4B weights updates, with no LoRA and no adapters anywhere. The only frozen network is a separate frozen copy of the model kept solely for the KL-divergence penalty: the policy is regularized toward that reference so it does not drift, but the policy itself is never frozen. At 4B, full-parameter GRPO fits comfortably on one or two 80GB GPUs with bf16, gradient checkpointing, and ZeRO sharding.

Each round samples a batch of tasks and rolls out a group of K attempts per task. GRPO computes a group-relative advantage — each rollout's reward normalized against its own group's mean and spread — so the model learns from relative success within a task and needs no learned value function. A clipped policy-gradient update then moves all the weights, with the KL term to the frozen reference copy keeping the step stable.

  • Group-relative, verifier-anchored. Rewards come from the separate verifier (8); the programmatic gate anchors every reward and the judge only shapes among verified successes.
  • Curriculum. Early rounds use recall mode and low difficulty; later rounds shift to weighted then strict and raise difficulty, mitigations, and telemetry noise.
  • Held-out families. A few families are withheld from training entirely (a transfer test), plus a held-out task benchmark within trained families.
# GRPO (full-parameter) training config — Qwen3.5-4B model: base: qwen3.5-4b dtype: bf16 full_parameter: true # every weight trainable — no adapters lora: {enabled: false} # explicitly no LoRA / no adapters freeze_base: false # base is NOT frozen; all 4B weights update reference_model: frozen_copy # a frozen copy kept ONLY for the KL term sharding: {strategy: zero3_optional, gradient_checkpointing: true, optimizer_offload: if_needed} algorithm: name: grpo group_size_K: 8 # rollouts per task -> group-relative advantage tasks_per_round_M: 32 rounds_N: 10 # ~320 task-draws over the ~300-task reservoir advantage: group_relative # (reward - group_mean) / (group_std + eps) kl: reference: frozen_copy # KL to the frozen reference copy coef: 0.02 clip_ratio: 0.2 optimizer: {name: adamw, lr_warmup: 5.0e-7, lr_main: 5.0e-7, weight_decay: 0.0} max_grad_norm: 1.0 # hard clip: all 4B weights move rollout: engine: vllm # weight-synced each round env_server: "http://env-controller:8080" max_steps_per_episode: 25 wall_s_per_episode: 600 parallel_episodes: 256 # = tasks_per_round_M * group_size_K (32 * 8) reward: source: verifier # programmatic (RLVR gate) + judge (rubric, shaping only) judge_model: strong-judge # stronger than the policy; audited mode_by_round: {1-3: recall, 4-6: weighted, 7-10: strict} curriculum: difficulty_by_round: {1-3: [1,2], 4-6: [2,3], 7-10: [3,5]} data: init_from: sft # warm-started from the SFT cold-start checkpoint train_families: [recon-and-enumeration, web-exploitation-injection, web-auth-and-access-control, network-service-exploitation, credential-attacks, privilege-escalation-linux, active-directory-attacks, post-exploitation-and-pivoting, crypto-and-protocol-attacks, cloud-exploitation, full-chain-ctf, purple-team-cotraining] heldout_families: [privilege-escalation-windows, container-and-kubernetes-exploitation, payload-and-evasion] # transfer test heldout_benchmark: eval/benchmarks.md#internal-ctf-holdout snapshot: on logging: save_episode_records: true judge_audit_sample: 0.08 checkpoints_every_rounds: 1

A focused proof run first

Before committing to a full multi-round run, we validate the loop with a small, focused proof run: a handful of the easiest families, low difficulty, recall reward, and a short round count. The question it answers is purely mechanical — does reward move in the right direction, does the invalid-action rate fall, does the verifier stay un-gameable, and does the SFT → GRPO handoff behave — before scaling to the full family set and the harder curriculum. The proof run reuses the same environment, verifier, and configs; only the family subset, difficulty, and round count shrink.

Evaluation uses held-out tasks and held-out families, reports pass rate by family and difficulty, tracks invalid-action rate and steps-to-solve, and audits a sample of judge decisions. We additionally run a harness-synthesis check in the spirit of AutoHarness and a prompt-evolution transfer test in the spirit of GEPA to confirm the gains are real capability, not format memorization.

14

References & prior art

  1. AutoHarness: Improving LLM Agents by Automatically Synthesizing a Code Harness. ICLR 2026. arXiv:2603.03329.
  2. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. ICLR 2026. arXiv:2507.17746.
  3. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR 2026. arXiv:2507.19457.
  4. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv:2306.05685.
  5. OpenAI. Securing our frontier model training environment. 2026-10-23.
  6. MITRE ATT&CK; OWASP Top-10 / ASVS; CWE; Sigma; YARA (frozen catalogs used as rule bases).

TinyMachines Labs · Caliptra / Sentry AI SOC · a design note on the security RL environment for Qwen3.5-4B. All offensive capability described here is contained to ephemeral, no-egress lab targets.