8:6 axes boxes and the outsider label in the system figure

Figures, data and code as they stand at submission, including the
bibliography record check that verifies every entry against its article
in ref/.

README.md

@@ -32,6 +32,12 @@ code/
   check_cov_*.py        ciphertext-only covariance-attack checks (referee M1)
   exp_real_sec.py       stage G: real BERT WordPiece token streams
   verify_math.py        closed-form checks V1-V11, PASS/FAIL and verify_math.csv
+  exp_learned.py        every KM (lrn.) artifact, one function per stage
+  run_learned_reg.py    runs those stages in order, sens before brute
+  merge_learned_rows.py folds the learned rows into the two table sources
+  report_learned.py     every learned number beside its structured one
+  diag_whygap.py        why a fixed key beats a learned one here
+  diag_jscc.py          where a learned mask would win instead
   replot_security.py    every result figure, from data/ to fig/
   make_tables.py        LaTeX rows of every result table, from data/
   feasibility_security.py   early CPU-sized study, kept for the record
@@ -60,10 +66,19 @@ python exp_users_csi.py      # load and channel-estimate sweeps
 python check_cov_attack.py   # ciphertext-only covariance attack
 python diag_maskdegen.py     # learned-key support degeneracy
 python check_family_enum.py  # ciphertext-only enumeration of the key family
+python run_learned_reg.py    # every KM (lrn.) artifact, regularized keys
+python merge_learned_rows.py # the learned rows of the two result tables
 python replot_security.py    # all figures from the CSVs
 python make_tables.py        # LaTeX rows of the result tables
 ```
 
+The learned keys are the regularized ones of Section V-C, trained under
+the two penalties rather than under the cross-entropy alone. Training on
+the cross-entropy alone drifts to disjoint sparse supports, which is an
+orthogonal slot allocation rather than a superposition; `diag_whygap.py`
+measures that drift and `exp_learned.learned_model` says why the
+regularized keys are the ones every figure draws.
+
 Seeds are fixed: training 1, evaluation 777, attacker key guess
 20260813, key recovery 4242, brute-force search 31, cross-scheme
 comparison 11, key refresh 5150. Re-running reproduces the released CSV
@@ -82,17 +97,17 @@ Logarithms in an entropy or an information rate are base two.
 
 | Artifact | Script | Data |
 |---|---|---|
-| Fig. 2 SER against SNR | `exp_full.stage_A` | `sec_snr.csv` |
+| Fig. 2 SER against SNR | `exp_full.stage_A`, `stage_N` | `sec_snr.csv`, `sec_snr_learned.csv` |
-| Fig. 3 key length | `exp_full.stage_B` | `sec_keylen.csv` |
+| Fig. 3 key length | `exp_full.stage_B`, `exp_learned.keylen`, `.keylen_perm` | `sec_keylen.csv`, `sec_keylen_learned.csv`, `sec_keylen_perm.csv` |
-| Fig. 4 jamming (4 schemes) | `exp_full.stage_C`, `stage_L` | `sec_jam_cmp.csv`, `sec_jam.csv` |
+| Fig. 4 jamming (5 curves) | `exp_full.stage_C`, `stage_L`, `exp_learned.jamming` | `sec_jam_cmp.csv`, `sec_jam.csv`, `sec_jam_learned.csv` |
-| Fig. 5 key sensitivity | `exp_full.stage_I` | `sec_sens_cmp.csv` |
+| Fig. 5 key sensitivity | `exp_full.stage_I`, `exp_learned.sens` | `sec_sens_cmp.csv`, `sec_sens_learned.csv` |
-| Fig. 6 brute-force search | `exp_full.stage_I`, `stage_F`, `stage_J` | `sec_brute_cmp.csv`, `sec_brute.csv` |
+| Fig. 6 brute-force search | `exp_full.stage_I`, `stage_F`, `stage_J`, `exp_learned.brute` | `sec_brute_cmp.csv`, `sec_brute.csv`, `sec_brute_learned.csv` |
-| Fig. 7 known-plaintext attack | `exp_kpa`, `exp_permkpa` | `kpa.csv`, `pkpa.csv` |
+| Fig. 7 known-plaintext attack | `exp_kpa`, `exp_permkpa`, `exp_learned.kpa` | `kpa.csv`, `pkpa.csv`, `kpa_learned.csv` |
-| Fig. 8 real token streams | `exp_real_sec` | `real_sec_ter.csv` |
+| Fig. 8 real token streams | `exp_real_sec`, `exp_learned.real` | `real_sec_ter.csv`, `real_sec_ter_learned.csv` |
-| Scheme comparison table | `exp_full.stage_E` | `sec_compare.csv` |
+| Scheme comparison table | `exp_full.stage_E`, `exp_learned.compare`, `merge_learned_rows` | `sec_compare.csv` |
-| Key family table | `exp_full.stage_D` | `sec_maskfam.csv`, `sec_regjam.csv` |
+| Key families (Sec. VI-G prose) | `exp_full.stage_D` | `sec_maskfam.csv`, `sec_regjam.csv` |
-| Headline recovery table | `exp_real_sec` | `real_sec_stats.json` |
+| Headline recovery (Sec. VI-H prose) | `exp_real_sec` | `real_sec_stats.json` |
-| Key refresh tables | `exp_refresh` | `refresh_summary.csv`, `refresh_kpa.csv` |
+| Key refresh table | `exp_refresh`, `exp_learned.refresh`, `merge_learned_rows` | `refresh_summary.csv`, `refresh_kpa.csv` |
 | Information-theoretic leakage | `exp_infotheory` | `infotheory.csv` |
 | Semantic similarity | `exp_semantic` | `semantic.csv` |
 | Load and channel-estimate sweeps | `exp_users_csi` | `users.csv`, `csi.csv` |
@@ -103,6 +118,8 @@ Run one stage on its own with `python code/exp_full.py stage_B`, or the whole ch
 | Covariance attack (Sec. IV) | `check_cov_attack` | `cov_attack.csv` |
 | Learned-key degeneracy (Sec. VI-F) | `diag_maskdegen` | `maskdegen.csv` |
 | Closed-form and symbolic checks | `verify_math` | `verify_math.csv` |
+| Why a fixed key wins here (not in the paper) | `diag_whygap` | `whygap.csv` |
+| Where a learned mask would win (not in the paper) | `diag_jscc` | `jscc.csv` |
 
 ## Security scope
 

code/check_bib_records.py

@@ -0,0 +1,126 @@
+# -*- coding: utf-8 -*-
+"""Verify every bibliography entry against the article's own first page.
+
+The standard asks for one verdict per entry against the publisher record
+(12.23), and for a missing issue number to be completed "when the
+publisher record shows one" (8.6). Both are answerable from ref/,
+because the stored PDF is the published article and its running head
+carries the volume, the issue when the journal has one, the year and the
+first page.
+
+This exists because an audit read eight entries with no `number` field
+as incomplete. They are not: IEEE now publishes TIFS, JSAC, TWC, TCOM
+and TSP with continuous volume pagination, and those articles' running
+heads read "VOL. n, YEAR" with no issue at all. Adding a number there
+would invent data. The check makes the distinction mechanical so the
+finding is not raised again.
+
+Entries with no stored PDF are reported as unverifiable rather than
+passed, so the count of what remains unchecked is visible.
+
+Run: python code/check_bib_records.py
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+BIB = ROOT / "references.bib"
+REF = ROOT / "ref"
+
+# a running head, in the several shapes the venues use
+HEADS = [
+    # IEEE journal with an issue: VOL. 22, NO. 12,  and VOL. IT-24, NO. 3,
+    re.compile(r"VOL\.\s*(?:[A-Z]{2}-)?(\d+)\s*,\s*NO\.\s*(\d+)", re.I),
+    # IEEE journal on continuous volume pagination: VOL. 20, 2025
+    re.compile(r"VOL\.\s*(\d+)\s*,\s*(?:19|20)\d{2}", re.I),
+    # a journal that prints volume(issue): 24(6):801-812
+    re.compile(r"\b(\d+)\((\d+)\)\s*:"),
+]
+def entries():
+    txt = BIB.read_text(encoding="utf-8")
+    for m in re.finditer(r"@(\w+)\s*\{([^,]+),(.*?)\n\}", txt, re.S):
+        yield m.group(2).strip(), m.group(1).lower(), m.group(3)
+
+
+def field(body, name):
+    m = re.search(r"\b%s\s*=\s*\{([^}]*)\}" % name, body)
+    return m.group(1).strip() if m else None
+
+
+def head_of(pdf):
+    import fitz
+    d = fitz.open(pdf)
+    t = " ".join(d[0].get_text().split())
+    d.close()
+    return t
+
+
+def main() -> int:
+    checked = ok = 0
+    problems, unverifiable, noissue = [], [], []
+    for key, kind, body in entries():
+        if key == "BSTcontrol":
+            continue
+        pdf = REF / (key + ".pdf")
+        if not pdf.exists():
+            unverifiable.append(key)
+            continue
+        checked += 1
+        head = head_of(pdf)
+        vol, num = field(body, "volume"), field(body, "number")
+        why = []
+
+        printed_vol = printed_num = None
+        for rx in HEADS:
+            m = rx.search(head)
+            if m:
+                printed_vol = m.group(1)
+                printed_num = m.group(2) if m.lastindex and m.lastindex > 1 \
+                    else None
+                break
+        if printed_vol and vol and printed_vol != vol:
+            why.append("volume %s printed, %s in bib" % (printed_vol, vol))
+        if printed_num and num and printed_num != num.split("--")[0]:
+            why.append("issue %s printed, %s in bib" % (printed_num, num))
+        if printed_num and not num:
+            why.append("issue %s printed, none in bib" % printed_num)
+        # a volume with no printed issue is the continuous-pagination case
+        # 8.6 exempts, and a scanned cover page that omits an issue the
+        # entry carries is not evidence against the entry
+        if not printed_num:
+            noissue.append(key)
+
+        pages = field(body, "pages")
+        if pages:
+            first = pages.split("--")[0].strip()
+            if first and first not in head.replace(",", ""):
+                why.append("first page %s not on the printed page" % first)
+
+        year = field(body, "year")
+        if year and year not in head:
+            why.append("year %s not on the printed page" % year)
+
+        if why:
+            problems.append((key, why))
+        else:
+            ok += 1
+
+    for key, why in problems:
+        print("  MISMATCH  %-26s %s" % (key, "; ".join(why)))
+    print()
+    print("verified against ref/: %d of %d entries, %d clean, %d mismatched"
+          % (checked, checked + len(unverifiable), ok, len(problems)))
+    if noissue:
+        print("printed record carries no issue number (%d): %s"
+              % (len(noissue), ", ".join(sorted(noissue))))
+    if unverifiable:
+        print("no stored PDF, not verifiable here: %s"
+              % ", ".join(sorted(unverifiable)))
+    return 1 if problems else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

code/check_consistency.py

@@ -55,8 +55,11 @@ chk("24 and 35 in tex", "$24$ to\n$35$~percent" in tex or "$24$ to $35$~percent"
 ew = [float(x["eve_wrong"]) for x in sn]
 ch = float(sn[0]["chance"])
 dev = max(abs(x - ch) for x in ew)
-chk("outsider at chance to 3.5e-4", dev < 3.6e-4, "max deviation %.2e" % dev)
+_ewl = [float(x["eve_wrong"]) for x in rows("sec_snr_learned.csv")]
-chk("3.5e-4 in tex", "$3.5\\times10^{-4}$" in tex, "searched tex",
+dev = max(dev, max(abs(x - ch) for x in _ewl))
+chk("outsider at chance to 4e-4, both families", dev < 4.0e-4,
+    "max deviation %.2e" % dev)
+chk("4e-4 in tex", "$4\\times10^{-4}$" in tex, "searched tex",
     needs_tex=True)
 
 # the main configuration's legitimate rate, the reference every later
@@ -121,14 +124,14 @@ chk("perm KPA at N=6 near its own legitimate",
 # --- refresh ----------------------------------------------------------
 rs = {x["scheme"]: x for x in rows("refresh_summary.csv")}
 chk("refresh 364.6 bits",
-    round(float(rs["Invariant"]["entropy_bits"]), 1) == 364.6,
+    round(float(rs["Invariant, KM (str.)"]["entropy_bits"]), 1) == 364.6,
-    "%.3f" % float(rs["Invariant"]["entropy_bits"]))
+    "%.3f" % float(rs["Invariant, KM (str.)"]["entropy_bits"]))
 chk("fixed key 23.8 bits",
     round(float(rs["None (fixed key)"]["entropy_bits"]), 1) == 23.8,
     "%.4f" % float(rs["None (fixed key)"]["entropy_bits"]))
 chk("invariant refresh free",
-    abs(float(rs["Invariant"]["legit"]) - float(rs["None (fixed key)"]["legit"]))
+    abs(float(rs["Invariant, KM (str.)"]["legit"]) - float(rs["None (fixed key)"]["legit"]))
-    < 0.001, "%.4f vs %.4f" % (float(rs["Invariant"]["legit"]),
+    < 0.001, "%.4f vs %.4f" % (float(rs["Invariant, KM (str.)"]["legit"]),
                                float(rs["None (fixed key)"]["legit"])))
 
 # --- real tokens ------------------------------------------------------
@@ -161,9 +164,10 @@ chk("ratio spans 1.32 to 1.54", round(min(rt), 2) == 1.32
 
 # the three secrets named in the setup
 chk("secret sizes: per-user direction, perm 256, pad 16",
-    all(t in tex for t in ["length-$64$ key direction per user",
+    all(t in " ".join(tex.split())
-                           "one permutation of $256$",
+        for t in ["length-$64$ key direction per user",
-                           "$16$ pad bits per user"]),
+                  "one permutation of $256$",
+                  "$16$ pad bits per user"]),
     "searched tex", needs_tex=True)
 
 chk("no stale d=64 configuration in tex",
@@ -208,12 +212,17 @@ ks = [int(x) for x in md["learned"]["support99_per_key"].split("/")]
 chk("learned keys degenerate: 5 to 8 of 64 entries",
     min(ks) == 5 and max(ks) == 8 and int(md["learned"]["L"]) == 64,
     md["learned"]["support99_per_key"])
-chk("learned support overlap 0.10",
+# independent supports of size a and b out of L overlap by max(a,b)/L
-    round(float(md["learned"]["mean_overlap"]), 2) == 0.10,
+# on this normalization, so the measured value is the chance level and
-    md["learned"]["mean_overlap"])
+# evidences the concentration rather than any disjointness
-chk("degeneracy numbers in tex",
+_ch = sum(max(a, b) for a, b in __import__("itertools").combinations(ks, 2))
-    "$5$ to $8$ of the $64$ entries" in tex
+_ch /= (len(ks) * (len(ks) - 1) / 2) * int(md["learned"]["L"])
-    and "overlapping by $0.10$" in " ".join(tex.split()),
+chk("learned support overlap is at chance, not below it",
+    float(md["learned"]["mean_overlap"]) <= _ch + 0.02,
+    "measured %s against chance %.3f" % (md["learned"]["mean_overlap"], _ch))
+chk("concentration numbers in tex",
+    "$5$ to $8$ of the $64$ entries" in " ".join(tex.split())
+    and "a digit is decided over a tenth of its period" in " ".join(tex.split()),
     "searched tex", needs_tex=True)
 
 # --- why the permutation key is granted a shared permutation ---------
@@ -224,7 +233,7 @@ chk("per-user permutation legitimate rate",
     abs(pv["per_user"] - 0.129) < 1e-3, "%.5f" % pv["per_user"])
 if HAVE_TEX:
     chk("quoted permutation cost in tex",
-        "from $0.053$ to $0.129$" in " ".join(tex.split()),
+        "at $0.129$ against $0.053$" in " ".join(tex.split()),
         "searched tex", needs_tex=True)
 
 
@@ -299,11 +308,39 @@ chk("learned family tracks the structured one over the SNR range",
                                 for r in _sl),
                             max(float(r["legit"]) / _sn[float(r["snr_db"])]
                                 for r in _sl)))
-chk("learned 0.064 at 10 dB",
+_l10 = [float(r["legit"]) for r in _sl if float(r["snr_db"]) == 10.0][0]
-    abs([float(r["legit"]) for r in _sl
+chk("learned 0.061 at 10 dB", abs(_l10 - 0.061) < 5e-4, "%.5f" % _l10)
-         if float(r["snr_db"]) == 10.0][0] - 0.064) < 5e-4,
+# The learned curves must be the regularized keys, not the unpenalized
-    "%.5f" % [float(r["legit"]) for r in _sl
+# ones. Both are measured in sec_maskfam.csv and they differ by 0.003,
-              if float(r["snr_db"]) == 10.0][0])
+# which is larger than the spread of either, so matching the right row
+# pins which family every figure draws. Cross-entropy alone drifts to a
+# slot allocation whose key space is a support rather than a sphere, so
+# drawing it would not support the key-space claim.
+_fam = {r["family"]: float(r["legit_ser"]) for r in rows("sec_maskfam.csv")}
+chk("the plotted learned family is the regularized one",
+    abs(_l10 - _fam["learned_reg"]) < abs(_l10 - _fam["learned"])
+    and abs(_l10 - _fam["learned_reg"]) < 1.5e-3,
+    "plotted %.5f, reg %.5f, plain %.5f"
+    % (_l10, _fam["learned_reg"], _fam["learned"]))
+
+_bl = {int(r["K"]): float(r["ser_mask"])
+       for r in rows("sec_brute_learned.csv")}
+chk("learned key resists a million random guesses",
+    abs(_bl[1_000_000] - 0.71) < 5e-3, "%.4f" % _bl[1_000_000])
+_kl = {(float(r["snr_db"]), int(r["n_frames"])): float(r["eve_ser"])
+       for r in rows("kpa_learned.csv")}
+chk("learned key falls to known plaintext like the structured one",
+    _kl[(10.0, 4)] < 0.08 and _kl[(10.0, 1)] > 0.9,
+    "N=1 %.3f, N=4 %.4f" % (_kl[(10.0, 1)], _kl[(10.0, 4)]))
+
+_rs = {float(r["snr_db"]): float(r["ter_legit"])
+       for r in rows("real_sec_ter.csv")}
+_rl = {float(r["snr_db"]): float(r["ter_legit"])
+       for r in rows("real_sec_ter_learned.csv")}
+_rat = [_rl[k] / _rs[k] for k in _rs]
+chk("learned keeps its uniform-source distance on real text",
+    all(1.10 < v < 1.25 for v in _rat),
+    "ratio %.2f to %.2f" % (min(_rat), max(_rat)))
 
 # --- trends, which the value assertions above cannot see ---------------
 _snr = rows("sec_snr.csv")
@@ -377,7 +414,8 @@ for _k, _c in [("V8 cross-period remainder", 0.0005),
         and float(_vm[_k]["abs_err"]) <= _c,
         _vm[_k]["empirical"] if _k in _vm else "row missing")
 chk("format-matched OMA quoted as 0.055",
-    "$0.055$ at $10$~dB against the proposed" in tex, "Section VI-B",
+    "$0.055$ at $10$~dB" in tex and "the proposed $0.053$" in tex,
+    "Section VI-B",
     needs_tex=True)
 
 # --- tables against their generator -----------------------------------
@@ -390,7 +428,8 @@ if HAVE_TEX:
     buf = io.StringIO()
     with contextlib.redirect_stdout(buf):
         make_tables.compare_table()
-        make_tables.maskfam_table()
+        # the key-family table was folded into the Section VI-F prose
+        pass
         make_tables.refresh_tables()
     rows = [r.strip() for r in buf.getvalue().split("\n")
             if r.rstrip().endswith(r"\\")]

code/check_family_enum.py

@@ -28,7 +28,7 @@ from pathlib import Path
 
 import torch
 
-from exp_full import MAIN_D, base_keys, get_model, main_model
+from exp_full import MAIN_D, base_keys, get_model_reg, main_model
 from sse_lib import DEVICE, rayleigh_gain, snr_to_sigma2, write_csv
 
 DATA = Path(__file__).resolve().parents[1] / "data"
@@ -117,7 +117,9 @@ def run():
     torch.manual_seed(SEED)
     rows = []
     _sweep(main_model(), "structured", rows)          # keys frozen to Walsh
-    _sweep(get_model(P=4, vu=16, d=MAIN_D, U=4, iters=4000, seed=1),
+    # the regularized keys of Section V-C, which are the learned family
+    # every figure draws; the unpenalized ones are a slot allocation
+    _sweep(get_model_reg(P=4, vu=16, d=MAIN_D, U=4, iters=4000, seed=1),
            "learned", rows)                           # keys trained in R^L
     write_csv(DATA / "family_enum.csv",
               ["family", "keying", "snr_db", "n_frames",

code/diag_jscc.py

@@ -0,0 +1,181 @@
+# -*- coding: utf-8 -*-
+"""Does learning the mask buy anything, and where would it?
+
+Nothing in the manuscript rests on this; it answers a design question
+the paper does not raise. Output in data/jscc.csv.
+
+What the run found, at 10 dB over three training seeds. A: with one
+user the mask does not matter, a Walsh row and no mask at all landing
+at 0.0133 and 0.0132 against 0.0148 for a learned key, so the mask
+carries no part of the source-channel map and only separates users.
+B: at the six key lengths where truncated Walsh rows are not exactly
+orthogonal, learning wins once, at L=14 with kappa 0.048, and loses at
+L=16, 20, 22 and 24 where the rows ARE orthogonal. C: raising the load
+to U=20 at L=16 gives learning its second win, by 1e-5 on an error
+rate of 0.9998, which is no win at all because both families have
+already collapsed. Every other point is a tie inside the seed spread.
+
+So the room for a learned mask is real but narrow, and it is where
+exact orthogonality does not exist rather than where the load is high.
+A learned mask beating a fixed one wants a loss that is not digit
+cross-entropy, or a source that is not uniform, or a channel that is
+not a scalar the receiver divides out.
+
+A joint source-channel view says a learned mask should beat a fixed one,
+since the fixed one lies inside the search space. It does not here, and
+these experiments say why, and where the picture changes.
+
+A. What job does the mask actually do? Run one user. With a single user
+   there is nobody to separate from, so if the mask carried any part of
+   the source-channel map its choice would still matter. Compare a Walsh
+   row, no mask at all, and a learned key, each with the codebook
+   trained around it. Equal error rates mean the mask is not part of
+   that map: the codebook is, and the mask only separates users. This
+   also has a one-line proof. A unit-modulus key has m^2 = 1, so it
+   cancels from the signal self-term and from the noise projection
+   alike, and the score is unchanged.
+
+B. Where is the structured family no longer optimal? The construction
+   supplies exactly orthogonal unit-modulus rows only at the lengths
+   where truncation preserves orthogonality. At L = 6, 10, 14, 18, 20
+   and 22 the truncated rows correlate, so no exactly orthogonal
+   unit-modulus family is available and learning has room to find a
+   better packing. Every length is run at several seeds, because a
+   single training run is not evidence of a family being better.
+
+C. Overload. Beyond U = L - 1 no orthogonal set of non-constant rows
+   exists at all, so the structured family has to reuse rows and the
+   comparison is decided by whatever packing learning finds.
+
+Run on a GPU host:  python code/diag_jscc.py
+"""
+from __future__ import annotations
+
+import csv
+import statistics
+import sys
+from pathlib import Path
+
+import torch
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from exp_full import (MAIN_D, base_keys, get_model_reg,  # noqa: E402
+                      mean_abs_xcorr)
+from sse_lib import DATA, DEVICE, SSE, eval_ser_sse, set_seed  # noqa: E402
+import sse_lib as L  # noqa: E402
+
+SNR = 10.0
+SEEDS = [1, 2, 3]
+FRAMES = 400_000
+
+
+def train_with_key(W0, d, P, vu, U, iters=4000, seed=1, frozen=True):
+    """Train the codebook around a given key, optionally holding it."""
+    set_seed(seed)
+    m = SSE(P=P, vu=vu, d=d, users=U).to(DEVICE)
+    with torch.no_grad():
+        m.W.copy_(W0.to(DEVICE))
+    m.W.requires_grad_(not frozen)
+    L.train_sse(m, iters=iters, batch=256, lr=3e-3, seed=seed)
+    m.calibrate_power()
+    return m
+
+
+def ms(vals):
+    """Mean and, when there is more than one, the sample spread."""
+    if len(vals) == 1:
+        return vals[0], 0.0
+    return statistics.mean(vals), statistics.stdev(vals)
+
+
+def part_a(rows):
+    """One user: does the choice of mask matter at all?"""
+    print("-- A. one user, d=%d, L=%d, %d seeds --"
+          % (MAIN_D, MAIN_D // 4, len(SEEDS)))
+    Lp = MAIN_D // 4
+    cases = [("Walsh row", base_keys(1, Lp), True),
+             ("all ones (no mask)", torch.ones(1, Lp), True),
+             ("learned, free", base_keys(1, Lp), False)]
+    for name, W0, frozen in cases:
+        v = [eval_ser_sse(train_with_key(W0, MAIN_D, 4, 16, 1, seed=s,
+                                         frozen=frozen),
+                          [SNR], frames=FRAMES)[0] for s in SEEDS]
+        mu, sd = ms(v)
+        print("   %-20s SER %.5f +- %.5f" % (name, mu, sd))
+        rows.append(["A one user", name, "%.5f" % mu, "%.5f" % sd, "", ""])
+
+
+def part_b(rows):
+    """Key lengths where no exactly orthogonal unit-modulus family exists."""
+    print("\n-- B. key length, U=4, %d seeds --" % len(SEEDS))
+    print("   %-4s %-9s %-18s %-18s %s"
+          % ("L", "kappa str", "structured SER", "learned SER", "verdict"))
+    for Lp in [6, 8, 10, 12, 14, 16, 18, 20, 22, 24]:
+        d = 4 * Lp
+        try:
+            W0 = base_keys(4, Lp)
+        except ValueError as e:
+            print("   %-4d skipped: %s" % (Lp, e))
+            continue
+        ks = mean_abs_xcorr(W0)
+        vs = [eval_ser_sse(train_with_key(W0, d, 4, 16, 4, seed=s),
+                           [SNR], frames=FRAMES)[0] for s in SEEDS]
+        vl = [eval_ser_sse(get_model_reg(P=4, vu=16, d=d, U=4, iters=4000,
+                                         seed=s),
+                           [SNR], frames=FRAMES)[0] for s in SEEDS]
+        (mus, sds), (mul, sdl) = ms(vs), ms(vl)
+        # a win only counts when it clears the spread of both runs
+        win = "learned" if mul + sdl < mus - sds else (
+            "structured" if mus + sds < mul - sdl else "tie")
+        print("   %-4d %-9.5f %.5f +- %.5f  %.5f +- %.5f  %s"
+              % (Lp, ks, mus, sds, mul, sdl, win))
+        rows.append(["B key length", "L=%d" % Lp, "%.5f" % mus,
+                     "%.5f" % sds, "%.5f" % mul, "%.5f/%s" % (ks, win)])
+
+
+def part_c(rows):
+    """Overload: more users than the construction has orthogonal rows."""
+    print("\n-- C. load at L=16, %d seeds --" % len(SEEDS))
+    Lp, d = 16, 64
+    print("   %-4s %-9s %-18s %-18s %s"
+          % ("U", "kappa str", "structured SER", "learned SER", "verdict"))
+    for U in [4, 8, 12, 15, 16, 20]:
+        try:
+            W0 = base_keys(U, Lp)
+        except ValueError:
+            # beyond the orthogonal rows the construction has to reuse
+            # them, which is the honest structured fallback
+            H = base_keys(Lp - 1, Lp)
+            W0 = H[[i % (Lp - 1) for i in range(U)]]
+        ks = mean_abs_xcorr(W0)
+        vs = [eval_ser_sse(train_with_key(W0, d, 4, 16, U, seed=s),
+                           [SNR], frames=FRAMES)[0] for s in SEEDS]
+        vl = [eval_ser_sse(get_model_reg(P=4, vu=16, d=d, U=U, iters=4000,
+                                         seed=s),
+                           [SNR], frames=FRAMES)[0] for s in SEEDS]
+        (mus, sds), (mul, sdl) = ms(vs), ms(vl)
+        win = "learned" if mul + sdl < mus - sds else (
+            "structured" if mus + sds < mul - sdl else "tie")
+        print("   %-4d %-9.5f %.5f +- %.5f  %.5f +- %.5f  %s"
+              % (U, ks, mus, sds, mul, sdl, win))
+        rows.append(["C load", "U=%d" % U, "%.5f" % mus, "%.5f" % sds,
+                     "%.5f" % mul, "%.5f/%s" % (ks, win)])
+
+
+def main():
+    print("device", DEVICE)
+    rows = []
+    part_a(rows)
+    part_b(rows)
+    part_c(rows)
+    out = DATA / "jscc.csv"
+    with open(out, "w", newline="") as f:
+        w = csv.writer(f)
+        w.writerow(["part", "case", "structured_ser", "structured_sd",
+                    "learned_ser", "kappa_and_verdict"])
+        w.writerows(rows)
+    print("\n[csv]", out)
+
+
+if __name__ == "__main__":
+    main()

code/diag_whygap.py

@@ -0,0 +1,120 @@
+# -*- coding: utf-8 -*-
+"""Why do structured keys beat learned ones on the legitimate error rate?
+
+The gap is 0.053 against 0.064 at 10 dB, and a reader may reasonably
+suspect that the learned keys are handicapped, since they alone are
+trained under the channel while the structured ones are fixed by
+construction. This measures where the gap comes from.
+
+Three questions, one experiment each.
+
+1. Is the gap a channel-adaptation failure? If it were, the two families
+   would differ by more at some channel qualities than at others. The
+   ratio across the SNR sweep answers this from data already on disk.
+
+2. Is the structured key a point that training can improve on? Start
+   training from the Walsh-Hadamard keys with the masks unfrozen and let
+   Adam move them. If the structured point is a genuine optimum, the
+   error rate stays or rises; if training is merely under-converged from
+   its random start, it falls.
+
+3. What does the learned key lose? Two candidates, measured directly:
+   residual cross-user correlation, which the analysis names as the
+   first-order leakage and interference term, and departure from unit
+   modulus, which spreads the key energy unevenly across the entries so
+   that a digit is decided over an effectively shorter support.
+
+Run: python code/diag_whygap.py
+"""
+from __future__ import annotations
+
+import csv
+import math
+import sys
+from pathlib import Path
+
+import torch
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from exp_full import (MAIN_D, base_keys, get_model, main_model,  # noqa: E402
+                      mean_abs_xcorr)
+from sse_lib import DATA, DEVICE, eval_ser_sse  # noqa: E402
+import sse_lib as L  # noqa: E402
+
+FRAMES = 300_000
+SNR = 10.0
+
+
+def modulus_stats(W):
+    """How far the key entries are from unit modulus, per user.
+
+    A unit-modulus key puts the same energy on every entry, so the
+    signal term does not depend on the key and every entry of the period
+    carries its share of the decision. The ratio below is the effective
+    fraction of the L entries the key actually uses, by the
+    participation ratio (sum a^2)^2 / (L sum a^4) with a the entry
+    magnitudes. It is one for a unit-modulus key and 1/L for a key that
+    puts everything on one entry.
+    """
+    a2 = W.pow(2)
+    pr = a2.sum(dim=1).pow(2) / (W.shape[1] * a2.pow(2).sum(dim=1))
+    return pr
+
+
+def report(name, model, rows):
+    W = model.masks().detach().cpu()
+    ser = eval_ser_sse(model, [SNR], frames=FRAMES)[0]
+    kap = mean_abs_xcorr(model.masks().detach())
+    pr = modulus_stats(W)
+    print("%-24s SER %.5f   kappa-bar %.5f   entry use %.3f"
+          % (name, ser, kap, float(pr.mean())))
+    rows.append([name, "%.5f" % ser, "%.5f" % kap, "%.4f" % float(pr.mean())])
+    return ser
+
+
+def main():
+    rows = []
+    print("main configuration d=%d, L=%d, 10 dB, %d frames\n"
+          % (MAIN_D, MAIN_D // 4, FRAMES))
+
+    print("-- the two families as the paper plots them --")
+    fix = main_model()
+    s_fix = report("structured (frozen)", fix, rows)
+    free = get_model(iters=4000)
+    s_free = report("learned (free start)", free, rows)
+
+    print("\n-- question 2: can training improve on the structured key? --")
+    # same trainer, same iterations, same seed, but the masks start at
+    # the Walsh-Hadamard point and are free to move
+    from sse_lib import SSE, set_seed
+    set_seed(1)
+    m = SSE(P=4, vu=16, d=MAIN_D, users=4).to(DEVICE)
+    with torch.no_grad():
+        m.W.copy_(base_keys(4, MAIN_D // 4).to(DEVICE))
+    m.W.requires_grad_(True)
+    L.train_sse(m, iters=4000, batch=256, lr=3e-3, seed=1)
+    m.calibrate_power()
+    s_warm = report("learned (Walsh start)", m, rows)
+
+    print("\nreading:")
+    print("  free start   %+.1f percent against the structured key"
+          % (100.0 * (s_free - s_fix) / s_fix))
+    print("  Walsh start  %+.1f percent against the structured key"
+          % (100.0 * (s_warm - s_fix) / s_fix))
+    if s_warm > s_fix:
+        print("  training moves off the structured point and pays for it,")
+        print("  so the structured key is not a point learning improves on.")
+    else:
+        print("  training improves on the structured point, so the gap is")
+        print("  under-convergence from the random start, not geometry.")
+
+    out = DATA / "whygap.csv"
+    with open(out, "w", newline="") as f:
+        w = csv.writer(f)
+        w.writerow(["family", "legit_ser", "kappa_bar", "entry_use"])
+        w.writerows(rows)
+    print("\n[csv]", out)
+
+
+if __name__ == "__main__":
+    main()

code/exp_full.py

@@ -194,11 +194,14 @@ def stage_N():
     The structured family is enumerable and closed under the elementwise
     product, the learned one is neither, so the paper reports both. This
     stage runs the same SNR sweep as stage_A with the keys trained in
-    R^L instead of frozen to Walsh-Hadamard rows, at the same frame
+    R^L under the regularized loss instead of frozen to Walsh-Hadamard
-    count, so the two are directly comparable.
+    rows, at the same frame count, so the two are directly comparable.
     """
     print("[N] security vs SNR, learned key family ...")
-    m = get_model(P=4, vu=16, d=MAIN_D, U=4, iters=4000, seed=1)
+    # the regularized loss of Section V-C, not the cross-entropy alone:
+    # see exp_learned.learned_model for why the unpenalized keys are not
+    # the family the paper claims
+    m = get_model_reg(P=4, vu=16, d=MAIN_D, U=4, iters=4000, seed=1)
     snr = [float(v) for v in range(0, 21, 2)]
     frames = 800_000
     legit = eval_ser_sse(m, snr, frames=frames)

code/exp_learned.py

@@ -11,7 +11,8 @@ ones.
 
 Every evaluation mirrors its structured counterpart exactly: same SNR,
 same frame counts, same seeds, same evaluators. Only the key family
-differs.
+differs. The learned keys are the regularized ones of Section V-C, not
+the unpenalized ones: see learned_model below for why.
 """
 from __future__ import annotations
 
@@ -21,16 +22,28 @@ from pathlib import Path
 import torch
 
 import exp_kpa
+import exp_refresh
 from exp_full import (MAIN_D, eval_ser_eve, eval_ser_jam, eve_wrong_mask,
-                      get_model, mean_abs_xcorr, oma_ser_keylen)
+                      get_model_reg, mean_abs_xcorr, oma_ser_keylen)
 from sse_lib import DATA, DEVICE, eval_ser_sse, write_csv
 
 SEED = 1
 
 
 def learned_model(d=MAIN_D, P=4, vu=16, U=4, iters=4000, seed=SEED):
-    """The learned counterpart of main_model: same everything, keys free."""
+    """The learned counterpart of main_model: same everything, keys free.
-    return get_model(P=P, vu=vu, d=d, U=U, iters=iters, seed=seed)
+
+    The keys are trained under the two penalties of the regularized loss
+    rather than under the cross-entropy alone. Cross-entropy on its own
+    has an attractor at disjoint sparse supports, which is an orthogonal
+    slot allocation: the keys it reaches carry 99 percent of their
+    energy on about six of the L entries, so a digit is decided over a
+    sixth of its period and the key set is a choice of support rather
+    than a dense direction in R^L. The penalties are the design of
+    Section V-C and hold that drift back, which is the realization the
+    paper claims for the learned family.
+    """
+    return get_model_reg(P=P, vu=vu, d=d, U=U, iters=iters, seed=seed)
 
 
 def keylen():
@@ -56,7 +69,7 @@ def jamming():
     """Fig. 4's learned curves."""
     print("[learned] jamming ...")
     m = learned_model()
-    jsr = [-10.0, -5.0, 0.0, 5.0, 10.0, 15.0, 20.0]
+    jsr = [float(v) for v in range(-10, 21, 2)]   # the grid Fig. 4's other curves use
     blind = eval_ser_jam(m, 10.0, jsr, frames=500_000, mode="blind", target=0)
     matched = eval_ser_jam(m, 10.0, jsr, frames=500_000, mode="matched",
                            target=0)
@@ -106,7 +119,7 @@ def refresh():
     W0, B0 = m.W.detach().clone(), m.B.detach().clone()
     base = eval_ser_sse(m, [10.0], frames=300_000)[0]
     out = []
-    for b in range(8):
+    for b in range(exp_refresh.BLOCKS):
         g = torch.Generator(device=DEVICE).manual_seed(5150 + b)
         xi = torch.randperm(m.L, generator=g, device=DEVICE)
         eps = torch.randint(2, (m.L,), generator=g, device=DEVICE) * 2.0 - 1.0
@@ -133,11 +146,15 @@ def compare():
     print("[learned] scheme comparison ...")
     m = learned_model()
     F = 300_000
-    legit = eval_ser_sse(m, [10.0], frames=F)[0]
+    # the table caption states a user-1 convention and every structured
-    out = eval_ser_eve(m, eve_wrong_mask(m.users, m.L,
+    # row honours it, so this row uses the same evaluator rather than the
-                                         seed=20260813).to(DEVICE),
+    # four-user average eval_ser_eve returns
-                       [10.0], frames=F)[0]
+    from exp_full import eval_scheme
-    ins = eval_ser_eve(m, m.masks().detach().roll(1, 0), [10.0], frames=F)[0]
+    legit = eval_scheme(m, 10.0, F)
+    out = eval_scheme(m, 10.0, F,
+                      rx_masks=eve_wrong_mask(m.users, m.L,
+                                              seed=20260813).to(DEVICE))
+    ins = eval_scheme(m, 10.0, F, rx_masks=m.masks().detach().roll(1, 0))
     jam = eval_ser_jam(m, 10.0, [0.0], frames=F, mode="blind", target=0)[0]
     write_csv(DATA / "compare_learned.csv",
               ["scheme", "legit_ser", "eve_out", "eve_in", "jam0_ser"],
@@ -157,3 +174,107 @@ def main():
 
 if __name__ == "__main__":
     main()
+
+
+def sens():
+    """Fig. 5's learned curve: eavesdropper SER against the fraction of
+    the key the attacker holds. correlated_masks builds a substitute at
+    a prescribed correlation to any real key, so the sweep applies to a
+    learned key exactly as to a sign pattern."""
+    from exp_full import correlated_masks
+    print("[learned] key sensitivity ...")
+    m = learned_model()
+    F, TR = 600_000, 12
+    true_m = m.masks().detach().cpu()
+    gen = torch.Generator().manual_seed(31)
+    fracs = [0.0, 0.2, 0.4, 0.6, 0.75, 0.85, 0.9, 0.92, 0.94, 0.955,
+             0.97, 0.985, 1.0]
+    rows = []
+    for f in fracs:
+        acc = [eval_ser_eve(m, correlated_masks(true_m, f, gen), [10.0],
+                            frames=F // TR, seed=777 + 17 * t)[0]
+               for t in range(TR)]
+        rows.append((f, sum(acc) / len(acc)))
+    write_csv(DATA / "sec_sens_learned.csv", ["frac", "ser_mask"], rows)
+    print("   f=0 %.4f  f=1 %.4f" % (rows[0][1], rows[-1][1]))
+
+
+def brute():
+    """Fig. 6's learned curve. The best-of-K correlation is a property of
+    the key space, which both realizations share at the same L, so only
+    the sensitivity mapping differs and it is re-read from the learned
+    sweep."""
+    import csv as _csv
+    import numpy as np
+    print("[learned] brute-force search ...")
+    with open(DATA / "sec_sens_learned.csv") as f:
+        cmp_rows = list(_csv.DictReader(f))
+    f_arr = np.array([float(r["frac"]) for r in cmp_rows])
+    mask_arr = np.array([float(r["ser_mask"]) for r in cmp_rows])
+    L = MAIN_D // 4
+    ks = [1, 3, 10, 30, 100, 300, 1_000, 3_000, 10_000, 30_000, 65_536,
+          100_000, 300_000, 1_000_000]
+    rng = np.random.default_rng(2026)
+    rows = []
+    for K in ks:
+        best = np.sqrt(rng.beta(0.5, (L - 1) / 2.0, size=(400, K)).max(1))
+        rows.append((K, float(np.mean(np.interp(best, f_arr, mask_arr)))))
+    write_csv(DATA / "sec_brute_learned.csv", ["K", "ser_mask"], rows)
+    print("   K=1e6 %.4f" % rows[-1][1])
+
+
+def real():
+    """Fig. 8's learned curves.
+
+    exp_real_sec writes fixed file names, so the structured artifacts are
+    held aside, the run is repeated with the learned model, its output is
+    copied to *_learned names, and the originals are put back. A failure
+    anywhere restores them.
+    """
+    import shutil
+    import exp_real_sec as R
+    print("[learned] real token streams ...")
+    names = ("real_sec_ter.csv", "real_sec_stats.json")
+    saved = {n: (DATA / n).read_bytes() for n in names
+             if (DATA / n).exists()}
+    orig = R.main_model
+    try:
+        R.main_model = lambda **kw: learned_model(
+            d=kw.get("d", MAIN_D), P=kw.get("P", 4),
+            vu=kw.get("vu", 16), U=kw.get("U", 4))
+        R.main()
+        for n in names:
+            if (DATA / n).exists():
+                shutil.copyfile(DATA / n,
+                                DATA / n.replace(".", "_learned.", 1))
+    finally:
+        R.main_model = orig
+        for n, blob in saved.items():
+            (DATA / n).write_bytes(blob)
+    print("   learned artifacts written, structured ones restored")
+
+
+def keylen_perm():
+    """Fig. 3's permutation-key curves.
+
+    The permutation scheme keeps the masks public and hides the frame
+    order instead, so its legitimate receiver inverts the permutation
+    and decodes as the public-mask receiver does, while its eavesdropper
+    holds the public masks but not the order. Both are swept over the
+    same key lengths as the structured family so the figure carries the
+    comparison scheme at every point rather than only at L = 64.
+    """
+    import torch
+    from exp_full import (eval_scheme, eval_scheme_permuted_eve, main_model)
+    print("[learned] key length, permutation key ...")
+    rows = []
+    for d in [32, 48, 64, 80, 96, 128, 192, 256]:
+        m = main_model(d=d)
+        gp = torch.Generator().manual_seed(11)
+        perms = torch.randperm(d, generator=gp)[None].repeat(m.users, 1)
+        lg = eval_scheme(m, 10.0, 500_000, perms=perms)
+        ev = eval_scheme_permuted_eve(m, 10.0, 500_000, perms)
+        rows.append((m.L, d, lg, ev))
+        print("   L=%3d  legit %.4f  eve %.4f" % (m.L, lg, ev))
+    write_csv(DATA / "sec_keylen_perm.csv",
+              ["L", "d", "legit_ser", "eve_ser"], rows)

code/make_tables.py

@@ -11,17 +11,17 @@ from pathlib import Path
 DATA = Path(__file__).resolve().parents[1] / "data"
 
 NAME = {
-    "proposed": r"\textbf{KM (structured)}",
+    "proposed": r"\textbf{KM (str.)}",
-    "proposed_learned": r"\textbf{KM (learned)}",
+    "proposed_learned": r"\textbf{KM (lrn.)}",
     "public_mask": "Public masks",
     "perm_key": r"Permutation key~\cite{chen2025shufflingtifs}",
     "index_cipher": "Index cipher",
-    "oma_plain": "OMA (no encryption)",
+    "oma_plain": "OMA",
     "random": "Random",
     "hadamard": "Structured",
     "learned": "Learned, plain",
     "learned_reg": r"Learned, regularized~\eqref{eq:regloss}",
-    "invariant_learned": r"\textbf{Invariant, learned keys}",
+    "invariant_learned": r"\textbf{Invariant, KM (lrn.)}",
 }
 RECEIVER = {
     "legit": "Legitimate", "oma": "OMA",
@@ -59,8 +59,9 @@ def cell(x: str, bold: bool, wide: bool = False) -> str:
 def compare_table():
     print("% Table: scheme comparison (from sec_compare.csv)")
     rows = list(csv.DictReader(open(DATA / "sec_compare.csv")))
-    order = ["public_mask", "perm_key", "index_cipher", "oma_plain",
+    # 9.3: the proposal first, as every figure legend lists it
-             "proposed", "proposed_learned"]
+    order = ["proposed", "proposed_learned", "public_mask", "perm_key",
+             "index_cipher", "oma_plain"]
     rows.sort(key=lambda r: order.index(r["scheme"]))
     # stage_E does not jam the orthogonal reference, because the jammer an
     # OMA user faces is targeted at public slots rather than mask-matched

code/merge_learned_rows.py

@@ -0,0 +1,75 @@
+# -*- coding: utf-8 -*-
+"""Fold the learned-key rows into the two table sources.
+
+Table IV reads sec_compare.csv and Table V reads refresh_summary.csv,
+and both are written by the structured stages, which know nothing about
+the learned family. Its rows were appended by hand, so a rerun of the
+learned stages left the tables behind. This does the fold, so both files
+are derived from data/ like every other table source.
+
+Run after code/run_learned_reg.py, before code/make_tables.py.
+"""
+from __future__ import annotations
+
+import csv
+from pathlib import Path
+
+DATA = Path(__file__).resolve().parents[1] / "data"
+
+
+def read(name):
+    with open(DATA / name) as f:
+        r = csv.DictReader(f)
+        return r.fieldnames, list(r)
+
+
+def write(name, fields, rows):
+    with open(DATA / name, "w", newline="") as f:
+        w = csv.DictWriter(f, fieldnames=fields)
+        w.writeheader()
+        w.writerows(rows)
+
+
+def upsert(rows, key, value, row):
+    """Replace the row carrying key==value, or append it."""
+    for i, r in enumerate(rows):
+        if r[key] == value:
+            rows[i] = row
+            return rows
+    rows.append(row)
+    return rows
+
+
+def main():
+    # Table IV: the learned scheme row, measured by exp_learned.compare
+    fields, rows = read("sec_compare.csv")
+    _, learned = read("compare_learned.csv")
+    assert len(learned) == 1, "compare_learned.csv should carry one row"
+    rows = upsert(rows, "scheme", "proposed_learned",
+                  {k: learned[0][k] for k in fields})
+    write("sec_compare.csv", fields, rows)
+    print("sec_compare.csv       proposed_learned  jam0 %s"
+          % learned[0]["jam0_ser"])
+
+    # Table V: the learned refresh row, averaged over the blocks that
+    # exp_learned.refresh measured, at the same entropy as the
+    # structured refresh because the invariance group is the same
+    fields, rows = read("refresh_summary.csv")
+    _, blocks = read("refresh_learned.csv")
+    lg = sum(float(r["legit_ser"]) for r in blocks) / len(blocks)
+    ev = sum(float(r["eve_ser"]) for r in blocks) / len(blocks)
+    ent = next(r["entropy_bits"] for r in rows
+               if r["scheme"] == "Invariant, KM (str.)")
+    rows = upsert(rows, "scheme", "Invariant, KM (lrn.)",
+                  {"scheme": "Invariant, KM (lrn.)",
+                   "legit": "%.6f" % lg, "eve": "%.6f" % ev,
+                   "entropy_bits": ent})
+    # 9.3: the proposal first, as every figure legend lists it
+    rows.sort(key=lambda x: 0 if x["scheme"].startswith("Invariant") else 1)
+    write("refresh_summary.csv", fields, rows)
+    print("refresh_summary.csv   Invariant, KM (lrn.)  legit %.5f  eve %.5f"
+          % (lg, ev))
+
+
+if __name__ == "__main__":
+    main()

code/replot_security.py

@@ -25,6 +25,7 @@ import math
 import matplotlib
 matplotlib.use("Agg")
 import matplotlib.pyplot as plt
+import matplotlib.ticker as mticker
 
 ROOT = Path(__file__).resolve().parents[1]
 DATA = ROOT / "data"
@@ -51,9 +52,11 @@ plt.rcParams.update({
     "grid.alpha": 0.6,
     "lines.linewidth": 1.5,
     "lines.markersize": 5.2,
-    "figure.figsize": (3.15, 2.25),   # shorter canvas: same printed width and font size, less page height
+    "figure.figsize": (3.15, 2.443),  # 8:6 axes box with the AXES_RECT below
     "pdf.fonttype": 42,
 })
+
+# one axes rectangle for every figure, so the boxes align across the page
 AXES_RECT = dict(left=0.215, right=0.970, top=0.955, bottom=0.225)
 
 C_LEGIT = "#c0392b"     # KM, structured keys
@@ -62,7 +65,8 @@ C_OMA = "#7f8c8d"       # orthogonal multiple access
 C_PUB = "#16a085"       # public masks
 C_PERM = "#8e44ad"      # permutation key
 C_PAD = "#a0522d"       # index cipher
-C_EVE = "#2c5fa8"       # an adversary of KM
+C_EVE = "#2c5fa8"       # an outsider of KM
+C_INS = "#00806b"       # an insider of KM, which sits just below it
 C_CH = "#95a5a6"        # chance and reference levels
 C_MATCH = C_PUB         # the matched jammer is what public masks admit
 
@@ -79,24 +83,23 @@ STY = {
     "perm":    dict(color=C_PERM, marker="X", ls="--"),
     "pad":     dict(color=C_PAD, marker="P", ls="-."),
     "eve":     dict(color=C_EVE, marker="s", ls="--"),
-    "insider": dict(color=C_EVE, marker="v", ls="-."),
+    "insider": dict(color=C_INS, marker="v", ls="-."),
 }
 
 # fixed label dictionary: tables and prose copy these strings verbatim
 LBL = {
-    "legit": "KM (structured)",
+    "legit": "KM (str.)",
-    "legit_learned": "KM (learned)",
+    "legit_learned": "KM (lrn.)",
     "oma": "OMA",
-    "eve_pub": "Eavesdropper, public masks",
+    "eve_pub": "Outsider, public masks",
-    "eve_key": "Eavesdropper",     # the wrong-key condition is in the caption
+    "eve_key": "Outsider, KM (str.)",     # the wrong-key condition is in the caption
     "chance": "Random guess",
     "nojam": "No jammer",
-    "mask": "KM (structured)",
+    "mask": "KM (str.)",
     "perm": "Permutation key",
     "pad": "Index cipher",
-    "insider": "Insider",
+    "insider": "Insider, KM (str.)",
-    "legit_ref": "Legitimate rate",
+    "outsider": "Outsider, KM (str.)",
-    "outsider": "Outsider",
 }
 # deliberate-layering style for the LOWER of two coinciding curves
 UNDER = dict(lw=2.6, alpha=0.85)          # thick filled line, layered under
@@ -223,10 +226,32 @@ def main_legit(snr_db="10"):
     raise KeyError("no %s dB row in sec_snr.csv" % snr_db)
 
 
+
+# Legend order, applied by place_legend to whatever subset a figure
+# draws: the proposal first, then the comparison schemes in the order of
+# Table IV, then adversaries, then reference levels. Entries not listed
+# keep their plot order after the ranked ones.
+LEGEND_ORDER = [
+    "KM (str.)", "KM (lrn.)",
+    "Public masks", "Permutation key", "Index cipher", "OMA",
+    "Outsider, KM (str.)", "Outsider, public masks", "Insider, KM (str.)",
+    "No jammer", "Random guess",
+]
+
+
+def _rank(label):
+    """Rank a legend label, matching the collection-SNR variants of
+    Fig. 7 on their scheme prefix so they stay together and in order."""
+    for i, name in enumerate(LEGEND_ORDER):
+        if label == name or label.startswith(name + ","):
+            return i
+    return len(LEGEND_ORDER)
+
+
 def place_legend(ax, cands=("lower left", "upper left", "center left",
                            "center right", "lower center", "upper right",
                            "upper center", "center", "lower right"),
-                 sizes=(9.2, 8.8, 8.4, 8.0, 7.6, 7.2, 6.8, 6.4), ncol=1):
+                 sizes=(9.2, 8.8, 8.4, 8.0, 7.6, 7.4), ncol=1):
     """Choose the location and font size whose box the fewest curve points
     fall inside, scored on rendered geometry rather than guessed from the
     data. The size sweep is what makes a long label set placeable: a
@@ -247,7 +272,11 @@ def place_legend(ax, cands=("lower left", "upper left", "center left",
     best = None
     for size in sizes:
         for loc in cands:
-            leg = ax.legend(loc=loc, prop={"size": size}, ncol=ncol,
+            h, l = ax.get_legend_handles_labels()
+            idx = sorted(range(len(l)), key=lambda k: (_rank(l[k]), k))
+            h = [h[k] for k in idx]
+            l = [l[k] for k in idx]
+            leg = ax.legend(h, l, loc=loc, prop={"size": size}, ncol=ncol,
                             handlelength=1.4, columnspacing=0.9,
                             handletextpad=0.5, borderaxespad=0.55,
                             framealpha=1.0)
@@ -270,13 +299,21 @@ def place_legend(ax, cands=("lower left", "upper left", "center left",
             if best is None or hits < best[2]:
                 best = (loc, size, hits)
             if hits == 0:
-                ax.legend(loc=loc, prop={"size": size}, ncol=ncol,
+                h, l = ax.get_legend_handles_labels()
+                idx = sorted(range(len(l)), key=lambda k: (_rank(l[k]), k))
+                h = [h[k] for k in idx]
+                l = [l[k] for k in idx]
+                ax.legend(h, l, loc=loc, prop={"size": size}, ncol=ncol,
                           handlelength=1.4, columnspacing=0.9,
                           handletextpad=0.5, borderaxespad=0.55,
                             framealpha=1.0)
                 PL_CHOSEN.append(size)
                 return best
-    ax.legend(loc=best[0], prop={"size": best[1]}, ncol=ncol,
+    h, l = ax.get_legend_handles_labels()
+    idx = sorted(range(len(l)), key=lambda k: (_rank(l[k]), k))
+    h = [h[k] for k in idx]
+    l = [l[k] for k in idx]
+    ax.legend(h, l, loc=best[0], prop={"size": best[1]}, ncol=ncol,
               handlelength=1.4, columnspacing=0.9, handletextpad=0.5,
               borderaxespad=0.55, framealpha=1.0)
     PL_CHOSEN.append(best[1])
@@ -304,11 +341,11 @@ def fig_snr():
                 label=LBL["eve_pub"])
     # this figure carries two eavesdroppers, so the bare label of the
     # key-length figure would not tell them apart
-    ax.semilogy(x, col(r, "eve_wrong"), **STY["eve"], label="Eavesdropper, keyed")
+    ax.semilogy(x, col(r, "eve_wrong"), **STY["eve"], label=LBL["eve_key"])
     # the chance level lies within 3.5e-4 of the wrong-key curve, so it is
     # drawn for reference but left out of the legend, which the caption
     # names instead; five long entries leave this figure no clear corner
-    ax.plot(x, col(r, "chance"), color=C_CH, ls="-.", lw=0.9)
+    ax.plot(x, col(r, "chance"), color=C_CH, ls=":", lw=0.9)
     ax.set_xlabel("SNR (dB)")
     ax.set_ylabel("SER")
     ax.set_xlim(min(x), max(x))
@@ -326,44 +363,58 @@ def fig_keylen():
     r = load("sec_keylen.csv")
     x = col(r, "L", int)
     fig, ax = plt.subplots()
-    ax.semilogy(x, col(r, "legit_ser"), **STY["km_str"], label=LBL["legit"])
+    ax.semilogy(x, col(r, "legit_ser"), **STY["km_str"], markevery=(0, 2),
+                label=LBL["legit"], **UNDER)
     rl = load("sec_keylen_learned.csv")
     ax.semilogy(col(rl, "L", int), col(rl, "legit_ser"), **STY["km_lrn"], label=LBL["legit_learned"])
     op = [(l, v) for l, v in zip(x, col(r, "oma")) if not math.isnan(v)]
     ax.semilogy([p[0] for p in op], [p[1] for p in op], **STY["oma"], label=LBL["oma"])
-    ax.semilogy(x, col(r, "eve_ser"), **STY["eve"], label=LBL["eve_key"])
+    ax.semilogy(x, col(r, "eve_ser"), **STY["eve"], markevery=(0, 2),
-    ax.set_ylim(top=22.0)    # headroom above the flat eavesdropper curve
+                label=LBL["eve_key"], **UNDER)
-    # an error rate cannot exceed one, and the room below the data holds
+    # the permutation key shares this physical layer, so its legitimate
-    # the legend, since every curve decays to the right
+    # curve lies on the structured one and appears at every key length
-    ax.set_ylim(top=1.4, bottom=1.2e-2)
+    # rather than only in the tables. Its outsider measures 0.99997 to
+    # 0.99999 and would lie on the outsider curve already drawn, in the
+    # same style as this one and with no legend entry of its own, so the
+    # caption says where it sits instead.
+    rp = load("sec_keylen_perm.csv")
+    assert min(float(r["eve_ser"]) for r in rp) > 0.999,         "the permutation outsider left the random-guess level"
+    ax.semilogy(col(rp, "L", int), col(rp, "legit_ser"), **STY["perm"],
+                markevery=(1, 2), label=LBL["perm"], **OVER)
+    # every curve is high at short key lengths and decays to the right,
+    # so the lower left is clear and the limits only frame the data
+    ax.set_ylim(top=1.4, bottom=8.0e-3)
     ax.set_xlabel("Key length $L$")
     ax.set_ylabel("SER")
     ax.set_xscale("log", base=2)
-    place_legend(ax)
+    place_legend(ax, cands=("lower left",))
     save(fig, "fig_sec_keylen")
 
 
 def fig_jam():
-    """Target-user SER against JSR in four cases. A linear axis is used
+    """Target-user SER against JSR in five cases, on a log ordinate.
-    because the range spans less than one decade, where a log axis would
+
-    print wide minor tick labels that crowd out the y label. The
+    With the unjammed reference the range spans 0.053 to 0.998, over a
-    no-jammer reference is named in the caption rather than in the
+    decade, so the axis carries two major ticks and the minor tick
-    legend, which keeps the legend four rows tall."""
+    labels that crowd a sub-decade log axis are suppressed. The room
+    below the data holds the legend, which is why the earlier linear
+    version reserved a band below zero instead. The no-jammer reference
+    is named in the caption rather than in the legend."""
     r = load("sec_jam_cmp.csv")
     x = col(r, "jsr_db")
     me = max(1, len(x) // 8)
     fig, ax = plt.subplots()
     rj = load("sec_jam_learned.csv")
-    ax.plot(col(rj, "jsr_db"), col(rj, "blind"), **STY["km_lrn"],
+    ax.semilogy(col(rj, "jsr_db"), col(rj, "blind"), **STY["km_lrn"],
             markevery=(1, me), label=LBL["legit_learned"])
-    ax.plot(x, col(r, "matched"), **STY["pub"],
+    ax.semilogy(x, col(r, "matched"), **STY["pub"],
             markevery=me, label="Public masks")
-    ax.plot(x, col(r, "oma_targeted"), **STY["oma"],
+    ax.semilogy(x, col(r, "oma_targeted"), **STY["oma"],
             markevery=me, label=LBL["oma"])
     # the two blind curves agree to 0.002; deliberate layering
-    ax.plot(x, col(r, "blind"), **STY["km_str"],
+    ax.semilogy(x, col(r, "blind"), **STY["km_str"],
             markevery=(0, me), label=LBL["mask"], **UNDER)
-    ax.plot(x, col(r, "perm_blind"), **STY["perm"],
+    ax.semilogy(x, col(r, "perm_blind"), **STY["perm"],
             markevery=(me // 2, me), label=LBL["perm"], **OVER)
     nojam = float(load("sec_jam.csv")[0]["nojam"])
     # the unjammed reference is named in the caption rather than in the
@@ -371,9 +422,12 @@ def fig_jam():
     # Behind the legend rather than through it. The placement guard skips
     # axis-spanning lines, so it cannot move the legend off this one, and
     # a reference drawn along the legend frame reads as part of the box.
-    ax.axhline(nojam, color=C_OMA, ls=(0, (1, 3)), lw=0.9, zorder=0)
+    ax.axhline(nojam, color=C_CH, ls=":", lw=0.9, zorder=0)
-    ax.set_ylim(-0.42, 1.05)
+    ax.set_ylim(2.5e-2, 1.4)
-    ax.set_yticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0])
+    # a log axis spanning little more than a decade prints minor labels
+    # like 6x10^-1 that consume the left margin, so only the decades are
+    # labelled
+    ax.yaxis.set_minor_formatter(mticker.NullFormatter())
     ax.set_xlabel("JSR (dB)")
     ax.set_ylabel("SER")
     ax.set_xlim(min(x), max(x))
@@ -390,6 +444,9 @@ def fig_sens():
     fig, ax = plt.subplots()
     ax.plot(x, col(r, "ser_mask"), **STY["km_str"],
             markevery=(0, 3), label=LBL["mask"], **UNDER)
+    rs = load("sec_sens_learned.csv")
+    ax.plot(col(rs, "frac"), col(rs, "ser_mask"), **STY["km_lrn"],
+            markevery=(1, 3), label=LBL["legit_learned"])
     ax.plot(x, col(r, "ser_perm"), **STY["perm"],
             markevery=(1, 3), label=LBL["perm"], **OVER)
     ax.plot(x, col(r, "ser_pad"), **STY["pad"],
@@ -398,11 +455,8 @@ def fig_sens():
     # copy of the configuration constants
     chance = float(load("sec_snr.csv")[0]["chance"])
     ax.axhline(chance, color=C_CH, ls=":", lw=0.9, label=LBL["chance"])
-    # the narration reads these curves against the legitimate rate
-    ax.axhline(main_legit(), color=C_OMA, ls=(0, (4, 2)), lw=0.9,
-               label=LBL["legit_ref"])
     ax.set_xlabel("Fraction of the key recovered")
-    ax.set_ylabel("Eavesdropper SER")
+    ax.set_ylabel("Outsider SER")
     ax.set_xlim(0, 1)
     place_legend(ax)
     save(fig, "fig_sec_sens")
@@ -420,11 +474,15 @@ def fig_brute():
                 markevery=(1, 3), label=LBL["pad"], **OVER)
     ax.semilogx(x, col(r, "ser_mask"), **STY["km_str"],
                 markevery=(2, 3), label=LBL["mask"])
-    legit = main_legit()
+    rb = load("sec_brute_learned.csv")
-    ax.axhline(legit, color=C_OMA, ls=(0, (4, 2)), lw=0.9,
+    ax.semilogx(col(rb, "K"), col(rb, "ser_mask"), **STY["km_lrn"],
-               label=LBL["legit_ref"])
+                markevery=(1, 3), label=LBL["legit_learned"])
+    # the same chance reference the sensitivity figure carries, so a
+    # curve sitting at the top is read as learning nothing
+    ax.axhline(float(load("sec_snr.csv")[0]["chance"]), color=C_CH,
+               ls=":", lw=0.9, label=LBL["chance"])
     ax.set_xlabel("Number of key guesses $K$")
-    ax.set_ylabel("Eavesdropper SER")
+    ax.set_ylabel("Outsider SER")
     ax.set_ylim(0.0, 1.05)      # keep the reference line off the spine
     place_legend(ax)
     save(fig, "fig_sec_brute")
@@ -434,19 +492,26 @@ def fig_real():
     r = load("real_sec_ter.csv")
     x = col(r, "snr_db")
     fig, ax = plt.subplots()
-    # insider and outsider still nearly coincide and are layered; the
+    # two pairs nearly coincide here, the two legitimate realizations
-    # legitimate and OMA curves are separate at this frame
+    # within a fifth of each other and the two adversaries both at the
+    # top, so each pair is layered and its markers staggered
     ax.semilogy(x, col(r, "ter_legit"), **STY["km_str"],
-                markevery=(0, 2), label=LBL["legit"], **UNDER)
+                markevery=(0, 3), label=LBL["legit"], **UNDER)
+    rt = load("real_sec_ter_learned.csv")
+    ax.semilogy(col(rt, "snr_db"), col(rt, "ter_legit"), **STY["km_lrn"],
+                markevery=(1, 3), label=LBL["legit_learned"], **OVER)
     ax.semilogy(x, col(r, "ter_oma"), **STY["oma"],
-                markevery=(1, 2), label=LBL["oma"], **OVER)
+                markevery=(2, 3), label=LBL["oma"])
     ax.semilogy(x, col(r, "ter_insider"), **STY["insider"],
-                markevery=(0, 2), label=LBL["insider"], **UNDER)
+                markevery=(0, 3), label=LBL["insider"], **UNDER)
     ax.semilogy(x, col(r, "ter_eve"), **STY["eve"],
-                markevery=(1, 2), label=LBL["outsider"], **OVER)
+                markevery=(2, 3), label=LBL["outsider"], **OVER)
     ax.set_xlabel("SNR (dB)")
     ax.set_ylabel("TER")
     ax.set_xlim(min(x), max(x))
+    # the adversary labels name their realization, which is wide, so
+    # the axis opens below the data to give the legend a clear corner
+    ax.set_ylim(bottom=3e-5)
     place_legend(ax)
     save(fig, "fig_sec_real")
 
@@ -457,14 +522,22 @@ def fig_kpa():
     comparison scheme."""
     r = load("kpa.csv")
     fig, ax = plt.subplots()
-    sty = {0.0: (C_LEGIT, "o"), 10.0: (C_EVE, "s"),
+    # the three curves are one scheme at three collection SNRs, so the
-           20.0: (C_PUB, "v")}
+    # colour stays the scheme's and the marker and line carry the SNR
-    for off, (snr, (c, mk)) in enumerate(sty.items()):
+    # one scheme, three collection SNRs, so the marker stays the
+    # scheme's and only the line style and the fill carry the SNR;
+    # borrowing another scheme's marker shape would read as that scheme
+    # the three curves are one scheme at three collection SNRs, and the
+    # 10 and 20 dB ones run within 0.001 of each other from four frames
+    # on, so face and width separate them rather than the dash alone
+    sty = {0.0: ("o", "-", 2.6, None), 10.0: ("o", "--", 1.5, "none"),
+           20.0: ("o", ":", 1.0, "none")}
+    for off, (snr, (mk, ls, lw, mfc)) in enumerate(sty.items()):
         rows = [row for row in r if float(row["snr_db"]) == snr]
         n = [float(row["n_frames"]) for row in rows]
         ser = [float(row["eve_ser"]) for row in rows]
-        ax.semilogx(n, ser, color=c, marker=mk, ls="-",
+        ax.semilogx(n, ser, color=STY["km_str"]["color"], marker=mk, ls=ls,
-                    markevery=(off, 4), markerfacecolor="none" if off else c,
+                    lw=lw, markevery=(off, 4), markerfacecolor=mfc,
                     label=f"KM (str.), {int(snr)} dB")
         if snr == 10.0:
             kl = [q for q in load("kpa_learned.csv")
@@ -475,20 +548,20 @@ def fig_kpa():
                         label=f"KM (lrn.), {int(snr)} dB")
     try:
         p = load("pkpa.csv")
-        ax.semilogx(col(p, "n_frames"), col(p, "eve_ser"), color=C_MATCH,
+        ax.semilogx(col(p, "n_frames"), col(p, "eve_ser"), **STY["perm"],
-                    marker="P", ls="--", markevery=(3, 4),
+                    markevery=(3, 4), label=LBL["perm"] + ", 20 dB")
-                    label=LBL["perm"] + ", 20 dB")
     except FileNotFoundError:
         print("[skip] pkpa.csv not present yet")
     # legitimate reference measured with the SAME estimator as the
     # eavesdropper curves, namely the four-user average of eval_ser_sse
     # in the main configuration, rather than the user-1 convention of the
     # scheme-comparison table
-    legit = main_legit()
+    # the same chance reference the sensitivity figure carries, so a
-    ax.axhline(legit, color=C_OMA, ls=(0, (4, 2)), lw=0.9,
+    # curve sitting at the top is read as learning nothing
-               label=LBL["legit_ref"])
+    ax.axhline(float(load("sec_snr.csv")[0]["chance"]), color=C_CH,
+               ls=":", lw=0.9, label=LBL["chance"])
     ax.set_xlabel("Known-plaintext frames $N$")
-    ax.set_ylabel("Eavesdropper SER")
+    ax.set_ylabel("Outsider SER")
     ax.set_xscale("log", base=2)
     # the 0 dB curve sweeps the upper-right, so anchor the legend at the
     # top edge past the steep drops, above every curve at large N
@@ -526,6 +599,12 @@ def main():
     run_all()
     if PL_CHOSEN:
         PL_FORCED = min(PL_CHOSEN)
+        # 7.4 canvas points is 6.0 on the page at the 0.74-column include
+        # width, which is the floor the standard sets. Falling below it
+        # would mean one figure had dragged every other one down.
+        assert PL_FORCED >= 7.4, (
+            "legend fell to %.1f pt; give the failing figure headroom "
+            "or move a reference entry to its caption" % PL_FORCED)
         print("[uniform] legend size %.1f pt on every figure" % PL_FORCED)
         PL_CHOSEN.clear()
         run_all()

code/report_learned.py

@@ -0,0 +1,106 @@
+# -*- coding: utf-8 -*-
+"""Every learned-family number the manuscript quotes, read from data/.
+
+Switching the learned family from the unpenalized keys to the
+regularized ones of Section V-C moves every learned value in the paper.
+This prints them next to their structured counterparts so the sentences
+that carry them can be updated from one place, and so a later rerun can
+be checked against what is printed.
+
+Run: python code/report_learned.py
+"""
+from __future__ import annotations
+
+import csv
+import json
+from pathlib import Path
+
+DATA = Path(__file__).resolve().parents[1] / "data"
+
+
+def rows(name):
+    with open(DATA / name) as f:
+        return list(csv.DictReader(f))
+
+
+def at(rs, key, val, col):
+    for r in rs:
+        if abs(float(r[key]) - val) < 1e-9:
+            return float(r[col])
+    raise KeyError("%s=%s not in the sweep" % (key, val))
+
+
+def main():
+    print("== Fig. 2, SER against SNR ==")
+    s = rows("sec_snr.csv")
+    l = rows("sec_snr_learned.csv")
+    ratios = []
+    for a, b in zip(s, l):
+        r = float(b["legit"]) / float(a["legit"])
+        ratios.append(r)
+        print("  %5s dB  str %.5f   lrn %.5f   ratio %.3f"
+              % (a["snr_db"], float(a["legit"]), float(b["legit"]), r))
+    print("  ratio range %.3f to %.3f" % (min(ratios), max(ratios)))
+
+    print("\n== Fig. 3, SER against key length ==")
+    s = rows("sec_keylen.csv")
+    l = rows("sec_keylen_learned.csv")
+    for a, b in zip(s, l):
+        print("  L=%-4s str %.5f   lrn %.5f   ratio %.3f   kappa %.5f"
+              % (a["L"], float(a["legit_ser"]), float(b["legit_ser"]),
+                 float(b["legit_ser"]) / float(a["legit_ser"]),
+                 float(b["mask_xcorr"])))
+    print("  learned outsider floor %.5f"
+          % min(float(r["eve_ser"]) for r in l))
+
+    print("\n== Fig. 4, jamming at JSR 0 dB ==")
+    print("  str blind  %.4f" % at(rows("sec_jam.csv"), "jsr_db", 0.0,
+                                   "blind"))
+    print("  lrn blind  %.4f" % at(rows("sec_jam_learned.csv"), "jsr_db",
+                                   0.0, "blind"))
+
+    print("\n== Fig. 6, best of K=1e6 guesses ==")
+    print("  str %.4f" % at(rows("sec_brute_cmp.csv"), "K", 1e6, "ser_mask"))
+    print("  lrn %.4f" % at(rows("sec_brute_learned.csv"), "K", 1e6,
+                            "ser_mask"))
+
+    print("\n== Fig. 7, known plaintext at 10 dB ==")
+    for name in ("kpa.csv", "kpa_learned.csv"):
+        r = [x for x in rows(name) if float(x["snr_db"]) == 10.0]
+        print("  %-16s N=2 %.4f  N=8 %.4f  N=64 %.4f"
+              % (name, at(r, "n_frames", 2, "eve_ser"),
+                 at(r, "n_frames", 8, "eve_ser"),
+                 at(r, "n_frames", 64, "eve_ser")))
+
+    print("\n== Fig. 8, real token streams ==")
+    s = rows("real_sec_ter.csv")
+    l = rows("real_sec_ter_learned.csv")
+    gaps = []
+    for a, b in zip(s, l):
+        for col in ("ter_eve", "ter_insider"):
+            gaps.append(abs(float(a[col]) - float(b[col])))
+        print("  %4s dB  legit str %.5f  lrn %.5f   ratio %.3f"
+              % (a["snr_db"], float(a["ter_legit"]), float(b["ter_legit"]),
+                 float(b["ter_legit"]) / float(a["ter_legit"])))
+    print("  largest adversary gap between families %.2e" % max(gaps))
+    for name in ("real_sec_stats.json", "real_sec_stats_learned.json"):
+        if (DATA / name).exists():
+            d = json.loads((DATA / name).read_text())
+            print("  %-28s %s" % (name, {k: d[k] for k in list(d)[:6]}))
+
+    print("\n== Table IV, scheme comparison ==")
+    for name in ("sec_compare.csv", "compare_learned.csv"):
+        for r in rows(name):
+            print("  %-20s %s" % (name, dict(r)))
+
+    print("\n== Table V, refresh ==")
+    l = rows("refresh_learned.csv")
+    print("  lrn legit  %.5f to %.5f"
+          % (min(float(r["legit_ser"]) for r in l),
+             max(float(r["legit_ser"]) for r in l)))
+    print("  lrn eve    %.5f" % (sum(float(r["eve_ser"]) for r in l)
+                                 / len(l)))
+
+
+if __name__ == "__main__":
+    main()

code/run_audit_fixes.py

@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+"""Rerun the four measurements the audit found mis-specified.
+
+refresh   the learned rows ran 8 blocks while the table says 24
+jamming   the learned curve ran a 5 dB grid inside a 2 dB figure
+compare   the learned row averaged four users inside a user-1 table
+enum      the enumeration attacks ran on the unpenalized learned keys
+"""
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import exp_learned as E
+import check_family_enum as F
+
+if __name__ == "__main__":
+    for fn in (E.refresh, E.jamming, E.compare, F.run):
+        print("=" * 60)
+        print("stage", fn.__module__ + "." + fn.__name__, flush=True)
+        fn()
+    print("[done] audit reruns complete")

code/run_learned_reg.py

@@ -0,0 +1,28 @@
+# -*- coding: utf-8 -*-
+"""Regenerate every learned-key artifact under the regularized loss.
+
+learned_model now trains under the two penalties of Section V-C, so
+every *_learned file has to be rebuilt from it. sens runs before brute
+because brute re-reads the sensitivity sweep it produced.
+"""
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import exp_learned as E
+import exp_full as F
+
+# stage_N is Fig. 2's learned SNR sweep and lives in exp_full, so it is
+# named here rather than in exp_learned's own list
+STAGES = [F.stage_N, E.keylen, E.jamming, E.kpa, E.refresh, E.compare,
+          E.sens, E.brute, E.real]
+
+if __name__ == "__main__":
+    only = sys.argv[1:]
+    for fn in STAGES:
+        if only and fn.__name__ not in only:
+            continue
+        print("=" * 60)
+        print("stage", fn.__name__, flush=True)
+        fn()
+    print("[done] every learned artifact rebuilt")

data/compare_learned.csv

@@ -1,2 +1,2 @@
 scheme,legit_ser,eve_out,eve_in,jam0_ser
-proposed_learned,0.06358416667,0.9998083333,0.9999833333,0.4046066667
+proposed_learned,0.06102,0.999,0.99998,0.3971366667

data/jscc.csv

@@ -0,0 +1,20 @@
+part,case,structured_ser,structured_sd,learned_ser,kappa_and_verdict
+A one user,Walsh row,0.01330,0.00007,,
+A one user,all ones (no mask),0.01325,0.00022,,
+A one user,"learned, free",0.01480,0.00045,,
+B key length,L=6,0.99610,0.00048,0.99722,0.11111/tie
+B key length,L=8,0.94056,0.00686,0.95188,0.00000/tie
+B key length,L=10,0.83318,0.03634,0.80853,0.06667/tie
+B key length,L=12,0.41391,0.00154,0.41663,0.00000/tie
+B key length,L=14,0.39963,0.00779,0.37367,0.04762/learned
+B key length,L=16,0.25810,0.00119,0.28998,0.00000/structured
+B key length,L=18,0.23796,0.00182,0.23525,0.03704/tie
+B key length,L=20,0.18811,0.00113,0.21565,0.00000/structured
+B key length,L=22,0.17300,0.00072,0.18993,0.03030/structured
+B key length,L=24,0.15290,0.00034,0.17582,0.00000/structured
+C load,U=4,0.25810,0.00119,0.28998,0.00000/structured
+C load,U=8,0.95022,0.01381,0.97322,0.00000/tie
+C load,U=12,0.99882,0.00005,0.99867,0.00000/tie
+C load,U=15,0.99976,0.00000,0.99976,0.00000/tie
+C load,U=16,0.99976,0.00001,0.99976,0.00000/tie
+C load,U=20,0.99982,0.00000,0.99981,0.02105/learned

data/kpa_learned.csv

@@ -1,43 +1,43 @@
 snr_db,n_frames,kappa,eve_ser
-0,1,0.2127109103,0.993336
+0,1,0.05034990469,0.997911625
-0,2,0.7559393242,0.666508
+0,2,0.05894429332,0.662401
-0,3,0.8627683729,0.392272125
+0,3,0.08379636761,0.37693875
-0,4,0.9179756209,0.218212875
+0,4,0.107222241,0.22233425
-0,5,0.945390512,0.14425925
+0,5,0.1292011227,0.152770125
-0,6,0.9590921029,0.107493
+0,6,0.1429406652,0.131741625
-0,8,0.9725584686,0.086112625
+0,8,0.1788643047,0.086013875
-0,10,0.9778045967,0.07933375
+0,10,0.2027079877,0.077310875
-0,12,0.9831736788,0.07420325
+0,12,0.2198284302,0.07391475
-0,16,0.9879393309,0.070712875
+0,16,0.2579282403,0.069537375
-0,24,0.9925390184,0.067800375
+0,24,0.3094305053,0.066363625
-0,32,0.9945808738,0.06649325
+0,32,0.3459532984,0.06496575
-0,48,0.9965663388,0.065265
+0,48,0.4114221439,0.063700375
-0,64,0.9975094497,0.065024875
+0,64,0.4645803586,0.06309375
-10,1,0.3165432975,0.948993875
+10,1,0.05442041112,0.950243625
-10,2,0.9290210679,0.19776825
+10,2,0.1407177548,0.181911125
-10,3,0.9781143948,0.095876875
+10,3,0.2204171771,0.080573875
-10,4,0.9896475986,0.070495125
+10,4,0.2837046772,0.072395125
-10,5,0.9934410676,0.067297125
+10,5,0.3326446027,0.06649075
-10,6,0.9953705788,0.06628575
+10,6,0.3704789877,0.0648705
-10,8,0.9971551418,0.06520175
+10,8,0.4370021872,0.063752375
-10,10,0.9979188025,0.064684875
+10,10,0.479882317,0.062964
-10,12,0.9982561454,0.06454375
+10,12,0.5199262805,0.062786125
-10,16,0.9987509355,0.06426625
+10,16,0.5935128644,0.062229125
-10,24,0.9992754847,0.064091125
+10,24,0.6796541184,0.061876875
-10,32,0.9994516179,0.06386675
+10,32,0.7379264548,0.0618305
-10,48,0.9996520028,0.063819875
+10,48,0.8053073436,0.061516125
-10,64,0.999746412,0.063847125
+10,64,0.8465657607,0.061403
-20,1,0.742194891,0.513137625
+20,1,0.06401134632,0.757281
-20,2,0.9947786465,0.06753025
+20,2,0.3653038088,0.06925225
-20,3,0.9986294076,0.06438075
+20,3,0.5262307867,0.06323275
-20,4,0.9992210969,0.064160375
+20,4,0.6446151808,0.06203725
-20,5,0.9994008377,0.064080125
+20,5,0.7232914954,0.06185625
-20,6,0.9995701849,0.063900375
+20,6,0.7802997425,0.06165275
-20,8,0.9997365534,0.063852875
+20,8,0.8357323289,0.061493375
-20,10,0.9998067141,0.063813
+20,10,0.8705290645,0.061302375
-20,12,0.9998438716,0.06370225
+20,12,0.8942843586,0.061368125
-20,16,0.9998808399,0.063670875
+20,16,0.9240833685,0.06133
-20,24,0.9999239221,0.06380125
+20,24,0.9489109725,0.061323375
-20,32,0.9999452353,0.063820625
+20,32,0.9614683628,0.06120425
-20,48,0.9999649763,0.063811
+20,48,0.97474062,0.061322875
-20,64,0.9999733046,0.06377525
+20,64,0.9812480465,0.06146825

data/real_sec_stats_learned.json

@@ -0,0 +1,32 @@
+{
+ "vocab_size": 30522,
+ "n_texts": 2000,
+ "frames": 24998,
+ "repeats": 8,
+ "decisions_per_point": 799936,
+ "distinct_tokens": 10486,
+ "token_collision": 0.006535221793661566,
+ "max_token_id": 29599,
+ "headlines_scored": 1948,
+ "headline_runs": 4,
+ "recovery": {
+  "20": {
+   "legit": 0.7183008213552361,
+   "eve": 0.0,
+   "insider": 0.0,
+   "oma": 0.6463039014373717
+  },
+  "24": {
+   "legit": 0.8805184804928131,
+   "eve": 0.0,
+   "insider": 0.0,
+   "oma": 0.8390657084188912
+  },
+  "28": {
+   "legit": 0.946611909650924,
+   "eve": 0.0,
+   "insider": 0.0,
+   "oma": 0.9319815195071869
+  }
+ }
+}

data/real_sec_ter_learned.csv

@@ -0,0 +1,9 @@
+snr_db,ter_legit,ter_eve,ter_insider,ter_oma
+0,0.4424778982,0.9998887411,0.9961971958,0.5232856128
+4,0.2185937375,0.9998887411,0.9948020842,0.2740881771
+8,0.09510385831,0.9998837407,0.9940082707,0.123424874
+12,0.03968067445,0.9998599888,0.9936469918,0.05201666133
+16,0.0160362829,0.999899992,0.9935182315,0.02118419474
+20,0.006499269942,0.999887491,0.9934544764,0.008630690455
+24,0.002587707017,0.9998849908,0.9934232239,0.003452776222
+28,0.001065085207,0.999887491,0.9934157233,0.001385110809

data/refresh_learned.csv

@@ -1,9 +1,25 @@
 block,legit_ser,eve_ser
-0,0.06381666667,0.9999375
+0,0.06113666667,0.9990225
-1,0.06395583333,0.9981941667
+1,0.06151666667,0.99906
-2,0.06397833333,0.999985
+2,0.06108666667,0.9999341667
-3,0.06386833333,0.9983758333
+3,0.0615975,0.9995841667
-4,0.06338666667,0.9999908333
+4,0.06189833333,0.9996233333
-5,0.06366416667,0.9999133333
+5,0.0608925,0.9997391667
-6,0.06398416667,0.9999433333
+6,0.06110333333,0.9999591667
-7,0.06390583333,0.9809091667
+7,0.06100083333,0.994235
+8,0.06136166667,0.9995625
+9,0.06109166667,0.999875
+10,0.06130416667,0.9996441667
+11,0.06130916667,0.9975941667
+12,0.06105666667,0.9999325
+13,0.06113166667,0.9998916667
+14,0.06178,0.9994783333
+15,0.06124583333,0.9997775
+16,0.06141083333,0.9998683333
+17,0.0612175,0.99964
+18,0.06126833333,0.9997041667
+19,0.06113916667,0.9999766667
+20,0.06117583333,0.9994191667
+21,0.06127166667,0.9999558333
+22,0.06118416667,0.9995833333
+23,0.06149666667,0.9997783333

data/refresh_summary.csv

@@ -1,5 +1,5 @@
 scheme,legit,eve,entropy_bits
+"Invariant, KM (str.)",0.05304121528,0.9995528819,364.5801064
+"Invariant, KM (lrn.)",0.061279,0.998895,364.5801064
 None (fixed key),0.05300666667,0.9997075,23.76910417
 Fresh orthogonal keys,0.1215548611,0.9996535069,23.76910417
-Invariant,0.05304121528,0.9995528819,364.5801064
-"Invariant, learned keys",0.063800,0.997200,364.5801064

data/sec_brute_learned.csv

@@ -0,0 +1,15 @@
+K,ser_mask
+1,0.9967152519
+3,0.9933139604
+10,0.9866396597
+30,0.976371298
+100,0.9609886779
+300,0.9443205866
+1000,0.92371647
+3000,0.8849669152
+10000,0.8426466786
+30000,0.8076524286
+65536,0.7837337919
+100000,0.7731184695
+300000,0.7358238662
+1000000,0.7103847689

data/sec_compare.csv

@@ -4,4 +4,4 @@ public_mask,0.0529425,0.0529425,0.0529425,0.83882
 perm_key,0.0528525,0.9999925,0.0528525,0.36425
 index_cipher,0.0529425,0.9999847412,0.9999847412,0.83882
 oma_plain,0.08056383667,0.08056383667,0.08056383667,nan
-proposed_learned,0.06358416667,0.9998083333,0.9999833333,0.4046066667
+proposed_learned,0.06156083333,0.9996875,0.9999841667,0.3973533333

data/sec_jam_learned.csv

@@ -1,8 +1,17 @@
 jsr_db,blind,matched,nojam
--10,0.117582,0.40832,0.062852
+-10,0.114422,0.38569,0.060494
--5,0.215158,0.657572,0.062852
+-8,0.143006,0.483718,0.060494
-0,0.404244,0.851282,0.062852
+-6,0.183328,0.586128,0.060494
-5,0.648786,0.94662,0.062852
+-4,0.239698,0.685262,0.060494
-10,0.839218,0.982652,0.062852
+-2,0.310968,0.772466,0.060494
-15,0.939304,0.994184,0.062852
+0,0.398118,0.840574,0.060494
-20,0.979206,0.99818,0.062852
+2,0.493594,0.893422,0.060494
+4,0.594688,0.929018,0.060494
+6,0.686344,0.953816,0.060494
+8,0.768698,0.970236,0.060494
+10,0.835132,0.980904,0.060494
+12,0.886322,0.98835,0.060494
+14,0.923646,0.992284,0.060494
+16,0.949318,0.995096,0.060494
+18,0.967192,0.996858,0.060494
+20,0.978764,0.998038,0.060494

data/sec_keylen_learned.csv

@@ -1,9 +1,9 @@
 L,d,legit_ser,eve_ser,mask_xcorr,oma
-8,32,0.9297855,0.9998735,0.007307400461,0.6849191155
+8,32,0.9594105,0.99997,0.003249221947,0.6849191155
-12,48,0.416604,0.9997065,0.005153660662,nan
+12,48,0.4157085,0.999767,0.003225991037,nan
-16,64,0.2762895,0.999912,0.007116591092,0.2747696909
+16,64,0.315952,0.9994685,0.01121100038,0.2747696909
-20,80,0.2076175,0.999383,0.003162040841,0.2289444229
+20,80,0.213059,0.9996415,0.00134725438,0.2289444229
-24,96,0.1829615,0.999894,0.002973971656,0.1961714033
+24,96,0.183031,0.9997345,0.001526024193,0.1961714033
-32,128,0.131901,0.999616,0.005575809628,0.1524639978
+32,128,0.1231225,0.999805,0.002514706925,0.1524639978
-48,192,0.090206,0.9994575,0.005743456539,0.1054308944
+48,192,0.084432,0.999786,0.001822981867,0.1054308944
-64,256,0.0635265,0.999637,0.006678360514,0.08056383667
+64,256,0.061412,0.9993955,0.002381352475,0.08056383667

data/sec_keylen_perm.csv

@@ -0,0 +1,9 @@
+L,d,legit_ser,eve_ser
+8,32,0.948556,0.999984
+12,48,0.414228,0.99995
+16,64,0.258384,0.999992
+20,80,0.187222,0.999986
+24,96,0.152004,0.999972
+32,128,0.10816,0.999976
+48,192,0.071734,0.999966
+64,256,0.053092,0.999976

data/sec_sens_learned.csv

@@ -0,0 +1,14 @@
+frac,ser_mask
+0,0.9999529167
+0.2,0.99770125
+0.4,0.9489816667
+0.6,0.6733195833
+0.75,0.29680625
+0.85,0.1140279167
+0.9,0.08714708333
+0.92,0.07980791667
+0.94,0.07278666667
+0.955,0.06856208333
+0.97,0.06626333333
+0.985,0.06358166667
+1,0.06154625

data/sec_snr_learned.csv

@@ -1,12 +1,12 @@
 snr_db,legit,eve_wrong
-0,0.45205625,0.999865625
+0,0.439821875,0.999816875
-2,0.325615625,0.9998425
+2,0.31516125,0.9997828125
-4,0.224483125,0.9998196875
+4,0.217330625,0.9997790625
-6,0.150013125,0.999801875
+6,0.1449359375,0.9997196875
-8,0.0982703125,0.9997996875
+8,0.0947978125,0.99971375
-10,0.06379375,0.9997821875
+10,0.0611025,0.999680625
-12,0.0409540625,0.999758125
+12,0.039268125,0.9996478125
-14,0.0259790625,0.9997684375
+14,0.0251025,0.999646875
-16,0.0165590625,0.999769375
+16,0.0159153125,0.999625625
-18,0.01049375,0.999760625
+18,0.0100140625,0.9996171875
-20,0.006573125,0.9997428125
+20,0.0063884375,0.999608125

data/verify_math.csv

@@ -15,4 +15,4 @@ V7 symbolic identities,exact,exact,0,0,PASS
 V8 cross-period remainder,0.0,0.000337,0.000337,0.0005,PASS
 V9 score-variance ratio,2.8,2.8252,0.0252,0.05,PASS
 V10 format-matched OMA at 10 dB,0.055,0.05520,0.00020,0.001,PASS
-V11 OMA closed form vs Monte Carlo,0.081118,0.080925,0.0024,0.01,PASS
+V11 OMA closed form vs Monte Carlo,0.081245,0.080925,0.0039,0.01,PASS

data/whygap.csv

@@ -0,0 +1,4 @@
+family,legit_ser,kappa_bar,entry_use
+structured (frozen),0.05259,0.00000,1.0000
+learned (free start),0.06363,0.00444,0.0916
+learned (Walsh start),0.06489,0.00478,0.2797