diff --git a/README.md b/README.md index 9eefb85..39794a0 100755 --- a/README.md +++ b/README.md @@ -29,6 +29,8 @@ Cross-Attention" (submitted to IEEE Transactions on Wireless Communications). study, attention maps, beta sweep, real-data study). - `fig/` — figure PDFs as included in the manuscript. +**Detailed per-experiment documentation (setup, procedure, metrics, key results, reviewer-concern map): [docs/EXPERIMENTS.md](docs/EXPERIMENTS.md).** + ## Figure/number → script → data map | Manuscript item | Script | Data | diff --git a/docs/EXPERIMENTS.md b/docs/EXPERIMENTS.md new file mode 100755 index 0000000..3268aa0 --- /dev/null +++ b/docs/EXPERIMENTS.md @@ -0,0 +1,246 @@ +# Experiment Documentation + +Detailed description of every experiment behind the manuscript +"Relevance-Aware Semantic Multiple Access via Meta-Learned User-Wise +Cross-Attention." Each entry states the reviewer concern it addresses, the +exact setup, the procedure, the metrics, the key results, and the artifact +paths. All experiments run on a single NVIDIA RTX A4500 (PyTorch 2.10, CUDA +12.8), seed 42, and write raw JSON results consumed only by +`rev2/plot_rev2.py`. + +**Common system model** (all rev2 experiments, matching manuscript Eq. (7)): +single superimposed uplink y = sum_v |h_v| e^{j dphi_v} (e_v (.) m_v) + n over +one d=64-dimensional frame; fixed {0,1} disjoint transmit block masks +(d/U coordinates per user, identical for every compared scheme); per-rail +noise std = sqrt(mean|y_tx|^2 / SNR); U=4 users, HIGH/LOW/MIX scenarios of +Table II unless stated. The decoder-side masks are separate learned soft +masks in [0,1]^d initialized at the block pattern. Training is the +manuscript's first-order meta-training aggregated over SNR tasks +{0,4,...,20} dB, Adam 1e-3 (mask logits 0.1), 250-300 epochs, batch 64; +evaluation uses 150-200 Monte Carlo batches of 64 per point. + +--- + +## E1 — Comparison fairness against optimal linear receivers +*Reviewer concerns: R1-10, R3-7 (d/U comparison "mathematically unfair"); +editor point 4.* + +- **Question.** Is the OFDMA/SFDMA subspace ceiling an artifact of weak + baselines, and how much of the UWCA gain is mere dimensionality? +- **Setup.** For each scenario the true relevance matrix B (B_uv = + beta_u beta_v for shared scene, else 0) and the per-realization channel + magnitudes are formed. Receivers evaluated on the *same* received frame: + `uwca` (trained), `ofdma`, `sfdma`, `noma` (full-band power-domain SIC, + powers 0.40/0.30/0.20/0.10), `lmmse_blind` (closed-form Wiener with + cross-covariances zeroed), `lmmse_genie` (closed form with true B and + channel gains, manuscript Eq. (23)), `tdma_proj` (random orthonormal + 16-dim projection per user — an arbitrary orthogonal partition). +- **Key results.** lmmse_blind = OFDMA at every SNR in all scenarios + (SER 0.509 vs 0.508 @10 dB HIGH), tdma_proj and SFDMA coincide with them; + HIGH @20 dB: blind 0.327 -> UWCA 0.110 -> genie 0.044 (UWCA recovers 77% + of the blind-to-genie gap); LOW: genie = blind (nothing to exploit), + UWCA within 0.02 SER. +- **Artifacts.** `rev2/e1_fair_baselines.py` -> + `rev2/data/e1_fair_baselines.json` -> `fig/fig_fair.pdf` (manuscript + Fig. 3). Trained checkpoints `e1_uwca_{HIGH,LOW,MIX}.pt` (reused by E4/E6). + +## E2 — Full complex-baseband phase errors and CSI error +*Reviewer concerns: R1-1 (phase model underestimates IUI), R2-3 (CSI).* + +- **Question.** Does the multi-user superposition amplify residual phase + errors into severe inter-user interference once nothing is absorbed into a + noise term? +- **Setup.** Complex channel with per-user residual dphi_u ~ N(0, sigma^2), + sigma in {0,5,10,15,20,30} deg; both rails simulated so every leakage path + exists. Three decoders: mismatch-trained (never saw phase errors), + phase-augmented (trained with sigma ~ U[0,20] deg), and a two-rail variant + whose keys/values read [Re;Im] (2d input). CSI sweep: amplitude error + h_hat = h(1+eps), eps ~ N(0, sigma_h^2), sigma_h up to 0.2, at 10 deg. + Learned-mask overlap (mean pairwise cosine) is also measured. +- **Key results.** Mismatch-trained @10 dB: SER 0.270 -> 0.291 (20 deg) -> + 0.332 (30 deg); @20 dB 0.110 -> 0.127; augmentation and I/Q reading change + curves by less than Monte Carlo spread -> leakage is second order because + it scales with sin(dphi) x mask overlap (measured 0.26). CSI: UWCA + unchanged within 0.004 (uses no explicit CSI). +- **Artifacts.** `rev2/e2_phase_iui.py` -> `rev2/data/e2_phase_iui.json`; + `fig/fig_phase2.pdf` (repo; numbers narrated in manuscript Sec. VII-E). + +## E3 — Dynamic user population +*Reviewer concerns: R1-6 (fixed U, costly retraining), R2-3 (fixed +identities).* + +- **Setup.** U_max = 8 mask slots (d=64, 8 dims/slot), HIGH-type correlation + (beta_u = 0.6, one scene). Active set resampled per frame; inactive users + transmit nothing and their softmax scores are masked to -inf (the + scheduler announces the active set). Compared: one model trained with the + full population always active (never retrained), an activity-sampled + model, and per-count oracle models retrained from scratch for + |A| in {2,4,6,8}. Metric: mean cosine (the sqrt(1/8)=0.35 subspace + ceiling saturates the binary SER at this scale). +- **Key results.** The single full-population model sustains fidelity over + every active count (c 0.34 -> 0.39 for |A| = 2 -> 8 @10 dB) and matches or + exceeds the per-count retrained models (0.33/0.32/0.36/0.39); per-count + retraining is counterproductive (sparse populations train each slot on a + fraction of the traffic). +- **Artifacts.** `rev2/e3_dynamic_users.py` -> + `rev2/data/e3_dynamic_users.json`; `fig/fig_dynusers.pdf` (repo). + +## E4 — Symbol-timing offsets (v3 = block-wise realignment) +*Reviewer concern: R1-8 (asynchronous reception / ISI).* + +- **Setup.** Per-user integer offsets delta_u ~ U{0..Delta} symbols, + Delta in {0,1,2,4,8}, shift each user's transmitted block within the + frame (edge energy lost). Conditions: uncorrected (UWCA and OFDMA), + corrected by block-wise realignment using pilot-estimated offsets + (each user's block region shifted back individually), and corrected with + a deliberately impaired estimator (+-1 symbol on 20% of users). + v1 (`e4_async` results in json) showed offset-augmented *training* cannot + repair unknown shifts; v2 showed whole-frame realignment breaks + cross-block alignment — both superseded by v3. +- **Key results.** Uncorrected offsets are catastrophic for *every* + embedding-level scheme (one symbol: UWCA 0.26 -> 0.73, OFDMA 0.51 -> + 0.76) because i.i.d. embedding coordinates fully decorrelate under a + one-symbol misalignment. With realignment the degradation is gradual + (0.266 -> 0.304 @Delta=1, 0.455 @Delta=8) and realigned UWCA stays below + realigned OFDMA (0.536-0.638) at every offset. The impaired estimator + costs 0.14 SER — whole-symbol residuals sacrifice the affected block, so + timing must be sub-symbol (standard timing advance). +- **Artifacts.** `rev2/e4_v3_async.py` -> `rev2/data/e4_v3_async.json`; + `fig/fig_async.pdf` (repo). + +## E5 — Nonlinear inter-user semantic structure +*Reviewer concerns: R1-7, R2-5, R3-2 (linear scalar model too restrictive).* + +- **Setup.** Embeddings e_u = normalize(g_u([s; p_u])) with fixed random + two-layer tanh view networks g_u per user (seed 7): users share the scene + s only through independent nonlinear transformations. Cases: shared scene + vs independent scenes (control). Schemes: trained UWCA, OFDMA, NOMA-SIC, + and the scalar-parameterized genie LMMSE fed the *measured* mean pairwise + cosine. +- **Key results.** Linear correlation is destroyed (mean cosine 0.006), so + the genie LMMSE collapses onto the ceiling (0.515 vs OFDMA 0.518 @10 dB) + — no scalar/linear receiver can represent the shared structure. UWCA + still attains 0.363 @10 dB / 0.220 @20 dB. The independent-scene control + (UWCA 0.441) isolates the manifold-prior share, so the further reduction + to 0.363 is pure nonlinear cross-user structure. +- **Artifacts.** `rev2/e5_nonlinear.py` -> `rev2/data/e5_nonlinear.json` + (numbers narrated in manuscript Sec. VII-G). + +## E6 — Residual orthogonality vs content preservation +*Reviewer concerns: R1-4, R2-2, R3-3 (semantic-orthogonality +self-contradiction).* + +- **Setup.** E1's trained HIGH decoder; per-sample Pearson correlation + across the 64 dimensions, averaged over user pairs and 100x64 samples per + SNR, for: input embeddings, decoded embeddings, decoding residuals + r_u = e_hat_u - e_u; OFDMA decoded correlation as reference. +- **Key results.** Decoded-embedding correlation rises with SNR from 0.31 + toward the 0.39 input level (shared content preserved, not stripped); + residual correlation falls 0.23 -> 0.14 (2.8x below input) — the + emergent residual orthogonality. OFDMA's decoded correlation is 0.00 at + every SNR: orthogonal access erases the inter-user semantic structure. +- **Artifacts.** `rev2/e6_residual_orth.py` -> + `rev2/data/e6_residual_orth.json` -> `fig/fig_resorth.pdf` (manuscript + Fig. 4). + +## E7 — Meta-adaptation beyond the SNR axis (v2) +*Reviewer concerns: R1-3, R3-4, R3-5 (MAML overkill for a 1-D lookup); +R1-9 (eta stability).* + +- **Setup.** Task family = 6 SNRs x {Rayleigh, Rician K=5 dB, K=10 dB} x + phase residual {0,10} deg (36 tasks). Systems: one initialization + meta-trained over the full family (300 epochs, eta and outer-gradient + norm logged every epoch); per-SNR specialist bank trained on + Rayleigh/no-phase (the 1-D lookup, nearest-SNR index). Held-out tests: + Rician K=20 dB + 15 deg (10/18 dB), Nakagami m=3 + 5 deg (a fading law in + no training task), Rayleigh + 20 deg @6 dB, plus an in-distribution + check; zero-shot and after S in {1,5,10,20} inner steps (SGD 0.02, one + 64-sample support batch). +- **Key results.** Family initialization transfers zero-shot with 9-46% + lower SER than the lookup on every held-out condition (e.g. 0.114 vs + 0.158 Rician-K20-15deg @10 dB; 0.139 vs 0.190 unseen Nakagami); inner-loop + adaptation is stable (within 0.005 over 20 steps). eta converges to 0.94 + (max 1.00), outer gradient norm <= 0.024 vs clip bound 5 — no softmax + saturation or divergence. +- **Artifacts.** `rev2/e7_v2_meta.py` -> `rev2/data/e7_v2_meta.json` + (includes eta/gradient trajectories). + +## E8 — End-to-end training with anti-collapse (v2) +*Reviewer concern: R1-5 (representation collapse prevents joint JSCC).* + +- **Setup.** Four configurations on HIGH: frozen encoder (reference); + naive end-to-end with the distortion measured against the encoder's own + output (the collapse-prone moving target); source-anchored end-to-end + (distortion vs normalize(x)); moving target + VICReg-style + variance-covariance regularizer (hinge at 1/sqrt(d) per-dim std, 25x + variance + 100/d covariance weights). Collapse-proof metric: batch + nearest-neighbor retrieval accuracy over the *encoded* gallery, plus the + effective rank of the encoder-output covariance. +- **Key results.** Naive collapses exactly as the reviewers expect: + retrieval falls to chance (0.005 vs 0.536 frozen @10 dB) — the pathology + is the moving-target objective, not the cross-attention. Source anchoring + keeps effective rank 59.6/64 and lands within 0.04 SER of frozen; VICReg + restores retrieval to 0.509 with a learned task-specific code. E2E + training is therefore demonstrated, and the frozen encoder in the main + experiments is a controlled-isolation choice. +- **Artifacts.** `rev2/e8_v2_e2e.py` -> `rev2/data/e8_v2_e2e.json`. + +## E9 — Online top-k relevance acquisition and measured overhead +*Reviewer concerns: R1-2, R3-6 (circular dependency), R2-4 (cost at large +populations).* + +- **Setup.** U=32, eight relevance clusters of four, k=4, briefly trained + full-attention model. Protocol: frames 1-3 full attention (needs no + relevance) while the BS accumulates beta_hat by EWMA (zeta=0.5) of + decoded-embedding cosines; from frame 4, per-row top-k via argpartition + on beta_hat. References: oracle top-k (peers selected from the true + clusters) and permanent full attention. Timing: decoder forward vs + EWMA-update + selection wall-clock for U in {8,16,32,64,128} + (256-sample frames, GPU, 20-run averages). +- **Key results.** From the first post-warm-up frame the online selection + matches the oracle exactly and slightly exceeds full attention + (discarding irrelevant peers discards their noise); ranking needs far + less accuracy than estimation because intra-cluster (~0.42) and + inter-cluster (~0) cosines are well separated. Estimation + selection + cost 0.02-0.04 ms vs the 0.3-5.4 ms decoder pass, incurred once per + relevance coherence interval. +- **Artifacts.** `rev2/e9_topk_online.py` -> + `rev2/data/e9_topk_online.json` (trajectory + timing). + +--- + +## Legacy experiments (retained from the original study) + +- **Three-scenario SER study** (manuscript Fig. 2): analytical simulation + `legacy/semantic_correlation_sim.py` + trained decoder-only overlay + `legacy/maml_semantic.py --decoder_only`; threshold sweep tau in + [0.30,0.50] and mean-cosine table in `legacy/revision_experiments.py`. +- **beta sweep** (monotone gain law): `legacy/revision_betasweep.py` + (figure `fig/fig4_beta_sweep.pdf`, narrated in Sec. VII-C). +- **Attention heatmaps**: `legacy/plot_figures.py` + (figure `fig/fig6_hlm.pdf`, narrated in Sec. VII-C). +- **Hyperparameter ablations** (S, lambda, d, H, K): + `legacy/revision_ablation.py`, `legacy/revision_dsweep.py`, + `legacy/revision_e2e.py` (Sec. VII-D). +- **Real-data study** (manuscript Fig. 5): UCI optical-recognition digits + (8x8=64 dims), `legacy/revision_realdata_train.py` + + `legacy/revision_realdata_plot.py` (Sec. VII-G). + +## Reviewer-concern -> experiment map + +| Concern | Experiment(s) | Manuscript | +|---|---|---| +| R1-1 phase IUI | E2 | Sec. III-A, VII-E | +| R1-2 / R3-6 top-k circularity | E9 | Sec. V-E, VII-H | +| R1-3 / R3-4 / R3-5 MAML vs lookup | E7 | Sec. V-E, VII-F | +| R1-4 / R2-2 / R3-3 orthogonality contradiction | E6 | Def. 3, Prop. 4, Fig. 4 | +| R1-5 representation collapse | E8 | Sec. VI-B | +| R1-6 dynamic U | E3 | Sec. IV-B, VII-E | +| R1-7 / R2-5 / R3-2 linear scalar model | E5 (+ real data) | Sec. III-B, VII-G | +| R1-8 asynchrony | E4 | Sec. III-A, VII-E | +| R1-9 eta stability | E7 logs | Sec. IV-A, VII-F | +| R1-10 / R3-7 d/U fairness | E1 | Prop. 3, Fig. 3, VII-B | +| R2-1 idealized proofs | assumption block + E1/E5/real data | Sec. V | +| R2-3 sync / CSI / identities / encoder | E4 / E2 / E3 / E8+E5 | Sec. III, VI-B, VII-E | +| R2-4 large-population cost | E9 timing | Sec. VII-H | +| R3-1 DSC novelty | (positioning) | Sec. II-A |