Role
Machine Learning (Research) Engineer · advised by Prof. Saeid Tizpaz-Niari
Timeline
Jun 2026 — Present
Year
2026
Category
Research · AI/ML
Survey notes
A metamorphic robustness benchmark measuring whether tax-question-answering LLMs stay correct when tax-irrelevant details change. 130 CPA-labeled VITA Form 6744 questions expand to 2,080 cases across four metamorphic relations (demographic name swaps, irrelevant context, paraphrase, sycophantic pushback), evaluated across three grounding regimes: base, RAG over IRS Pub 4491, and neurosymbolic (PAL) code execution. The headline metric, Conditional Violation Rate, isolates robustness from raw accuracy: among cases a model got right, how often does a tax-irrelevant change flip it wrong? Thesis: neurosymbolic grounding is the most robust, since only cleaned, typed inputs ever reach the executor.
Notable terrain
- 01Four metamorphic relations, each a (generator, validity predicate, oracle) triple formally specified in MR_SPEC
- 02Three grounding regimes: base/NoRAG, dense-retrieval RAG with per-request chunk-ID logging, sandboxed PAL execution
- 03Conditional Violation Rate isolates robustness from accuracy; a TaxMR score ranks every (model, regime) pair
- 04Content-addressable cache and crash-safe streamed logging, so no case is ever recomputed
- 05Cluster-bootstrap confidence intervals and exact McNemar significance tests
- 06Phase 1 complete: full pipeline validated end-to-end on Qwen 2.5 7B; Phase 2 scales unchanged to frontier models
Next sheet
FairLint DL