🪐 EDM-ARS · v5 · Open Source
A multi-agent LLM pipeline that turns a dataset and a research prompt into a complete, reviewer-ready academic paper — across prediction, causal inference and psychometrics — then scores it against thresholds calibrated on papers those venues actually published.
Overview
Inspired by FARS, EDM-ARS is an open-source, domain-specific multi-agent LLM pipeline that automates the complete workflow of educational data mining research. Given a curated dataset and a research prompt, it formulates a research question, engineers the analytic sample, runs the analysis, retrieves real citations via the Semantic Scholar API with three-layer verification (exact title match, Jaccard similarity, CrossRef cross-validation), and produces a complete LaTeX manuscript in ACM sigconf or APA-7 journal format — with a built-in Critic agent that enforces methodological rigor through automated peer review and targeted revision loops.
v5 covers five study types: predictive modeling, three causal-inference designs (selection-on-observables, individualized treatment regimes, difference-in-differences), and psychometrics (IRT, measurement invariance, DIF, cognitive diagnosis). Ten estimation methods are certified against synthetic data with known ground truth before they are allowed near a real dataset. Four datasets are curated: HSLS:09, ELS:2002, ASSISTments 2009–10, and a constructed ELS×HSLS cross-cohort panel.
Each dataset carries a three-tier variable registry: hand-curated Tier 1 variables with educational annotations, auto-generated Tier 2 variables, and Tier 3 exclusions (weights, IDs) enforced programmatically. Survey-design realities — school clustering, sentinel missing codes, legitimate skips, suppressed identifiers — are encoded per dataset rather than left for the model to infer.
70 composable skills across four layers — task type, dataset, methodology and writing — are matched at runtime and injected into each agent's prompt. Adding a capability means adding a skill, not enlarging a prompt. Skills marked mandatory bypass the size cap, because a rule whose violation corrupts output silently must always reach the agent.
Every estimator faces a synthetic-data gate before it touches real data: a data-generating process with a known true effect, and the method must recover it. Difference-in-differences was held back until it hit 0.006 bias and detected pre-trend violations at a rate of 1.0.
Prompting a model not to invent numbers does not stop it. Every figure in the manuscript is reconciled against the analysis artifacts, unverifiable claims are flagged, and a paper whose numbers cannot be traced is blocked from passing the gate — regardless of how well it reads.
Architecture
test_protected.csv with pre-encoding subgroup labels for fairness analysisRevision loop — on REVISE, targeted instructions are routed back to ProblemFormulator, DataEngineer, Analyst, or OutlineAgent selectively. Up to 2 cycles before the Writer is unblocked regardless.
Features
6 Specialized Agents
Coordinated by a state-machine orchestrator. Each agent has its own system prompt, temperature and model routing, configurable per stage. The reference configuration runs the whole pipeline on DeepSeek-v4-pro; Anthropic, OpenAI and MiniMax backends are supported through the same interface.
End-to-End Automation
From a raw CSV and a research prompt to a compiled ACM LaTeX paper — with real citations, methodology validation, and SHAP explainability figures.
Self-Healing Pipeline
Contract validation at every stage boundary. Auto-patching for classifiable errors (SHAP failure, dtype mismatch, missing column) before falling back to LLM repair.
Live Academic Citations
The ProblemFormulator queries the Semantic Scholar API with exponential-backoff retry logic to retrieve and validate real, current citations.
6-Model Battery
Logistic Regression, Random Forest, XGBoost, ElasticNet, MLP, and a Stacking Ensemble are trained, compared, and reported with SHAP explainability. A Model Quality Gate (AUC ≥ 0.60 for classification, R² ≥ 0.05 for regression) runs before SHAP, so a model that has learned nothing is not given an interpretation.
Docker Sandboxing
LLM-generated analysis code executes inside a Docker sandbox (network-disabled). Gracefully falls back to subprocess when Docker is unavailable.
Causal Inference New
Selection-on-observables (matching, IPW, regression adjustment, AIPW/TMLE), causal forests for heterogeneous effects, individualized treatment regimes, and difference-in-differences. Estimands are declared before estimation; balance, overlap and unmeasured-confounding sensitivity are reported as a matter of course.
Psychometrics New
Item response theory, graded response models, measurement invariance testing, differential item functioning, and cognitive diagnosis (DINA / G-DINA) — executed in R through a bridge, with degeneracy reported rather than hidden.
Numeric Reconciliation New
Every number in the manuscript is matched back to the analysis outputs that produced it. An audit once found a paper that scored well while reporting fabricated values for fields the analysis left empty; that class of failure is now blocked deterministically rather than discouraged by prompt.
Calibrated Review Gate New
Papers are scored by LSAR against the 25th percentile of papers the target venue actually published — not an arbitrary threshold. Borderline verdicts trigger median-of-three sampling, because a single review carries real measurement noise.
Measured Token Accounting New
Every run records prompt, completion and cached-input tokens per call. Rates live in configuration, so a price change re-prices past runs without re-running them, and an unpriced model yields null rather than a silent zero.
Measured Results
Fifteen complete papers have been generated and scored through the calibrated review gate. Every study type has at least one paper that clears its venue's threshold. Scores span 3.7 to 7.5 — the low end is reported here for the same reason it is reported in the papers themselves.
That cost is measured, not estimated: one fully instrumented run made 84 LLM calls — $0.091 for the six-agent pipeline and $0.055 for six sampled LSAR reviews. 81% of input tokens were prompt-cache hits, billed roughly ten times lower, which is the dominant lever and one that summed token counts cannot see. An earlier figure on this page quoted several dollars per paper; it came from a hardcoded rate constant rather than a measurement, and it was wrong by two orders of magnitude.
Companion Tool · v0.4
Inspired by PaperReview.ai, LSAR is an automated, agentic paper reviewer for learning science venues. A six-stage pipeline ingests a PDF, identifies the venue, retrieves related work, and produces a structured review across 8 quality dimensions for 12 venues — four conferences and eight journals.
Alignment with venue scope and topic fit
Originality and contribution beyond prior work
Strength of conceptual framework and motivation
Soundness of research design and analysis
Quality and interpretation of evidence
Impact on the field and practical implications
Ethical considerations and bias awareness
Clarity, structure, and readability of writing
Calibrated thresholds. A raw review score means little on its own — nothing anchors “6.5” to any real standard. LSAR reviews papers a venue actually published and takes the 25th percentile of their scores. A submission below that sits below the weakest quartile of what the venue accepted. Current gates: EDM 6.3 (15 anchors), AERA Open 6.6 (11), JEDM 5.15 (10), JLA 5.4 (9).
And what those numbers cannot do. Measured test–retest deviation is 1.9 points, so a single review is a noisy instrument and borderline cases are sampled three times and taken at the median. Anchor corpora run 9–15 papers per venue: these are working thresholds, not population parameters. A gate is a triage signal for authors, never an acceptance decision.
Venue Detection — automatically classifies papers for their intended conference or allows manual specification
Batch Processing — review multiple papers in a single run from directories or CSV manifests
Multiple Formats — generates reviews in Markdown, JSON, and PDF reports with modular pipeline stages
Review Health Checks — a truncated review is rejected rather than scored. 21% of reviews were once cut off between Strengths and Weaknesses, keeping the praise and losing the criticism, and scored +0.83 too high as a result. Truncation is now a hard error at three layers.
EDM-ARS generates a paper → LSAR reviews it across 8 dimensions → feedback drives targeted revisions — a closed-loop generate-then-review cycle for automated research refinement.
Publications
Technical reports and demo papers generated by EDM-ARS. Demo papers are full ACM-formatted manuscripts produced by the pipeline on HSLS:09 prediction tasks.
Built With
Scope & Roadmap
Every phase on the original roadmap — polymorphism, findings memory, causal inference, multi-dataset support — has shipped. The current release spans five study types across four datasets, with the controlled human evaluation now in progress.