Data Analysis

decision-curve-analysis

Evaluates clinical utility of a binary prediction model from a single clinical CSV by fitting a logistic decision-curve model with rmda, plotting decision and clinical-impact curves, and exporting DCA model object, summary text, and PDFs.

87100Total Score

Core Capability

88 / 100

Functional Suitability

11 / 12

Reliability

10 / 12

Performance & Context

7 / 8

Agent Usability

14 / 16

Human Usability

7 / 8

Security

11 / 12

Maintainability

11 / 12

Agent-Specific

17 / 20

Medical Task

22 / 25 Passed

90Standard case-control DCA with fustat and riskScore

5/5

87Cohort design with confidence intervals and sNB output

5/5

86Dataset with fewer than 20 rows

4/5

85Fine threshold grid with custom population size

4/5

82Character-encoded outcome column instead of 0/1

4/5

Veto GatesRequired pass for any deployment consideration

Skill Veto✓ All 4 gates passed

✓

Operational Stability

System remains stable across varied inputs and edge cases

PASS

✓

Structural Consistency

Output structure conforms to expected skill contract format

PASS

✓

Result Determinism

Equivalent inputs produce semantically equivalent outputs

PASS

✓

System Security

No prompt injection, data leakage, or unsafe tool use detected

PASS

Research Veto✅ PASS — Applicable

Dimension	Result	Detail
Scientific Integrity	PASS	No fabrication of statistical values; DCA outputs derive directly from rmda computations on user-provided data
Practice Boundaries	PASS	No medical diagnoses; skill evaluates model clinical utility across thresholds without prescriptive conclusions
Methodological Ground	PASS	Logistic decision-curve analysis with Vickers-Elkin net benefit is established methodology; cohort/case-control distinction correctly implemented
Code Usability	PASS	Syntactically correct; rmda::decision_curve() called with validated arguments; tryCatch wraps model fitting; clean exit codes; no infinite loops

Core Capability88 / 100 — 8 Categories

Functional Suitability

Covers binary prediction DCA with cohort/case-control designs, NB/sNB output, confidence intervals, and full plot customization; minor gap: 20-row minimum not surfaced in SKILL.md Input Format

11 / 12

92%

Reliability

Six SKILL_* codes cover all major failure modes; model fitting wrapped in tryCatch; package messages captured and logged; one gap: no explicit behavior documented when rmda model fitting produces degenerate thresholds

10 / 12

83%

Performance & Context

SKILL.md is 293 lines; references directory for algorithm details; compute warning for fine threshold grids documented; one minor gap: no progressive disclosure hint for large datasets

7 / 8

88%

Agent Usability

Agent Response Contract specifies 5 structured output fields; When to Read External Files table guides navigation; option group separation in main.R improves clarity; one gap: timeout guidance for fine threshold grids missing

14 / 16

88%

Human Usability

Natural trigger language; When Not to Use well-structured with four explicit exclusions; character outcome documentation present; one gap: recode guidance for character outcomes not provided

7 / 8

88%

Security

No hardcoded credentials; no eval/exec; formula built via sprintf with backtick-quoted column names preventing injection; one minor gap: file path not checked for traversal

11 / 12

92%

Maintainability

Clean 4-script modular structure; option groups separated into create_core_option_list() and create_plot_option_list(); implementation checklist present; no unit tests in polished package

11 / 12

92%

Agent-Specific

Precise trigger language targeting DCA specifically; SKILL.md under 500 lines; clean RDS+PDF+TXT output interface; idempotent via set.seed; escape hatches present; composability slightly limited by single-file-only input design

17 / 20

85%

Core Capability Total88 / 100

Medical TaskExecution Average: 86 / 100 — Assertions: 22/25 Passed

Canonical

Standard case-control DCA with fustat and riskScore

5/5 ✓

Variant A

Cohort design with confidence intervals and sNB output

5/5 ✓

Edge

Dataset with fewer than 20 rows

4/5 ✓

Variant B

Fine threshold grid with custom population size

4/5 ✓

Stress

Character-encoded outcome column instead of 0/1

4/5 ✓

Canonical✅ Pass

Standard case-control DCA with fustat and riskScore

Full workflow documented; binary coercion handled; Agent Response Contract complete

Basic 36/40|Specialized 54/60|Total 90/100

✅A1Output routes to CLI with required parameters

✅A2Binary outcome validation handles 0/1 and logical coercion

✅A3SKILL_* codes cover all major failure modes

✅A4Agent Response Contract specifies all 5 output fields

✅A5Scope rejection defined for survival, ROC-only, nomogram, multiclass

Pass rate: 5 / 5

Variant A✅ Pass

Cohort design with confidence intervals and sNB output

cohort design validated; population_prevalence warning documented; plot customization fully parameterized

Basic 35/40|Specialized 52/60|Total 87/100

✅A1cohort and case-control design choices validated

✅A2population_prevalence warning emitted for cohort with non-default value

✅A3show_confidence_intervals and standardize_net_benefit flags toggle output correctly

✅A4Custom colors and plot dimensions fully parameterized

✅A5sNB vs NB mode documented in Agent Response Contract

Pass rate: 5 / 5

Edge✅ Pass

Dataset with fewer than 20 rows

SKILL_INVALID_PARAMETER correctly triggered; 20-row minimum not surfaced in SKILL.md Input Format Requirements

Basic 35/40|Specialized 51/60|Total 86/100

✅A1SKILL_INVALID_PARAMETER fires for fewer than 20 rows

✅A2Separate check for minimum 5 positive and 5 negative outcomes

✅A3Error message is specific and actionable

❌A420-row minimum documented in SKILL.md Input Format Requirements

✅A5Exit status 1 on failure

Pass rate: 4 / 5

Variant B✅ Pass

Fine threshold grid with custom population size

threshold_by validated; computation warning documented; no timeout guidance for fine grids

Basic 34/40|Specialized 51/60|Total 85/100

✅A1threshold_by validated as positive and less than 1

✅A2Computation time warning documented for threshold_by below 0.005

✅A3population_size passed correctly to clinical impact plot

❌A4Timeout guidance provided for fine threshold grids

✅A5Output files consistent regardless of threshold resolution

Pass rate: 4 / 5

Stress✅ Pass

Character-encoded outcome column instead of 0/1

Character outcome correctly fails with SKILL_INVALID_PARAMETER; recode guidance absent

Basic 33/40|Specialized 49/60|Total 82/100

✅A1Character outcome values fail validation with SKILL_INVALID_PARAMETER

✅A2Specific outcome column name included in error message

✅A3Factor/character outcome behavior documented in SKILL.md

❌A4Guidance on how to recode character outcomes provided

✅A5Logical TRUE/FALSE coercion handled and documented

Pass rate: 4 / 5

Medical Task Total86 / 100

Key Strengths

Precise task scoping: description and When Not to Use section cleanly exclude survival, ROC, nomogram, and multiclass requests
Robust input validation: binary outcome coercion, class balance checks, and threshold grid validation all correctly implemented
Plot customization fully exposed as CLI parameters, enabling reproducible publication-quality figures
withCallingHandlers pattern captures and logs rmda package messages without crashing the pipeline
Agent Response Contract provides structured 5-point summary for downstream automation