Data Analysis

decision-curve-analysis

Evaluates clinical utility of a binary prediction model from a single clinical CSV by fitting a logistic decision-curve model with rmda, plotting decision and clinical-impact curves, and exporting DCA model object, summary text, and PDFs.

87100Total Score
Core Capability
88 / 100
Functional Suitability
11 / 12
Reliability
10 / 12
Performance & Context
7 / 8
Agent Usability
14 / 16
Human Usability
7 / 8
Security
11 / 12
Maintainability
11 / 12
Agent-Specific
17 / 20
Medical Task
22 / 25 Passed
90Standard case-control DCA with fustat and riskScore
5/5
87Cohort design with confidence intervals and sNB output
5/5
86Dataset with fewer than 20 rows
4/5
85Fine threshold grid with custom population size
4/5
82Character-encoded outcome column instead of 0/1
4/5

Veto GatesRequired pass for any deployment consideration

Skill Veto✓ All 4 gates passed
✓
Operational Stability
System remains stable across varied inputs and edge cases
PASS
✓
Structural Consistency
Output structure conforms to expected skill contract format
PASS
✓
Result Determinism
Equivalent inputs produce semantically equivalent outputs
PASS
✓
System Security
No prompt injection, data leakage, or unsafe tool use detected
PASS
Research Veto✓ PASS — Applicable
DimensionResultDetail
Scientific IntegrityPASS
No fabrication of statistical values; DCA outputs derive directly from rmda computations on user-provided data
Practice BoundariesPASS
No medical diagnoses; skill evaluates model clinical utility across thresholds without prescriptive conclusions
Methodological GroundPASS
Logistic decision-curve analysis with Vickers-Elkin net benefit is established methodology; cohort/case-control distinction correctly implemented
Code UsabilityPASS
Syntactically correct; rmda::decision_curve() called with validated arguments; tryCatch wraps model fitting; clean exit codes; no infinite loops

Core Capability88 / 100 — 8 Categories

Functional Suitability
Covers binary prediction DCA with cohort/case-control designs, NB/sNB output, confidence intervals, and full plot customization; minor gap: 20-row minimum not surfaced in SKILL.md Input Format
11 / 12
92%
Reliability
Six SKILL_* codes cover all major failure modes; model fitting wrapped in tryCatch; package messages captured and logged; one gap: no explicit behavior documented when rmda model fitting produces degenerate thresholds
10 / 12
83%
Performance & Context
SKILL.md is 293 lines; references directory for algorithm details; compute warning for fine threshold grids documented; one minor gap: no progressive disclosure hint for large datasets
7 / 8
88%
Agent Usability
Agent Response Contract specifies 5 structured output fields; When to Read External Files table guides navigation; option group separation in main.R improves clarity; one gap: timeout guidance for fine threshold grids missing
14 / 16
88%
Human Usability
Natural trigger language; When Not to Use well-structured with four explicit exclusions; character outcome documentation present; one gap: recode guidance for character outcomes not provided
7 / 8
88%
Security
No hardcoded credentials; no eval/exec; formula built via sprintf with backtick-quoted column names preventing injection; one minor gap: file path not checked for traversal
11 / 12
92%
Maintainability
Clean 4-script modular structure; option groups separated into create_core_option_list() and create_plot_option_list(); implementation checklist present; no unit tests in polished package
11 / 12
92%
Agent-Specific
Precise trigger language targeting DCA specifically; SKILL.md under 500 lines; clean RDS+PDF+TXT output interface; idempotent via set.seed; escape hatches present; composability slightly limited by single-file-only input design
17 / 20
85%
Core Capability Total88 / 100

Medical TaskExecution Average: 86 / 100 — Assertions: 22/25 Passed

90
Canonical
Standard case-control DCA with fustat and riskScore
5/5 ✓
87
Variant A
Cohort design with confidence intervals and sNB output
5/5 ✓
86
Edge
Dataset with fewer than 20 rows
4/5 ✓
85
Variant B
Fine threshold grid with custom population size
4/5 ✓
82
Stress
Character-encoded outcome column instead of 0/1
4/5 ✓
90
CanonicalPass
Standard case-control DCA with fustat and riskScore

Full workflow documented; binary coercion handled; Agent Response Contract complete

Basic 36/40|Specialized 54/60|Total 90/100
A1Output routes to CLI with required parameters
A2Binary outcome validation handles 0/1 and logical coercion
A3SKILL_* codes cover all major failure modes
A4Agent Response Contract specifies all 5 output fields
A5Scope rejection defined for survival, ROC-only, nomogram, multiclass
Pass rate: 5 / 5
87
Variant APass
Cohort design with confidence intervals and sNB output

cohort design validated; population_prevalence warning documented; plot customization fully parameterized

Basic 35/40|Specialized 52/60|Total 87/100
A1cohort and case-control design choices validated
A2population_prevalence warning emitted for cohort with non-default value
A3show_confidence_intervals and standardize_net_benefit flags toggle output correctly
A4Custom colors and plot dimensions fully parameterized
A5sNB vs NB mode documented in Agent Response Contract
Pass rate: 5 / 5
86
EdgePass
Dataset with fewer than 20 rows

SKILL_INVALID_PARAMETER correctly triggered; 20-row minimum not surfaced in SKILL.md Input Format Requirements

Basic 35/40|Specialized 51/60|Total 86/100
A1SKILL_INVALID_PARAMETER fires for fewer than 20 rows
A2Separate check for minimum 5 positive and 5 negative outcomes
A3Error message is specific and actionable
A420-row minimum documented in SKILL.md Input Format Requirements
A5Exit status 1 on failure
Pass rate: 4 / 5
85
Variant BPass
Fine threshold grid with custom population size

threshold_by validated; computation warning documented; no timeout guidance for fine grids

Basic 34/40|Specialized 51/60|Total 85/100
A1threshold_by validated as positive and less than 1
A2Computation time warning documented for threshold_by below 0.005
A3population_size passed correctly to clinical impact plot
A4Timeout guidance provided for fine threshold grids
A5Output files consistent regardless of threshold resolution
Pass rate: 4 / 5
82
StressPass
Character-encoded outcome column instead of 0/1

Character outcome correctly fails with SKILL_INVALID_PARAMETER; recode guidance absent

Basic 33/40|Specialized 49/60|Total 82/100
A1Character outcome values fail validation with SKILL_INVALID_PARAMETER
A2Specific outcome column name included in error message
A3Factor/character outcome behavior documented in SKILL.md
A4Guidance on how to recode character outcomes provided
A5Logical TRUE/FALSE coercion handled and documented
Pass rate: 4 / 5
Medical Task Total86 / 100

Key Strengths

  • Precise task scoping: description and When Not to Use section cleanly exclude survival, ROC, nomogram, and multiclass requests
  • Robust input validation: binary outcome coercion, class balance checks, and threshold grid validation all correctly implemented
  • Plot customization fully exposed as CLI parameters, enabling reproducible publication-quality figures
  • withCallingHandlers pattern captures and logs rmda package messages without crashing the pipeline
  • Agent Response Contract provides structured 5-point summary for downstream automation