Data Analysis

roc-diagnostic-performance

Use when evaluating diagnostic biomarker performance from case-control expression data with logistic regression and ROC curves, exporting coefficient and AUC tables together with a ROC PDF. NOT for: survival analysis, time-to-event outcomes, multiclass classification, calibration curves, decision-curve analysis, or nomogram construction.

85100Total Score

Core Capability

91 / 100

Functional Suitability

12 / 12

Reliability

10 / 12

Performance & Context

7 / 8

Agent Usability

15 / 16

Human Usability

7 / 8

Security

11 / 12

Maintainability

12 / 12

Agent-Specific

17 / 20

Medical Task

23 / 25 Passed

86Basic ROC run with FOXP3, CD45, CD3E markers on Disease vs Control

5/5

83Custom group column and plot customization options

5/5

79Sample ID mismatch and insufficient case count

4/5

83Timeout parameter and saved model bundle verification

5/5

77Partial marker list with some absent genes and logistic failure

4/5

Veto GatesRequired pass for any deployment consideration

Skill Veto✓ All 4 gates passed

✓

Operational Stability

System remains stable across varied inputs and edge cases

PASS

✓

Structural Consistency

Output structure conforms to expected skill contract format

PASS

✓

Result Determinism

Equivalent inputs produce semantically equivalent outputs

PASS

✓

System Security

No prompt injection, data leakage, or unsafe tool use detected

PASS

Research Veto✅ PASS — Applicable

Dimension	Result	Detail
Scientific Integrity	PASS	No fabricated statistics; logistic regression and pROC-based AUC are standard methods; three canonical references cited; output is computed values only
Practice Boundaries	PASS	No diagnostic conclusions produced; skill is scoped to AUC values and coefficient tables; When Not to Use explicitly excludes clinical decision making
Methodological Ground	PASS	Multivariable logistic regression for binary case-control classification is methodologically appropriate; individual marker ROC comparison is a standard diagnostic evaluation pattern
Code Usability	PASS	Script syntax valid; set.seed() present; modular 7-file structure; dependency check at startup; no logic errors in code review

Core Capability91 / 100 — 8 Categories

Functional Suitability

All documented use cases covered; full model vs individual marker comparison well-designed; output schema thorough

12 / 12

100%

Reliability

SKILL_* codes comprehensive including SKILL_SAMPLE_MISMATCH; set.seed() present; minor gap: partial marker drop may occur silently

10 / 12

83%

Performance & Context

SKILL.md is 308 lines; references well-utilized; slightly heavier than minimal but within acceptable range

7 / 8

88%

Agent Usability

Typical user requests listed in When to Use; argument table complete; minor gap: no note on how to interpret AUC < 0.5

15 / 16

94%

Human Usability

Trigger description uses natural user language; When Not to Use clearly states exclusions; high discoverability

7 / 8

88%

Security

No credentials or secrets; set.seed prevents variance; pROC dependency check at startup prevents partial execution

11 / 12

92%

Maintainability

Seven-file modular structure; automated smoke test (run_smoke_test.R) included; test data bundled; references documented

12 / 12

100%

Agent-Specific

Trigger description precise with NOT-for exclusions; escape hatches documented; composability limited by dual required input file design

17 / 20

85%

Core Capability Total91 / 100

Medical TaskExecution Average: 81.6 / 100 — Assertions: 23/25 Passed

Canonical

Basic ROC run with FOXP3, CD45, CD3E markers on Disease vs Control

5/5 ✓

Variant A

Custom group column and plot customization options

5/5 ✓

Edge

Sample ID mismatch and insufficient case count

4/5 ✓

Variant B

Timeout parameter and saved model bundle verification

5/5 ✓

Stress

Partial marker list with some absent genes and logistic failure

4/5 ✓

Canonical✅ Pass

Basic ROC run with FOXP3, CD45, CD3E markers on Disease vs Control

pROC dependency prevents execution; SKILL_PACKAGE_NOT_FOUND fires correctly; code review confirms set.seed and modular structure

Basic 34/40|Specialized 52/60|Total 86/100

✅A1SKILL_PACKAGE_NOT_FOUND fires at startup with named missing package

✅A2set.seed(options$seed) present in main.R

✅A3Four required arguments validated before analysis begins

✅A47-file modular script structure confirmed

✅A5No security issues detected in script code

Pass rate: 5 / 5

Variant A✅ Pass

Custom group column and plot customization options

Group column auto-detection and plot arguments confirmed; legend_position validation confirmed; pROC blocks execution

Basic 33/40|Specialized 50/60|Total 83/100

✅A1--group_col argument documented with auto-detect fallback

✅A2Plot customization arguments (plot_width, plot_height, plot_title) documented

✅A3legend_position validated against allowed values

✅A4line_colors documented as comma-separated hex values

✅A5Output files include both model bundle and tables

Pass rate: 5 / 5

Edge✅ Pass

Sample ID mismatch and insufficient case count

SKILL_SAMPLE_MISMATCH documented; minimum sample thresholds documented; runtime verification blocked by pROC absence

Basic 31/40|Specialized 48/60|Total 79/100

✅A1SKILL_SAMPLE_MISMATCH documented for non-overlapping sample IDs

✅A2Minimum 10 matched samples, 2 cases, 2 controls requirement documented

✅A3SKILL_EMPTY_DATA documented for no usable rows after filtering

❌A4Runtime validation of sample mismatch error verified in live execution

✅A5Expression matrix gene identifier uniqueness requirement documented

Pass rate: 4 / 5

Variant B✅ Pass

Timeout parameter and saved model bundle verification

Timeout and model bundle documented; coefficient table schema confirmed; pROC blocks execution

Basic 33/40|Specialized 50/60|Total 83/100

✅A1--timeout_seconds present and apply_time_limit() called in main.R

✅A2data/roc_model.rds documented as saved model bundle with data and genes

✅A3model_coefficients.csv column schema documented

✅A4roc_auc_summary.csv documents full model and individual marker AUCs

✅A5session_info.txt included in output file list

Pass rate: 5 / 5

Stress✅ Pass

Partial marker list with some absent genes and logistic failure

SKILL_EMPTY_DATA and SKILL_INVALID_PARAMETER documented; partial marker drop warning gap identified; pROC blocks execution

Basic 30/40|Specialized 47/60|Total 77/100

✅A1SKILL_EMPTY_DATA documented for no markers remaining after filtering

✅A2SKILL_INVALID_PARAMETER documented for logistic fitting failure

❌A3Warning emitted when some but not all marker genes are found and used

✅A4Logistic regression stability addressed in error handling

✅A5Skill does not produce clinical diagnostic conclusions

Pass rate: 4 / 5

Medical Task Total81.6 / 100

Key Strengths

Comprehensive output schema with coefficient table, AUC summary, model bundle, and ROC PDF covers all downstream needs
set.seed() with configurable --seed argument ensures full reproducibility
SKILL_SAMPLE_MISMATCH is a unique and valuable error code for the dual-file input pattern
7-file modular script structure with automated smoke test runner demonstrates mature engineering