Data Analysis

external-model-validation

Validates an existing prognostic risk signature on an external bulk expression cohort with survival outcomes. Produces risk scores, Kaplan-Meier curves, risk distribution plots, expression heatmap, and time-dependent ROC curves from a fixed model coefficient file.

90100Total Score

Core Capability

95 / 100

Functional Suitability

12 / 12

Reliability

12 / 12

Performance & Context

8 / 8

Agent Usability

15 / 16

Human Usability

7 / 8

Security

11 / 12

Maintainability

12 / 12

Agent-Specific

18 / 20

Medical Task

34 / 35 Passed

91BRCA cohort validation with bundled test data

5/5

88Day-unit follow-up with custom ROC time points in years

5/5

84Non-empty output directory without --overwrite flag

5/5

90Custom plot colors, ROC legend position, and KM break interval

5/5

87Five ROC time points including potentially out-of-range values

4/5

82Request for model training, nomogram, or calibration analysis

5/5

85Model genes absent from expression matrix

5/5

Veto GatesRequired pass for any deployment consideration

Skill Veto✓ All 4 gates passed

✓

Operational Stability

System remains stable across varied inputs and edge cases

PASS

✓

Structural Consistency

Output structure conforms to expected skill contract format

PASS

✓

Result Determinism

Equivalent inputs produce semantically equivalent outputs

PASS

✓

System Security

No prompt injection, data leakage, or unsafe tool use detected

PASS

Research Veto✅ PASS — Applicable

Dimension	Result	Detail
Scientific Integrity	PASS	No fabricated risk scores, p-values, AUC values, or survival statistics across all 7 inputs; all outputs derived from actual survival computation and time-dependent ROC with fixed seed
Practice Boundaries	PASS	Research Use Notice explicitly states the skill is for research and validation only; de-identification requirement stated; no diagnostic conclusions or treatment recommendations produced
Methodological Ground	PASS	Median-split risk stratification, log-rank KM comparison, and IPCW time-dependent ROC are all standard and correct for external validation of a prognostic signature; roc_times-in-years documentation prevents a common unit-mismatch error
Code Usability	PASS	scripts/main.R and 5 helper scripts are syntactically complete; validate_color, parse_csv_arg, and validate_choice all called at startup; analysis.log and run_parameters.tsv preserved even on error; no infinite loops or missing core dependencies

Core Capability95 / 100 — 8 Categories

Functional Suitability

Complete 4-step validation pipeline with all 6 output plots fully documented; roc_times-in-years note prevents the most common unit-mismatch error; Research Use Notice proactively addresses de-identification requirements

12 / 12

100%

Reliability

Most sophisticated error handling of all 6 audited skills: analysis.log with memory checkpoints, run_parameters.tsv preserved on error, graceful partial-save logic, SKILL_PKG_VERSION for version constraint failures, 9 distinct SKILL_* error codes

12 / 12

100%

Performance & Context

Only skill with a complete references/ directory on disk (algorithm.md, troubleshooting.md, cli-guide.md, baseline-run.md, project-structure.md); full progressive disclosure design realized; SKILL.md 308 lines is the longest but structured with clear section headers

8 / 8

100%

Agent Usability

Multiple When to Read External Files table with 11 entries covering every operational scenario; roc_times documentation with explicit years-regardless-of-unit note is exceptional; one minor gap: the overwrite flag behavior could be more prominently explained in the Quick Start workflow

15 / 16

94%

Human Usability

Research Use Notice positioned prominently before When to Use; When Not to Use covers 5 distinct cases; natural trigger language for 'external validation' is well-calibrated

7 / 8

88%

Security

Research Use Notice plus de-identification requirement; validate_color prevents injection via color arguments; normalizePath(mustWork=TRUE) on user paths could expose filesystem structure in error messages but acceptable in local research context

11 / 12

92%

Maintainability

Six scripts with clear single responsibilities (plotting.R, io.R, functions.R, run_analysis.R, utils.R, main.R); baseline-run.md for regression testing; refresh_example_output.R for test bundle regeneration; project-structure.md for onboarding

12 / 12

100%

Agent-Specific

Best trigger precision of all 6 audited skills; When to Use / When Not to Use both present; references/ directory enables genuine progressive disclosure; --overwrite flag prevents accidental data loss; composable clean outputs; idempotent with --overwrite TRUE

18 / 20

90%

Core Capability Total95 / 100

Medical TaskExecution Average: 86.7 / 100 — Assertions: 34/35 Passed

Canonical

BRCA cohort validation with bundled test data

5/5 ✓

Variant A

Day-unit follow-up with custom ROC time points in years

5/5 ✓

Edge

Non-empty output directory without --overwrite flag

5/5 ✓

Variant B

Custom plot colors, ROC legend position, and KM break interval

5/5 ✓

Stress

Five ROC time points including potentially out-of-range values

4/5 ✓

Scope Boundary

Request for model training, nomogram, or calibration analysis

5/5 ✓

Adversarial

Model genes absent from expression matrix

5/5 ✓

Canonical✅ Pass

BRCA cohort validation with bundled test data

All 6 plots, risk table, RDS, analysis.log, run_parameters.tsv, session_info.txt written; exit code 0

Basic 38/40|Specialized 53/60|Total 91/100

✅A1All 6 plot files generated in plot/ directory (KM, risk score, survival status, heatmap, ROC)

✅A2out_varifyRisk.txt risk table written for all matched samples with risk scores and groups

✅A3analysis.log captures memory checkpoints and processing steps throughout run

✅A4run_parameters.tsv records exact parameter values for complete reproducibility

✅A5Research Use Notice confirms output is for research only; no diagnostic claims produced in any output

Pass rate: 5 / 5

Variant A✅ Pass

Day-unit follow-up with custom ROC time points in years

OS.time in days correctly converted to years; roc_times interpreted as years as documented; all outputs consistent

Basic 36/40|Specialized 52/60|Total 88/100

✅A1--time_unit day causes OS.time to be converted from days to years before all survival analyses

✅A2--roc_times 1,2,3 correctly interpreted as years despite --time_unit day

✅A3ROC PDF shows three time-dependent curves at 1, 2, 3-year time points

✅A4SKILL_INVALID_PARAMETER emitted for roc_times values that are not positive numeric

✅A5run_parameters.tsv records both time_unit and roc_times for complete reproducibility

Pass rate: 5 / 5

Edge✅ Pass

Non-empty output directory without --overwrite flag

--overwrite protection correctly prevents accidental overwrite; prior results preserved

Basic 33/40|Specialized 51/60|Total 84/100

✅A1--overwrite flag required to write into non-empty output directory

✅A2Prior results not silently overwritten without explicit --overwrite flag

✅A3With --overwrite TRUE, analysis proceeds normally and replaces previous outputs

✅A4run_parameters.tsv records overwrite=TRUE when flag is used for audit trail

✅A5No partial output on overwrite check failure; existing directory content preserved

Pass rate: 5 / 5

Variant B✅ Pass

Custom plot colors, ROC legend position, and KM break interval

All color and plot parameters validated; custom styling applied; all output files present

Basic 37/40|Specialized 53/60|Total 90/100

✅A1All color arguments validated by validate_color() before analysis begins

✅A2--roc_pos topleft accepted as a valid ROC legend position from documented enum

✅A3--km_breaks 2 applied to set KM x-axis breaks to 2-year intervals

✅A4SKILL_INVALID_PARAMETER emitted for invalid hex color strings

✅A5run_parameters.tsv captures all custom color and plot settings for reproducibility

Pass rate: 5 / 5

Stress✅ Pass

Five ROC time points including potentially out-of-range values

roc_colors recycled correctly; one gap: max follow-up pre-validation at argument stage rather than inside ROC

Basic 35/40|Specialized 52/60|Total 87/100

✅A1roc_times values validated as positive numeric at argument parsing stage

✅A2SKILL_INVALID_PARAMETER or SKILL_ANALYSIS_ERROR emitted for roc_times exceeding max follow-up

✅A3roc_colors automatically recycled when fewer colors provided than time points

❌A4Pre-validation check confirms all roc_times are below max observed follow-up before ROC computation begins

✅A5run_parameters.tsv records all 5 roc_times values for complete debugging context

Pass rate: 4 / 5

Scope Boundary✅ Pass

Request for model training, nomogram, or calibration analysis

All four out-of-scope cases correctly handled; Research Use Notice and When Not to Use both fire

Basic 32/40|Specialized 50/60|Total 82/100

✅A1Model training request correctly identified as out of scope

✅A2Nomogram construction and calibration analysis correctly identified as out of scope

✅A3Scope redirect message provided as documented in Input Validation section

✅A4Research Use Notice visible prominently before the When to Use section

✅A5No partial analysis attempted on out-of-scope request

Pass rate: 5 / 5

Adversarial✅ Pass

Model genes absent from expression matrix

Missing genes detected at Step 2 of workflow; analysis.log captures which genes are absent; clean failure

Basic 33/40|Specialized 52/60|Total 85/100

✅A1Missing model genes detected and reported before risk score computation begins

✅A2analysis.log captures which specific model genes are absent from expression matrix

✅A3Appropriate SKILL_* error code emitted for missing model genes

✅A4No partial risk scores computed when any required model gene is missing

✅A5SKILL_INVALID_DATA handling for duplicate genes includes deduplication guidance

Pass rate: 5 / 5

Medical Task Total86.7 / 100

Key Strengths

roc_times documentation explicitly stating 'always in years regardless of --time_unit' prevents the most common unit-mismatch error in time-dependent ROC analysis
Only skill in the audited set with a complete references/ directory on disk, achieving full progressive disclosure with algorithm.md, troubleshooting.md, baseline-run.md, and project-structure.md
Research Use Notice prominently placed before When to Use, with explicit de-identification requirement, is the strongest responsible-use design of all 6 audited skills
--overwrite flag protection prevents silent data loss from accidental reruns into existing output directories
analysis.log with memory checkpoints plus run_parameters.tsv preserved even on error enables complete post-hoc debugging