Academic Writing
arxiv-preflight
Run a submission-readiness preflight on a manuscript before arXiv upload. Detects LLM artifacts, hallucinated references, placeholder content, AI-use disclosure gaps, and arXiv moderation risks.
90100Total Score
Core Capability
92 / 100
Functional Suitability
12 / 12
Reliability
10 / 12
Performance & Context
7 / 8
Agent Usability
15 / 16
Human Usability
7 / 8
Security
10 / 12
Maintainability
12 / 12
Agent-Specific
19 / 20
Medical Task
29 / 32 Passed
93Clean LaTeX project preflight
4/4
94LLM meta-comments and placeholders detection
5/5
88Hallucinated reference detection via external APIs
4/5
93Multi-file LaTeX with input includes
5/5
82PDF input without LaTeX source (pdftotext unavailable)
4/5
88User requests full peer review (out of scope)
4/4
82AI patterns inside verbatim blocks and inline quotes
3/4
Veto GatesRequired pass for any deployment consideration
Skill Veto✓ All 4 gates passed
✓
Operational Stability
System remains stable across varied inputs and edge cases
PASS✓
Structural Consistency
Output structure conforms to expected skill contract format
PASS✓
Result Determinism
Equivalent inputs produce semantically equivalent outputs
PASS✓
System Security
No prompt injection, data leakage, or unsafe tool use detected
PASSResearch VetoN/A — Not Applicable
| Dimension | Result | Detail |
|---|---|---|
| Scientific Integrity | N/A | N/A |
| Practice Boundaries | N/A | N/A |
| Methodological Ground | N/A | N/A |
| Code Usability | N/A | N/A |
Core Capability92 / 100 — 8 Categories
Functional Suitability
All promised use cases covered; MVP scope section explicitly delineates what is and is not supported
12 / 12
100%
Reliability
Graceful network degradation, PDF fallback, unresolved-include warnings; error messages could be slightly more actionable
10 / 12
83%
Performance & Context
SKILL.md concise at ~70 lines; references loaded on-demand; minor inefficiency in sequential 4-script pipeline
7 / 8
88%
Agent Usability
Clear numbered workflow, consistent severity taxonomy, explicit 'When to read which reference' section; agent-assisted steps slightly less specified
15 / 16
94%
Human Usability
Comprehensive trigger phrases in description; PDF-only input degrades without clear user guidance on next steps
7 / 8
88%
Security
No hardcoded secrets, env var for email, pathlib for paths; file path input not explicitly sanitized against traversal
10 / 12
83%
Maintainability
Excellent modularity: data-driven pattern library, 4 independent scripts, clean JSON interfaces, eval fixtures with assertions
12 / 12
100%
Agent-Specific
Precise trigger, perfect progressive disclosure, idempotent scripts, clear escape hatches; composability slightly limited by full-orchestration assumption
19 / 20
95%
Core Capability Total92 / 100
Medical TaskExecution Average: 88.6 / 100 — Assertions: 29/32 Passed
93
Canonical
Clean LaTeX project preflight
4/4 ✓
94
Variant A
LLM meta-comments and placeholders detection
5/5 ✓
88
Edge
Hallucinated reference detection via external APIs
4/5 ✓
93
Variant B
Multi-file LaTeX with input includes
5/5 ✓
82
Stress
PDF input without LaTeX source (pdftotext unavailable)
4/5 ✓
88
Scope Boundary
User requests full peer review (out of scope)
4/4 ✓
82
Adversarial
AI patterns inside verbatim blocks and inline quotes
3/4 ✓
93
Canonical✅ Pass
Clean LaTeX project preflight
Zero artifact findings; PASS_WITH_FIXES due to network-incomplete reference verification
Basic 38/40|Specialized 55/60|Total 93/100
✅A1Decision is PASS or PASS_WITH_FIXES
✅A2No BLOCKER findings reported
✅A3AI-use disclosure recommendation is no action
✅A4Every cite key resolves to a BibTeX entry
Pass rate: 4 / 4
94
Variant A✅ Pass
LLM meta-comments and placeholders detection
10 BLOCKER findings correctly identified with file:line locations
Basic 38/40|Specialized 56/60|Total 94/100
✅A1Decision is HOLD
✅A2At least one BLOCKER cites META-01
✅A3At least one BLOCKER cites PLACE-01
✅A4At least one finding cites PLACE-06
✅A5Every finding includes a file:line location
Pass rate: 5 / 5
88
Edge✅ Pass
Hallucinated reference detection via external APIs
Fabricated refs correctly flagged BLOCKER; real refs matched via OpenAlex
Basic 36/40|Specialized 52/60|Total 88/100
✅A1liang2023quantum is flagged BLOCKER or HIGH
✅A2nguyen2024recursive is flagged BLOCKER or HIGH
✅A3Real references not flagged as BLOCKER
✅A4Report explains why flagged refs could not be matched
❌A5Network-incomplete status clearly surfaced
Pass rate: 4 / 5
93
Variant B✅ Pass
Multi-file LaTeX with input includes
Child files correctly processed; findings attributed to correct source files
Basic 38/40|Specialized 55/60|Total 93/100
✅A1method.tex included in manuscript.json raw_text
✅A2results.tex included in manuscript.json raw_text
✅A3At least one BLOCKER cites META-01 from method.tex
✅A4At least one BLOCKER cites PLACE-01 or PLACE-06 from results.tex
✅A5Decision is HOLD
Pass rate: 5 / 5
82
Stress✅ Pass
PDF input without LaTeX source (pdftotext unavailable)
Graceful degradation with clear warnings; reference checks cannot run without text extraction
Basic 34/40|Specialized 48/60|Total 82/100
✅A1PDF input detected and handled without crash
✅A2Warning includes install instructions for pdftotext
✅A3Report clearly states limitations of PDF-only input
✅A4Skill does not silently skip checks
❌A5Full reference verification still attempted on extracted text
Pass rate: 4 / 5
88
Scope Boundary✅ Pass
User requests full peer review (out of scope)
Skill correctly identifies out-of-scope request and explains boundaries
Basic 36/40|Specialized 52/60|Total 88/100
✅A1Skill does not attempt full peer review
✅A2Skill explains what is out of scope
✅A3Skill offers what it CAN do instead
✅A4No hallucinated methodology assessment produced
Pass rate: 4 / 4
82
Adversarial✅ Pass
AI patterns inside verbatim blocks and inline quotes
Verbatim/lstlisting/comment suppression works; one false positive on inline backtick-quoted mention
Basic 34/40|Specialized 48/60|Total 82/100
✅A1Patterns inside verbatim blocks are suppressed
✅A2Patterns inside lstlisting blocks are suppressed
✅A3LaTeX comments are suppressed
❌A4Inline backtick-quoted mentions not falsely flagged
Pass rate: 3 / 4
Medical Task Total88.6 / 100
Key Strengths
- Data-driven pattern library parsed at runtime from markdown tables — adding new patterns requires no code change
- Excellent modularity: 4 independent scripts with clean JSON interfaces, 4 reference docs, eval fixtures with assertions
- Conservative design philosophy: minimizes false positives, never calls papers AI-generated from style alone
- Graceful network degradation: marks sections INCOMPLETE rather than silently skipping, prevents false PASS
- Comprehensive trigger description covers natural user phrasings without over-triggering