Academic Writing

arxiv-preflight

Run a submission-readiness preflight on a manuscript before arXiv upload. Detects LLM artifacts, hallucinated references, placeholder content, AI-use disclosure gaps, and arXiv moderation risks.

90100Total Score
Core Capability
92 / 100
Functional Suitability
12 / 12
Reliability
10 / 12
Performance & Context
7 / 8
Agent Usability
15 / 16
Human Usability
7 / 8
Security
10 / 12
Maintainability
12 / 12
Agent-Specific
19 / 20
Medical Task
29 / 32 Passed
93Clean LaTeX project preflight
4/4
94LLM meta-comments and placeholders detection
5/5
88Hallucinated reference detection via external APIs
4/5
93Multi-file LaTeX with input includes
5/5
82PDF input without LaTeX source (pdftotext unavailable)
4/5
88User requests full peer review (out of scope)
4/4
82AI patterns inside verbatim blocks and inline quotes
3/4

Veto GatesRequired pass for any deployment consideration

Skill Veto✓ All 4 gates passed
✓
Operational Stability
System remains stable across varied inputs and edge cases
PASS
✓
Structural Consistency
Output structure conforms to expected skill contract format
PASS
✓
Result Determinism
Equivalent inputs produce semantically equivalent outputs
PASS
✓
System Security
No prompt injection, data leakage, or unsafe tool use detected
PASS
Research VetoN/A — Not Applicable
DimensionResultDetail
Scientific IntegrityN/A
N/A
Practice BoundariesN/A
N/A
Methodological GroundN/A
N/A
Code UsabilityN/A
N/A

Core Capability92 / 100 — 8 Categories

Functional Suitability
All promised use cases covered; MVP scope section explicitly delineates what is and is not supported
12 / 12
100%
Reliability
Graceful network degradation, PDF fallback, unresolved-include warnings; error messages could be slightly more actionable
10 / 12
83%
Performance & Context
SKILL.md concise at ~70 lines; references loaded on-demand; minor inefficiency in sequential 4-script pipeline
7 / 8
88%
Agent Usability
Clear numbered workflow, consistent severity taxonomy, explicit 'When to read which reference' section; agent-assisted steps slightly less specified
15 / 16
94%
Human Usability
Comprehensive trigger phrases in description; PDF-only input degrades without clear user guidance on next steps
7 / 8
88%
Security
No hardcoded secrets, env var for email, pathlib for paths; file path input not explicitly sanitized against traversal
10 / 12
83%
Maintainability
Excellent modularity: data-driven pattern library, 4 independent scripts, clean JSON interfaces, eval fixtures with assertions
12 / 12
100%
Agent-Specific
Precise trigger, perfect progressive disclosure, idempotent scripts, clear escape hatches; composability slightly limited by full-orchestration assumption
19 / 20
95%
Core Capability Total92 / 100

Medical TaskExecution Average: 88.6 / 100 — Assertions: 29/32 Passed

93
Canonical
Clean LaTeX project preflight
4/4 ✓
94
Variant A
LLM meta-comments and placeholders detection
5/5 ✓
88
Edge
Hallucinated reference detection via external APIs
4/5 ✓
93
Variant B
Multi-file LaTeX with input includes
5/5 ✓
82
Stress
PDF input without LaTeX source (pdftotext unavailable)
4/5 ✓
88
Scope Boundary
User requests full peer review (out of scope)
4/4 ✓
82
Adversarial
AI patterns inside verbatim blocks and inline quotes
3/4 ✓
93
CanonicalPass
Clean LaTeX project preflight

Zero artifact findings; PASS_WITH_FIXES due to network-incomplete reference verification

Basic 38/40|Specialized 55/60|Total 93/100
A1Decision is PASS or PASS_WITH_FIXES
A2No BLOCKER findings reported
A3AI-use disclosure recommendation is no action
A4Every cite key resolves to a BibTeX entry
Pass rate: 4 / 4
94
Variant APass
LLM meta-comments and placeholders detection

10 BLOCKER findings correctly identified with file:line locations

Basic 38/40|Specialized 56/60|Total 94/100
A1Decision is HOLD
A2At least one BLOCKER cites META-01
A3At least one BLOCKER cites PLACE-01
A4At least one finding cites PLACE-06
A5Every finding includes a file:line location
Pass rate: 5 / 5
88
EdgePass
Hallucinated reference detection via external APIs

Fabricated refs correctly flagged BLOCKER; real refs matched via OpenAlex

Basic 36/40|Specialized 52/60|Total 88/100
A1liang2023quantum is flagged BLOCKER or HIGH
A2nguyen2024recursive is flagged BLOCKER or HIGH
A3Real references not flagged as BLOCKER
A4Report explains why flagged refs could not be matched
A5Network-incomplete status clearly surfaced
Pass rate: 4 / 5
93
Variant BPass
Multi-file LaTeX with input includes

Child files correctly processed; findings attributed to correct source files

Basic 38/40|Specialized 55/60|Total 93/100
A1method.tex included in manuscript.json raw_text
A2results.tex included in manuscript.json raw_text
A3At least one BLOCKER cites META-01 from method.tex
A4At least one BLOCKER cites PLACE-01 or PLACE-06 from results.tex
A5Decision is HOLD
Pass rate: 5 / 5
82
StressPass
PDF input without LaTeX source (pdftotext unavailable)

Graceful degradation with clear warnings; reference checks cannot run without text extraction

Basic 34/40|Specialized 48/60|Total 82/100
A1PDF input detected and handled without crash
A2Warning includes install instructions for pdftotext
A3Report clearly states limitations of PDF-only input
A4Skill does not silently skip checks
A5Full reference verification still attempted on extracted text
Pass rate: 4 / 5
88
Scope BoundaryPass
User requests full peer review (out of scope)

Skill correctly identifies out-of-scope request and explains boundaries

Basic 36/40|Specialized 52/60|Total 88/100
A1Skill does not attempt full peer review
A2Skill explains what is out of scope
A3Skill offers what it CAN do instead
A4No hallucinated methodology assessment produced
Pass rate: 4 / 4
82
AdversarialPass
AI patterns inside verbatim blocks and inline quotes

Verbatim/lstlisting/comment suppression works; one false positive on inline backtick-quoted mention

Basic 34/40|Specialized 48/60|Total 82/100
A1Patterns inside verbatim blocks are suppressed
A2Patterns inside lstlisting blocks are suppressed
A3LaTeX comments are suppressed
A4Inline backtick-quoted mentions not falsely flagged
Pass rate: 3 / 4
Medical Task Total88.6 / 100

Key Strengths

  • Data-driven pattern library parsed at runtime from markdown tables — adding new patterns requires no code change
  • Excellent modularity: 4 independent scripts with clean JSON interfaces, 4 reference docs, eval fixtures with assertions
  • Conservative design philosophy: minimizes false positives, never calls papers AI-generated from style alone
  • Graceful network degradation: marks sections INCOMPLETE rather than silently skipping, prevents false PASS
  • Comprehensive trigger description covers natural user phrasings without over-triggering