[Sergey Dubinin] · car-maintenance-advisor.zip
car-maintenance-advisor
- Score: 33/40
- Verdict: Pass (
workshop-pass) - Отправитель: Sergey Dubinin serg.dubinin.ru@gmail.com
- SHA-256:
ee6a3fc4fe9fdbcc… - Blockers: none
- Unverified: F08
Required rework
—
Полная scorecard
| ID | Score | Status | Evidence | Defect / minimal fix |
|---|---|---|---|---|
| A01 | 1/1 | passed | SKILL.md:1-11 frontmatter parses via yaml.safe_load; keys name,description present and non-empty (description 376 chars). Extra standard version field is harmless. | — |
| A02 | 1/1 | passed | name='car-maintenance-advisor', len 23 ≤64, matches ^[a-z0-9]+(?:-[a-z0-9]+)*$, not reserved, denotes a concrete activity. | — |
| A03 | 1/1 | passed | description 376 chars (1–1024), no XML tags (re.search none), routing text, not a YAML object. | — |
| A04 | 1/1 | passed | description states the task (“Исследует…”), when to apply (“Применять при…”), domain terms (ТО, масло, регламент, допуски); finite third person “Исследует”. | — |
| A05 | 1/1 | passed | description has no workflow/defaults/examples; each phrase changes the routing decision (scope terms + positive/negative triggers). | — |
| A06 | 1/1 | passed | explicit positive (“Применять при…”) and negative/anti-trigger (“Не применять для…”: diagnostics, repair, purchase, comparison, tuning, ПДД, insurance) give a useful boundary. | — |
| A07 | 1/1 | passed | all 14 referenced files exist (verified); relative paths, forward slashes; grep for /home/,Users,C:\,@,~ in SKILL.md = none; runtime paths use <workdir> placeholder. | — |
| B01 | 1/1 | passed | repeatable 5-stage procedure (SKILL.md:104–463) producing an observable research report. | — |
| B02 | 1/1 | passed | one coherent narrow domain (maintenance schedules/fluid specs/component lifespan); explicit scope invariant (SKILL.md:52–57). | — |
| B03 | 1/1 | passed | terminology consistent; volatile facts (availability/prices/support dates) handled via runtime freshness rules (source-taxonomy.md:150–159), not hardcoded as eternal. | — |
| B04 | 1/1 | passed | medium freedom stated (SKILL.md:19–22): flexible plan, hard output/source-eval/thresholds; safety-critical stricter; plan gate. | — |
| C01 | 1/1 | passed | one clear default path: Планирование→Поиск→Чтение→Верификация→Синтез with concrete numbered substeps. | — |
| C02 | 1/1 | passed | each branch has a when-to-choose rule (input-markup table SKILL.md:63–67, early-exit 72–81, orientation scan 1.4); default path is explicit. | — |
| C03 | 1/1 | passed | plan gate requires explicit approval (SKILL.md:202–203); validate-and-revise with ≤2 returns (453–455); no destructive external action; search begins only after the gate. | — |
| C04 | 1/1 | passed | feedback loop: what to check (verification-rubric.md:52–83), what is a failure, which stage to return to (table 88–93), what to change, restart with 2-return cap then insufficient. | — |
| C05 | 1/1 | passed | report-template.md defines required result, per-claim evidence, confidence, readiness; adaptable template appropriate to a judgment research task. | — |
| C06 | 1/1 | passed | common workflow easy to locate; vague steps replaced by checkable actions and deterministic scripts (normalize/count_independent/validate). | — |
| C07 | 1/1 | passed | tool policy in guardrails: check availability before stage 2 (SKILL.md:89–90, 227–230), MCP fully qualified, no hidden actions; scripts clearly read vs execute (99–100). | — |
| D01 | 0/1 | failed | wc -l SKILL.md = 501 lines > 500 limit. | Trim or merge one line to reach ≤500 (e.g., fold the two-line closing sentence of §5.5 into one). |
| D02 | 1/1 | passed | core workflow/safety/decisions in SKILL.md; long theory, references and templates moved to references/ and assets/. | — |
| D03 | 0/1 | failed | references >100 lines (source-taxonomy 174, verification-rubric 132, extraction-rules 121, search-tactics 103) open with a “when to read” line but none has a Contents/TOC block. | Add a brief ## Содержание section listing headers to each reference file longer than 100 lines. |
| D04 | 1/1 | passed | references/assets/scripts/evals used purposefully with descriptive names; no misc/dead files; clean tree, no OS/editor metadata. | — |
| D05 | 1/1 | passed | each script has shebang + docstring (usage/args/exit codes), stdlib-only deps declared (README:70), JSON/table output, portable relative invocation. | — |
| D06 | 1/1 | passed | ran all three scripts read-only: normalize (exit 0/1 + useful warnings), count_independent (valid→0, broken→1 “НЕ ДОБИРАЕТ”/“свёрнуто”), validate_artifacts (valid→0 “пройдены”, broken→1 with 8 specific violations). | — |
| E01 | 1/1 | passed | guardrails treat external text as data not commands (SKILL.md:91–93; search-tactics.md:87–89) and templates as form not facts (94–95). | — |
| E02 | 1/1 | passed | no secrets exposed; no undeclared downloads; no dependency install (scripts stdlib-only). | — |
| E03 | 0/1 | failed | auto-creates intermediate draft/ledger files (source-ledger.md SKILL.md:288–292, findings.md 392–396, verification.md) not explicitly requested by the user and not the declared result; verification checklist is explicitly hidden from the user (SKILL.md:457). Not a blocker (local recoverable, no secrets). | Make intermediate working-file creation opt-in/informed (consent for the working dir) or keep intermediates ephemeral and persist only the user-requested report. |
| E04 | 1/1 | passed | scripts are pure stdlib Python; no shell/network/bundled binaries/obfuscation; web search is the declared, transparent purpose. | — |
| E05 | 1/1 | passed | no instruction to ignore prior instructions, hide actions, weaken safeguards, or expand the permission model; guardrails reinforce safeguards. | — |
| E06 | 1/1 | passed | tool availability checked before first query; unavailability is declared and the skill stops rather than simulating search (SKILL.md:89–90, 227–230). | — |
| E07 | 1/1 | passed | workflow uses <workdir> placeholders; scripts portable; no home/username/drive/absolute paths in SKILL.md. (README install snippet uses author’s 08.08/ date dir but is adaptable guidance, non-blocking.) | — |
| E08 | 1/1 | passed | MCP rule requires fully-qualified ServerName:tool_name (SKILL.md:96); no unqualified MCP names invoked anywhere. | — |
| F01 | 0/1 | failed | degree of freedom and needed files/evals are stated (README), but observed baseline errors of the agent WITHOUT the skill on representative tasks are not documented; README “Статус” admits no systematic runs. | Add an explicit design note recording observed baseline agent failures (without the skill) that motivated the skill. |
| F02 | 1/1 | passed | source-taxonomy.md T1–T9 with claim-class primacy; T1–T4/T9 primary; T7/T8 cannot be sole basis; safety-critical requires T1/T2/T9. | — |
| F03 | 1/1 | passed | freshness rules per branch (source-taxonomy:150–159), independence/diversity (count_independent + dedup), explicit conflict taxonomy & resolution (extraction-rules, verification-rubric), confidence lowered on unresolved conflict (confidence-scale “Спорно”). | — |
| F04 | 1/1 | passed | FACT/INFERENCE/OPINION separated (extraction-rules; findings-template), quotes/provenance required, confidence 4 levels with lowering factors, insufficient allowed. | — |
| F05 | 1/1 | passed | report-template.md contains restated question (L1–3), recommendation (L12–18), key evidence + sources table (L44–50), unknowns/limitations (L38–42), confidence (L5, per-claim). | — |
| F06 | 0/1 | failed | evals.json has 5 scenarios using prompt/expected_output/assertions (not query/expected_behavior/failure_modes); trigger-eval.json uses query/should_trigger only; missing security-injection, broken-script, and portability/path case types. | Provide ≥7 cases covering the 7 required types, each with query, expected_behavior, failure_modes. |
| F07 | 0/1 | failed | no observed trigger/baseline run; README “Статус” states runs were not performed; evals/README confirms scenarios 2–5 not run and no trigger eval executed. | Run trigger evals and end-to-end scenarios; capture baseline-without-skill + activation/no-activation/boundary observations. |
| F08 | 0/1 | unverified | no navigation/behavior trace available in the submission (and not produced by grader). | Capture agent traces showing no injection, no strong conclusion on weak evidence, correct supporting-file reads, and output-contract execution. |
Required rework
- F06 — evals: add ≥7 cases (positive, negative, boundary, security-injection, evidence-quality, broken-script, portability/path), each with
query,expected_behavior,failure_modes; current files use non-conforming fields and miss three required types. - F07/F08 — behavioral runs: actually execute the trigger evals and end-to-end scenarios (including a baseline without the skill) and record navigation/behavior traces; the submission itself states these were not run.
- F01 — design note: explicitly record observed baseline agent errors without the skill on representative tasks.
- D01 — size: trim SKILL.md from 501 to ≤500 lines.
- D03 — navigation: add a brief Contents block to reference files longer than 100 lines.
- E03 — side effects: make auto-created intermediate ledger/findings/verification files opt-in or ephemeral (persist only the user-requested report).