Metric dictionary

Skills Metric Dictionary

Canonical Skills analytics metric definitions, source precedence, edge cases, and dashboard rules.

Skills Metric Dictionary

This dictionary defines the MVP Skills analytics metrics before API, dashboard, and recommendation work hard-code local meanings. It is forward-looking from the configured analytics start date and is scoped to agent-touched Azure Boards PBIs/tasks plus their immediate parent/child context.

See Agent Analytics Metadata Contract for the metadata fields and Analytics Data Governance for field-level privacy rules.

Source Priority

Use the narrowest reliable source for each metric:

Valid agent-analytics:v1 metadata blocks parsed from Azure Boards comments.
Normalized collector records emitted by scripts/agent-analytics-collector-lib.mjs.
Azure Boards snapshot fields for title, type, state, parent ID, URL, changed date, and limited revision context.
Collector run records for freshness, parse failures, and pre-collection counts.
Recommendation records generated from normalized or aggregate dashboard evidence.

Do not infer agent work from Azure Boards identity alone. Agents often write through a human user's Azure DevOps identity, so explicit metadata wins when it exists.

Cross-Cutting Rules

The official MVP analytics window starts at analyticsStartAt. Current dev defaults use 2026-05-16T00:00:00Z.
pre-collection means the work item existed or changed before the analytics window. It is distinct from no agent usage.
Missing data is not zero. Unknown token counts, unknown attribution, missing completion evidence, and unsupported host fields must remain unknown or unavailable.
Event-level metrics count metadata events. Work-item metrics deduplicate by Azure Boards work item ID and use the latest normalized record unless the metric says otherwise.
Dashboard labels should say when a number means "with valid Skills metadata" instead of implying all Azure Boards work.

Metric Definitions

Metric	Source Of Truth	Calculation Rule	Grain	Missing Or Zero Rule	Known Limitation
Metadata events collected	Collector run record	`acceptedBlocks` after parser validation and dedupe.	Run, project, date range	Zero means no accepted metadata blocks in the queried run/window.	Does not count rejected or malformed comments.
Parse failures	Collector run record and `parseFailures`	Count rejected metadata blocks by safe error code.	Run, project, work item	Zero means parser saw no invalid blocks in scope.	Failure records must not include raw comment text.
PBI count with Skills metadata	Normalized records	Unique latest work items where `workItemType` is `Product Backlog Item`.	Work item, project, date range	No record means no valid Skills metadata was collected for that PBI.	Not a total Azure Boards PBI count.
Task count with Skills metadata	Normalized records	Unique latest work items where `workItemType` is `Task`.	Work item, project, date range	No record means no valid Skills metadata was collected for that task.	Not a total Azure Boards task count.
Touched by agent	Normalized records	Work item has at least one valid metadata event whose `actorKind`, `createdByKind`, `completedByKind`, or inferred `attributionKind` is `agent` or `mixed`.	Work item, PBI, task	Unknown attribution remains unknown, not human.	A human-created item can still be agent-touched.
Agent-created	Metadata payload plus normalized record	Count work items with `createdByKind=agent` or a `work_item.created`/`task.started` event whose actor is agent and creation evidence is explicit.	Work item	Missing `createdByKind` is unknown.	Does not use Azure Boards created-by identity alone.
Human-created	Metadata payload plus revision context	Count work items with `createdByKind=human` or clear revision evidence outside the agent workflow.	Work item	Missing evidence is unknown.	Conservative because agent and human may share the same board identity.
Mixed workflow	Normalized record	Count work items whose `attributionKind` is `mixed`, or whose metadata and revision context show both human and agent participation.	Work item	Do not split one mixed item into one human and one agent item.	Mixed does not explain direction of handoff by itself.
Unknown attribution	Normalized record	Count work items whose `attributionKind` is `unknown`.	Work item	Unknown is a first-class category.	High unknown counts usually indicate missing or unsupported metadata.
Pre-collection work	Collector `preCollectionWorkItems`	Count related work items with `changedAt` before `analyticsStartAt`.	Work item, project	Pre-collection never counts as no agent usage.	Does not reconstruct historical agent activity.
Completed by agent	Metadata payload	Count explicit `task.completed` or workflow completion evidence with `completedByKind=agent` or `actorKind=agent`.	Event, task	Missing completion metadata is unknown.	Current MVP does not infer completion from board state alone.
Token estimate total	Metadata `tokenEstimate`	Sum `totalTokens` only where `tokenEstimate.available=true`.	Event, model, agent host, skill version	Unavailable token estimates are excluded from totals and counted separately.	Estimates are not billing records and may be uneven across hosts.
Token estimate unavailable	Metadata `tokenEstimate`	Count records where `tokenEstimate.available=false` or token support is absent.	Event, host, model	This is a support/coverage signal, not a usage number.	A host with no token counters can still be fully supported for metadata.
Cost estimate	Derived from tokens and approved pricing table	Multiply available token components by a versioned pricing table when one is approved.	Event, model, date range	If tokens or pricing are unavailable, cost is unknown.	No MVP cost value is an official invoice or billing record.
Cycle impact	Azure Boards iteration/state fields plus normalized events	Compare accepted metadata activity and completion outcomes within a sprint/cycle once iteration data is available.	Sprint, team, project	Before iteration fields exist, report unavailable.	Do not use pre-collection gaps as process problems.
Recommendation confidence	Recommendation record	Use stored confidence from the deterministic or LLM recommendation job, with cited evidence.	Recommendation, date range	Missing confidence is unknown.	Confidence is an evidence-quality signal, not probability of correctness.
Collector freshness	Collector run record	Difference between current time and latest successful run or watermark.	Project, collector scope	Missing run means collector status is unknown.	Requires scheduled collector run history.

Edge Cases

Agent-created PBI with human-edited tasks: count the PBI as agent-created, count each task according to its own metadata/revision evidence, and count the parent workflow as mixed only when dashboard rollup explicitly combines parent and child activity.

Human-created PBI with agent-created tasks: keep the PBI human-created or unknown, count agent-created tasks separately, and mark the PBI as agent-touched when child activity is included in the rollup.

Agent-touched work with missing token estimates: keep attribution metrics, set token support to unavailable, and exclude the item from token totals. Do not show zero tokens.

Malformed metadata: count a parse failure, omit the malformed block from normalized dashboard records, and surface safe error codes for operators.

Related pre-collection work item: show pre-collection when it appears through a parent/child relation, and do not include it in forward-looking agent adoption rates.

API And Dashboard Requirements

Summary APIs should include analyticsStartAt and enough run metadata for the UI to label forward-looking windows.
Work-item APIs should expose unknown and pre-collection states instead of dropping them silently.
Token and cost cards should show available, unavailable, and not-applicable states distinctly.
Recommendation inputs should use these metric definitions and the governance policy rather than raw Azure Boards comments or descriptions.
Dashboard copy should avoid "all PBIs" or "all tasks" unless the API is actually querying all Azure Boards work.