Developer Hub · schema diber-evaluation-v0.9.0

Integrate, test, and audit with DIBER Core.

Interactive playground, installation, CLI, API reference, JSON contracts, metric formulas, repeated-run statistics, comparison methodology, and the complete report schema—all in a single technical reference.

Open the Core Tester Explore a visual report

Deterministic

The same input and configuration produce the same calculation.

No AI judge

The Core computes metrics from structured classifications and ground truth.

Run-aware

Aggregations, standard deviation, and intervals across identical configurations.

Auditable

Metrics, denominators, transitions, and relations remain fully inspectable.

Live Core Tester

Run the engine directly in your browser.

The default example includes two configurations, repeated runs, context documents, relations, and behavioral data. You can replace it with your own JSON and inspect both the overview and the full report.

1. ground_truth.json

2. evaluations.json

Click Run Engine to generate the report.

3. report.json

How the Core works

DIBER separates the test contract, point-by-point observations, and computation. This prevents a metric from obscuring how the underlying assessment was constructed.

Ground Truth

Material points, importance, source documents, and semantic relations.

Runs

Model, prompt, documents actually available to the run, and behavioral metadata.

Evaluations

For each point: coverage + status, plus any input assessment.

Report

Metrics, identical-configuration groups, comparisons, transitions, and the model/prompt matrix.

NPM & Quickstart

The contract shown on this page is designed for a JavaScript/Node.js library that can be integrated into backends and test suites.

npm install @anonipro/diber-core

import { buildExperimentReport, CB_CONFIG } from '@anonipro/diber-core';

const report = buildExperimentReport({
  experiment,
  groundTruth,
  evaluations
});

console.log(report.runs[0].metrics.effective_information);

Runtime configuration

CB_CONFIG is exposed to allow explicit weight overrides. If you change the weights, version the test configuration as well to preserve comparability.

CLI and CI/CD quality gates

Generate a deterministic report and apply explicit thresholds. The pipeline makes decisions based on declared numbers and rules, not on a second model.

npx diber evaluate --truth ./tests/gt.json --evals ./tests/runs.json --out ./report.json

npx diber assert --max-hallucination 0.02 --max-math-error 0.0 --max-extra-hallucination-penalty 0.5 --min-quality 0.90 ./report.json

import { readFileSync } from 'node:fs';

const report = JSON.parse(readFileSync('./report.json', 'utf8'));
const latest = report.runs.at(-1);

if (!latest?.valid) throw new Error('DIBER: invalid run');
if (latest.metrics.gt_hallucination_rate > 0.02) {
  throw new Error('DIBER gate: gt_hallucination_rate > 2%');
}
if (latest.metrics.math_error_rate > 0.0) {
  throw new Error('DIBER gate: math_error_rate > 0');
}
if (latest.metrics.addressed_quality < 0.90) {
  throw new Error('DIBER gate: addressed_quality < 90%');
}

API Reference

Main Core API surface represented in the v0.9.0 material.

buildExperimentReport({ experiment, groundTruth, evaluations })

Full report

Evaluates runs and builds identical-configuration groups, comparisons, point-by-point transitions, and the model/prompt matrix.

evaluateExperiment({ experiment, groundTruth, evaluations })

Run evaluation

Returns valid/invalid runs and the primitive and derived metrics for each execution.

calculateRunMetrics({ run, groundTruth, evaluation })

Single run

Normalizes evaluations, calculates metrics, counts, relations, and availability, and preserves the associated assessments.

CB_CONFIG

Configuration

Supported dictionaries, importance weights, and numeric scores used by the calculation.

Schema: Ground Truth

Defines the material scope of the test. The Core indexes points by point_id.

Field	Type	Req.	Description
points	Array<Object>	Yes	List of material points that can be evaluated.
points[].point_id	String	Yes	Unique ID. It must match the ID used in evaluations.
points[].importance	String	No	`Critical`, `High`, `Medium`, `Low`. If omitted, the Core uses the `undefined` weight.
points[].weight	Number	No	Numeric override of the weight derived from `importance`.
points[].source_document_id	String	No	Links the point to its source document to calculate `available_to_model`.
relations	Array<Object>	No	Semantic/logical relations between points.
relations[].source_point_id	String	Yes*	Source point of the relation.
relations[].target_point_id	String	Yes*	Target point of the relation.
relations[].type	String	Yes*	`depends_on`, `supports`, `contradicts`. *Required when the relation is present.

{
  "points": [
    { "point_id": "PT-001", "importance": "Critical", "source_document_id": "DOC-A" },
    { "point_id": "PT-002", "importance": "High",     "source_document_id": "DOC-B" },
    { "point_id": "PT-003", "importance": "Medium" }
  ],
  "relations": [
    { "source_point_id": "PT-001", "target_point_id": "PT-002", "type": "depends_on" }
  ]
}

Schema: Experiment / Runs

Describes the batch and the operating conditions of each execution.

Field	Type	Req.	Use
experiment_id	String	No	Batch ID. If omitted, the report uses `null`.
runs	Array<Object>	Yes	List of executions. The Core requires this to be an array.
runs[].run_id	String	Yes	Key used to retrieve `evaluations[run_id]`.
runs[].model_id	String	No	Used in aggregations, comparisons, and the model/prompt matrix.
runs[].prompt_id	String	No	Used in aggregations, comparisons, and the model/prompt matrix.
runs[].context_document_ids	Array<String>	No	Documents actually available to the run; used to derive `available_to_model`.
runs[].behavioral_test	Object	No	Numeric behavioral dimensions and/or observed transitions.

Schema: Evaluations

Dictionary indexed by run_id. Each entry contains any input assessment and the point-by-point classifications.

Field	Type	Req.	Description
evaluations[run_id]	Object	Yes*	Required for the run to be processed; a run without an entry is skipped.
input_data_assessment	Object	No	Input metadata, such as `overall_exposure`, the presence of sensitive data, or context notes.
evaluations	Array<Object>	Yes	Array of classifications.
evaluations[].point_id	String	Yes	Must match a point in the Ground Truth; unknown IDs generate warnings.
evaluations[].coverage	String	No	If omitted, it is normalized to `unrated`.
evaluations[].status	String	No	If omitted, it is normalized to `unrated`.
extra_claims	Array<Object>	No	Unrequested statements added by the model, not covered by Ground Truth.
extra_claims[].status	String	Yes*	Status of the extra claim: `hallu`, `over`, `error`.
extra_claims[].weight	Number	No	Weight of the extra claim for penalty calculation.

{
  "experiment": {
    "experiment_id": "EXP-001",
    "runs": [
      {
        "run_id": "run-01",
        "model_id": "model-a",
        "prompt_id": "prompt-v1",
        "context_document_ids": ["DOC-A", "DOC-B"],
        "behavioral_test": { "tone_adaptation_rate": 0.80 }
      }
    ]
  },
  "evaluations": {
    "run-01": {
      "input_data_assessment": { "overall_exposure": "low", "personal_data": true },
      "evaluations": [
        { "point_id": "PT-001", "coverage": "addressed", "status": "correct" }
      ],
      "extra_claims": [
        { "status": "hallu", "weight": 1.0, "excerpt": "Invented unrequested data" }
      ]
    }
  }
}

Coverage / Status dictionary

Coverage answers “was the point addressed?”, while Status answers “how was it handled?”. The two dimensions are not interchangeable.

Coverage

addressed — the point was materially addressed.
should — required but omitted.
gap — a recognized information gap.
out_context — the source was unavailable to the run.
not_relevant — the source was available, but the point was not required.
unrated — not classified.

Status / score

correct → 1.00
partial → 0.50
inference → 0.25
over → 0.25
error, math_error, hallu, omission → 0.00
missing → 1.00
na, unrated → null

Context rule

In the v0.9.0 playground, if a run declares context_document_ids, a point whose source_document_id is absent from that context is marked available_to_model: false. If it had been marked as addressed, normalization changes it to out_context.

Metrics & denominators

Metrics are not opaque labels: the denominator is part of their meaning. The formulas implemented in the Core Tester are shown here.

coverage_raw

addressed_count / (addressed_count + should_count)

Unweighted coverage of materially relevant points.

coverage_weighted

Σ weight(addressed) / Σ weight(addressed + should)

Gives greater influence to higher-weight points.

addressed_quality

Σ statusScore × weight / Σ weight, coverage=addressed only

Measures the quality of what the response actually addressed.

usable_information

Σ weight(correct | partial) / Σ weight(addressed)

Weighted share of addressed points classified as correct or partial.

effective_information

Σ addressed(weight × statusScore) / Σ weight(addressed + should)

Combines coverage and quality into a single effective measure.

defect rates

Σ weight(defect matching coverage) / Σ weight(addressed + should)

hallucination_rate, error_rate, math_error_rate use addressed defects; omission_rate uses coverage=should.

relation_consistency

consistent_relations / total_relations

Null when there are no evaluable relations.

Null does not mean zero

When no denominator exists or a dimension is not applicable, the Core returns null. This prevents “not measurable” from being turned into “zero performance”.

Logical relations

Relations make it possible to check structural consistency between premises and conclusions, rather than looking only at the isolated status of each point.

depends_on

Dependency

In Core v0.9.0, a target marked correct is inconsistent when the source is error, hallu, omission or missing.

supports

Support

A relation is inconsistent when the source is hallu.

contradicts

Contradiction

The type is supported by the schema. In the supplied Core Tester, it does not yet introduce an additional rule beyond tracking the relation.

Context availability & Input Exposure

DIBER distinguishes what was materially required from what was actually available to the model.

available_to_model

A point can declare source_document_id.
A run can declare context_document_ids.
If the source document is not present in the run context, available_to_model=false.
This value is preserved in normalized_evaluations.

input_data_assessment

This is an optional object associated with an individual evaluation/run.
It generates the new input_profile to track overall_exposure, personal_data, special_category_data, and confidential_data.
The Core preserves it in the report for data security policies.

Repeated runs & aggregate statistics

Runs with the same combination of model, prompt, and context documents are grouped under groups.exact_configurations.

n

Number of valid numeric observations.

mean / min / max

Center and extrema of the metric.

range / variance / stddev

Absolute and statistical dispersion.

CI 95%

mean ± 1.96 × stddev / √n, clamped to the 0–1 range.

Statistical note

The implemented interval uses the 1.96 normal approximation. With very small samples, it should be treated as an operational indicator rather than a substitute for a more sophisticated inferential methodology.

A/B methodology audit

Before interpreting a delta, DIBER makes explicit what changed between the two runs.

replication

0 variables changed

Same configuration. Useful for measuring variability across runs.

strictly_controlled

1 variable changed

Controlled comparison: only the model, prompt, or document set changes.

multidimensional

2+ variables changed

The delta can still be calculated, but it should not be causally attributed to a single change.

Variables tracked in the Core Tester

model, prompt, context_documents. Each comparison also includes metric_deltas, counts of status_transitions and point-by-point transitions.

Model / Prompt Matrix

The report groups runs by model_id + prompt_id and calculates the same statistics for numeric metrics.

model_prompt_matrix: [
  {
    key: "model-a__prompt-v2",
    model_id: "model-a",
    prompt_id: "prompt-v2",
    run_count: 5,
    metrics: {
      effective_information: { n, mean, min, max, range, variance, stddev, ci_lower, ci_upper },
      ...
    }
  }
]

Behavioral profile

Runs can include a behavioral_test. The page automatically aggregates numeric dimensions and counts any explicit categorical transitions.

Numeric dimensions

Examples already used in the material: irony_recognition_rate, tone_adaptation_rate. Any numeric property is aggregated using n/mean/min/max/range/variance/stddev/CI.

behavior_transition

When present as a string, it is counted by frequency in the aggregated profile. It is not inferred automatically by an LLM.

Principle

The Core does not invent the semantics of a behavioral dimension: it aggregates the structured signals supplied by your evaluation pipeline.

Full CB_CONFIG

Central configuration used by the v0.9.0 Core Tester.

const CB_CONFIG = {
  version: "0.9.0",
  coverage: {
    addressed: true, should: true, gap: true,
    out_context: true, not_relevant: true, unrated: true
  },
  status: {
    correct: true, partial: true, omission: true,
    inference: true, over: true, error: true,
    math_error: true, hallu: true, missing: true,
    na: true, unrated: true
  },
  defaultImportanceWeight: {
    Critical: 2.00,
    High: 1.00,
    Medium: 0.50,
    Low: 0.25,
    undefined: 1.00
  },
  statusScore: {
    correct: 1.00,
    partial: 0.50,
    inference: 0.25,
    over: 0.25,
    error: 0.00,
    math_error: 0.00,
    hallu: 0.00,
    omission: 0.00,
    missing: 1.00,
    na: null,
    unrated: null
  },
  coverageScore: {
    addressed: 1.00,
    should: 0.00,
    gap: null,
    out_context: null,
    not_relevant: null,
    unrated: null
  }
};

Validation, errors, and warnings

Validation distinguishes errors that prevent a run from being processed from warnings that flag inconsistencies while keeping the data inspectable.

Errors

evaluation_missing
ground_truth_missing
evaluations_not_array
ground_truth_points_not_array
point_id_missing
experiment.runs must be an array (experiment-level exception)

Warning

unknown_point:<id>
unknown_coverage:<value>
unknown_status:<value>

Run without evaluations

In the v0.9.0 flow, a run present in experiment.runs but without evaluations[run_id] is ignored during evaluateExperiment; it is not automatically converted into an invalid run.

Report output: diber-evaluation-v0.9.0

Complete structure produced by the Core Tester included on this page.

Top level

schema, deterministic, ai_judge
experiment_id, run_count
runs
groups.exact_configurations
comparisons
model_prompt_matrix
behavioral_profile
input_exposure_summary

For each run

valid + validation
metrics
counts.coverage + counts.status
relations
normalized_evaluations
input_data_assessment
behavioral_test
context_document_ids

{
  "schema": "diber-evaluation-v0.9.0",
  "deterministic": true,
  "ai_judge": false,
  "experiment_id": "EXP-DEMO-01",
  "run_count": 4,
  "runs": [ ... ],
  "groups": { "exact_configurations": { ... } },
  "comparisons": [
    {
      "run_a": "baseline-01",
      "run_b": "candidate-01",
      "methodology": {
        "changed_variables": ["prompt"],
        "comparison_type": "strictly_controlled",
        "is_valid_controlled": true
      },
      "metric_deltas": { ... },
      "status_transitions": { ... },
      "point_transitions": [ ... ]
    }
  ],
  "model_prompt_matrix": [ ... ],
  "behavioral_profile": { ... },
  "input_exposure_summary": { ... }
}

View this model as a visual report →DIBER Free →