llm

compare_llms

compare_llms(
    systems,
    data,
    metric,
    seeds=(0, 1, 2, 3, 4),
    *,
    test="welch",
    alpha=0.05,
    correction="holm",
    cache=None,
)

llm_judge

llm_judge(
    judge,
    rubric,
    *,
    parse=parse_score,
    template=default_template,
    seed=0,
)

Return a pointwise metric (output, reference) -> float that asks judge (a user-supplied, provider-agnostic judge(prompt, seed) -> reply) to score each output against rubric, parsing the reply to a float.