> ## Documentation Index
> Fetch the complete documentation index at: https://docs.heybee.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Get Rlhf Score Consistency

> Rater test-retest consistency from silently re-served pairs.

~3% of scoring submissions re-serve an already-scored pair (blind); the second
rating lands in sample_score_retests next to the original. This aggregates
per-axis agreement (Pearson r, MAE, drift) — the ceiling on how much any
reward model trained on these labels can be trusted.



## OpenAPI

````yaml /openapi.json get /v1/rlhf/experiments/{experiment_id}/score-consistency
openapi: 3.1.0
info:
  description: Customer-owned evaluation lifecycle API.
  title: HeyBee Public API
  version: 1.0.0
servers:
  - description: Production
    url: https://api.heybee.app
security: []
paths:
  /v1/rlhf/experiments/{experiment_id}/score-consistency:
    get:
      tags:
        - RLHF
      summary: Get Rlhf Score Consistency
      description: >-
        Rater test-retest consistency from silently re-served pairs.


        ~3% of scoring submissions re-serve an already-scored pair (blind); the
        second

        rating lands in sample_score_retests next to the original. This
        aggregates

        per-axis agreement (Pearson r, MAE, drift) — the ceiling on how much any

        reward model trained on these labels can be trusted.
      operationId: >-
        get_rlhf_score_consistency_v1_rlhf_experiments__experiment_id__score_consistency_get
      parameters:
        - in: path
          name: experiment_id
          required: true
          schema:
            format: uuid
            title: Experiment Id
            type: string
        - in: header
          name: authorization
          required: false
          schema:
            anyOf:
              - type: string
              - type: 'null'
            title: Authorization
        - in: header
          name: X-API-Key
          required: false
          schema:
            anyOf:
              - type: string
              - type: 'null'
            title: X-Api-Key
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/RlhfScoreConsistencyResponse'
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      deprecated: true
      security:
        - BearerAuth: []
        - ApiKeyAuth: []
components:
  schemas:
    RlhfScoreConsistencyResponse:
      properties:
        axes:
          items:
            $ref: '#/components/schemas/RlhfScoreConsistencyAxis'
          title: Axes
          type: array
        experiment_id:
          title: Experiment Id
          type: string
        retests:
          items:
            $ref: '#/components/schemas/RlhfScoreRetestPair'
          title: Retests
          type: array
        total_retests:
          title: Total Retests
          type: integer
      required:
        - experiment_id
        - total_retests
        - axes
        - retests
      title: RlhfScoreConsistencyResponse
      type: object
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          title: Detail
          type: array
      title: HTTPValidationError
      type: object
    RlhfScoreConsistencyAxis:
      description: Per-axis test-retest agreement from silently re-served pairs.
      properties:
        axis_key:
          title: Axis Key
          type: string
        exact_match_rate:
          title: Exact Match Rate
          type: number
        mae:
          description: Mean absolute difference |retest - original|.
          title: Mae
          type: number
        mean_diff:
          description: Mean (retest - original); positive = rater drifting upward.
          title: Mean Diff
          type: number
        'n':
          title: 'N'
          type: integer
        pearson_r:
          anyOf:
            - type: number
            - type: 'null'
          description: >-
            Pearson r between original and retest scores (None if n < 3 or zero
            variance).
          title: Pearson R
        within_one_rate:
          description: Fraction of retests within ±1 of the original.
          title: Within One Rate
          type: number
      required:
        - axis_key
        - 'n'
        - mae
        - mean_diff
        - exact_match_rate
        - within_one_rate
      title: RlhfScoreConsistencyAxis
      type: object
    RlhfScoreRetestPair:
      properties:
        axis_key:
          title: Axis Key
          type: string
        created_at:
          anyOf:
            - type: string
            - type: 'null'
          title: Created At
        original_score:
          title: Original Score
          type: number
        retest_score:
          title: Retest Score
          type: number
        sample_id:
          title: Sample Id
          type: string
      required:
        - sample_id
        - axis_key
        - original_score
        - retest_score
      title: RlhfScoreRetestPair
      type: object
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
              - type: string
              - type: integer
          title: Location
          type: array
        msg:
          title: Message
          type: string
        type:
          title: Error Type
          type: string
      required:
        - loc
        - msg
        - type
      title: ValidationError
      type: object
  securitySchemes:
    BearerAuth:
      bearerFormat: Supabase JWT
      scheme: bearer
      type: http
    ApiKeyAuth:
      in: header
      name: X-API-Key
      type: apiKey

````