# Rating AI Use hosted AI or connect a device in Agents → Connections → Local AI. Use your own question to evaluate text with people, AI, or both. Training is optional. ## Supported tasks The current GLiNER2.5 multilingual evaluator supports custom binary questions for English and German text. Define a question and two answers, such as Yes / No or Supported / Unsupported. You do not need a training dataset to start. Examples include checking a summary against a source, reviewing wording against a style guide, or checking a document against a stated requirement. Add context or reference material when useful. Whether the model performs well on your task must be tested. This version evaluates text, within the model’s 512-token input limit. It does not execute code or inspect images, audio or video. Human review remains available for the materials supported by its submission flow. Customer-reply approval remains an optional preset. Older registered models keep their original question; enabling custom questions requires a compatible model and your explicit permission. ## Start in Chat [Open Chat](/chat), sign in and choose Send and rate for your first prompt. No installation, model-provider key, model selection or reviewer invitation is required. The saved answer and advisory rating appear together. Existing human-review requirements still apply. You can keep using hosted AI. Your computer does not need to stay on, and the displayed usage allowances still apply. Use public, synthetic or safely redacted material during the Alpha. An external agent the owner connects starts with the same hosted AI review. Creating its connection message allows Rating AI to process the agent’s checked replies; it does not permit training. To use another model, open Agents → Setup → AI settings. If an older model offers Enable custom questions, choose it to authorize that model. ## Check an answer in Chat The saved review question measures one criterion, such as whether an answer follows the request. It does not establish that every factual claim is correct. Simple arithmetic equations are checked separately with an exact calculator; unsupported notation and non-arithmetic meaning remain unchecked. Choose Check facts online to request an online assessment. After your permission, the selected answer and its review context go to the configured OpenAI model and its web search through Vercel AI Gateway. This option is unavailable with local drafting or local ratings. The displayed usage limits apply. AI selects up to five statements and cites up to three retrieved sources. Factual claims can be supported, contradicted or left with insufficient evidence; opinions are not fact-checked. Open the source links to inspect the evidence yourself. The provider does not return source excerpts, so none are displayed. Selection can miss claims, sources can be wrong or outdated, and the evidence judgment is made by AI. These checks do not validate confidence, prove the whole answer correct or replace required human review. ## Connect local AI Open Agents → Connections → Local AI and choose Connect a device. Create a connection code and follow the evaluator installation instructions. Enter the private, one-time code in the evaluator, start it and check its connection in RateLoop. Setup can be completed before your first Chat message. One connection can support Chat drafts, Chat ratings, or both. These choices apply only to Chat; external agents keep their saved AI settings. Chat’s AI settings link opens Connections. [Open Local AI](/agents/connections?view=local-ai) · [Install and run the evaluator](https://github.com/Noc2/rateloop-evaluator/blob/main/docs/operations.md) ## Run Chat ratings on your device Under Chat ratings, confirm the review question and select a compatible model, then explicitly allow it for future ratings. This changes future Chat ratings; it does not change answer generation, required human review or earlier results. Connecting does not authorize training or copy historical datasets. Content entered on this website is still stored by RateLoop. Future rating content, including its necessary context, is sent to the selected device. This is local AI execution, not a fully offline or customer-only storage workflow. Keep the device running while it rates work. If it goes offline, RateLoop does not silently switch to hosted AI. Reconnect it, or explicitly choose Allow hosted ratings for future work. Disconnect removes the device's access; earlier results retain their recorded model and processing location. ## Run Chat drafts on your device Under Chat drafts in Connections → Local AI, select a connected device and model, run the test prompt, then choose Allow for new chats. Start a new Chat, or explicitly allow an earlier conversation’s context to be sent to the selected device and model. Drafts remain stored by RateLoop. Rating and training permissions are separate; an offline device never triggers hosted fallback. ## Try your saved setup Open Agents → Setup → Try an example. Enter text and add optional context. The test uses the selected agent’s saved question and review balance. Check the displayed reviewer and allowance consequences before running it. Testing does not authorize training. - **Human review:** Your invited reviewers answer. No AI job is created and no evaluator connection is needed. - **Rating AI:** The connected model rates the case. No human review is assigned; the result is labeled as AI. - **AI + human:** Both review the same case. The AI answer is withheld until the human outcome, so reviewers can answer without seeing it. The agent’s fixed question and answer labels remain in force. This form requires a saved fixed question. Test policies with a question per request through the connected agent. The test runs the enabled review types once and uses one review allowance; it does not test automatic sampling. The test does not change the saved policy. An AI rating is advisory, not human approval, qualified human assurance, or proof that a required human review happened. Missing inputs, permission or a compatible model produce an explicit error, not a completed rating. ## Guide AI with examples Choose Add example beside your AI question to supply up to four short texts with their expected answers. Examples guide that question without changing model weights. They are saved with the question and sent to the evaluator, so use public, synthetic or safely redacted material. Examples share the 512-token limit with the question, text and context. Changing an example creates a new evaluation definition; earlier calibration does not carry over. Keep comparison data separate from these examples. ## Set an agent’s review balance Independent AI review is a separate Rating AI evaluation of the submitted output. It does not mean simply trusting the AI that produced it. In the connected agent’s review settings, Impact of an AI mistake sets the starting human share. Saved AI balances use a fixed question and matching answer labels. New agents start with independent AI review only once the owner grants processing permission; human review is optional and can be added at any time. - Independent AI review only: Rating AI rates every eligible case from the start. No human review or initial human calibration is required. - Between the endpoints: human review starts at the displayed share, and independent AI review covers the remaining eligible cases. Evidence, sampling and drift rules can adjust human coverage above its minimum. - Human only: every eligible case goes to people. No AI evaluation is created. The share is a target across eligible cases, not a guarantee for a small batch. If adaptation is unavailable, it stays fixed. Existing human-only policies keep their settings until explicitly changed. Assigned work and publication permissions stay unchanged. The agent must supply the text for evaluation; hashes alone cannot be rated. Context and reference material are optional. With private learning enabled, mixed review can compare selected human outcomes with AI while keeping AI answers hidden from reviewers. ## Train with your own examples In Setup, open Train with your examples. The form shows whether your workspace uses hosted training or a connected device. Check the processing location before granting permission. Use Back to setup to return without another row of tabs. 1. Add examples: define the question, upload CSV or JSONL, and preview the field mapping and labels. Use at least three source groups; related documents or conversations must share a group. Unlabeled reference material is not automatically training data. 2. Import only public, synthetic or safely redacted examples that you have the right to use. Grant private-learning permission separately from AI evaluation. Imports are limited to 2 MB and up to 2,000 rows; the connected trainer may allow fewer. 3. Compare the original model on held-out examples. Then choose Train a private candidate. Training needs enough examples of each answer for a separate validation set and does not change the active model. 4. Review the comparison on the same benchmark, including sample count, agreement interval and mistakes. A candidate with insufficient evidence or no improvement is marked for testing only. Choose Activate candidate explicitly. Restore previous model provides rollback. Imported owner, AI-assisted and synthetic labels are reference judgments, not qualifying human review. AI-only evaluations do not create human labels. Supplied instructions and reference documents do not train the model unless you separately start an authorized training run. Imported examples expire after 30 days. Delete dataset removes stored examples, stops their training jobs and retires affected candidates. Revocation blocks future authorized use; it does not erase an example’s influence from existing weights. ## Learn from future reviews In Training, choose the saved question and allow private learning from future reviews. A compatible hosted service or connected device collects authorized text, context, evidence and eligible human judgments for that question. Existing history is excluded; reviewers use their normal form. Batches need at least 200 new source groups, with separate later examples for comparison. Training is limited to one batch per day and 200 steps. Check the displayed evidence count and waiting reason. Each candidate is compared with the current model and reference methods; you explicitly activate an improved candidate. Stop learning halts future collection and pending training. Examples expire after 30 days. Deleting examples or withdrawing their permission makes affected datasets and candidates unavailable. An improvement on the saved question does not establish general factual accuracy or validated confidence. ## Start with public datasets The open-source evaluator includes a local preparation tool for HelpSteer2, HelpSteer3 Principle and OpenPII 1M. Choose one evaluation criterion, keep the source attribution, and review the prepared labels before importing. Nothing is downloaded or trained automatically. HelpSteer2 provides human scores; HelpSteer3 Principle uses AI-assisted judgments; OpenPII 1M is synthetic. Imported public data does not qualify a model for human-assurance decisions. Dataset licenses and source versions are recorded with the prepared files. [Prepare a public dataset](https://github.com/Noc2/rateloop-evaluator#public-datasets-and-local-diagnostics) ## Understand the measurements Model uncertainty shows how concentrated or divided the answer scores are. It is experimental score spread, not the probability that the evaluation is wrong. Raw scores are not calibrated probabilities; calibration, when available, applies only to its recorded model and question. Training comparisons measure agreement with the dataset’s labels on unseen examples. Their intervals reflect sample uncertainty. Test representative cases and inspect mistakes before relying on a model; integration tests establish operation, not accuracy. Check the number of examples for each answer. Balanced agreement gives each answer equal weight, so a common answer cannot hide failures on the rarer one. Results by answer shows which answers the model finds correctly and where it is undecided. Model fit in Evaluate and improve shows how often Rating AI agreed with reviewers on each model's reviewed answers. That agreement is context only: a model meets the review standard from human reviews alone. ## Choose where data is processed Website submissions pass through RateLoop’s workspace storage. The hosted base model runs on RateLoop-managed Railway infrastructure in Amsterdam without an external AI model provider. The website workflow is not offline. In the Alpha, use only public, synthetic or safely redacted material. Where hosted training is enabled, training and private candidates run on RateLoop-managed infrastructure in Amsterdam. Your computer can be off. Evaluation requests queue while that worker trains. A connected-device setup still requires that device online; existing datasets are not moved automatically. The standalone open-source evaluator can also run on your hardware with local inference and training. You operate that installation yourself. Sharing examples or model weights requires separate permission. [Install and run the evaluator](https://github.com/Noc2/rateloop-evaluator/blob/main/docs/operations.md) · [Open Setup](/agents/review-setup?view=settings) ## Learning permissions AI evaluation, training a workspace-private candidate, and improving a shared model are separate uses. Enabling evaluation or inviting reviewers does not authorize training. Private-learning permission does not authorize use for other workspaces. Shared model training is not enabled by this workflow. Any future shared-learning option requires a separate, explicit opt-in and defined data scope, retention, and withdrawal terms. The organization must also have the necessary rights in the material and address the rights of people it concerns. Operational records used to run and secure the service are not an automatic training license. Removing examples or withdrawing future permission does not by itself erase their influence from an already trained model; model deletion or retraining must be handled separately.