Accuracy Benchmarks, Hallucination Guards, and Latency SLA

Evaluation frameworks, statutory accuracy benchmarks, retrieval-augmented verification (RAG), and latency benchmarks.

1. Overview & Technical Architecture#

To ensure mission-critical reliability, custom models undergo continuous evaluation against our proprietary EthioBiz-Bench test suite.

1. Benchmark Evaluation Results#

Benchmark Category	Sovereign Model	Generic Base Model	Pass Threshold	Status
Tax Calculation Accuracy	99.4%	68.2%	98.0%	PASSED
Store Voucher Workflow Logic	98.7%	54.1%	95.0%	PASSED
Amharic Grammar & Coherence	94.6%	71.3%	90.0%	PASSED
Hallucination Rate (Financial)	0.4%	14.8%	<1.0%	PASSED

2. Hallucination Safeguards#

Every numerical figure returned by the model in financial contexts is verified against the underlying PostgreSQL ledger via an automated post-generation checker. Discrepant figures trigger an automatic regeneration with zero temperature.

2. Prerequisites & Infrastructure Requirements#

Runtime Environment: Python 3.12+, PostgreSQL 16 with pgvector extension.
Security Access Group: Ashewa AI / Administrator (Technical ID: muk_ai.group_ai_manager)
Hardware Profile: Minimum 16 GB System RAM (CPU inference) or NVIDIA GPU with 8+ GB VRAM.

3. Step-by-Step Configuration & Verification#

Step 1: Navigating to Model Management#

Access your SmartERP instance at https://smarterp.et/odoo.
Open Settings → Technical → Ashewa AI → Model Registry.
Select the custom model profile.

{screenshot:54-evaluation-benchmarks-and-performance-step1-navigation}

Step 2: Testing Endpoint Health & Response Verification#

Click the Run Diagnostic Health Check button.
Verify token generation speed (tokens/sec) and latency benchmarks.
Review sample response in the diagnostic console.

{screenshot:54-evaluation-benchmarks-and-performance-step2-verification}

4. Advanced Configuration Points#

Automated Tasks: muk_ai.ir_cron_model_health_monitor runs every 15 minutes.
Multi-Tenant Boundaries: Custom model fine-tuning adapters are strictly segregated per tenant database.

5. Comprehensive Troubleshooting Matrix#

Symptom / Error	Probable Root Cause	Resolution Action
`Connection refused: Model server unreachable`	Inference daemon stopped or port blocked by firewall.	Restart model service (`systemctl restart vllm`) and verify firewall rules.
`Token limit exceeded on input prompt`	Input context exceeded window size.	Enable context truncation in model profile settings.

Last updated: 2026-10-30