Knitify Pet Health Veterinary

Abstract

We evaluated Knitify Pet Health Veterinary against four Gemini configurations on 30 veterinary clinical queries. The veterinary domain exposes the most severe citation hallucination gap of any domain tested. Gemini 2.5 Flash fabricates 94% of veterinary citations. Even Gemini 3.0 Pro hallucinates 48% of references. Knitify maintains 98-100% citation fidelity across all tiers and query categories.

This has direct implications for veterinary practice: species-specific dosing, drug interactions, and contraindications differ significantly from human medicine, and fabricated references to non-existent veterinary studies cannot be verified or acted upon.

1. Citation Fidelity

Citation fidelity measures whether each cited reference is a real paper that exists in PubMed and is relevant to the claim it supports.

How citation fidelity is measured: Each reference cited by Gemini is checked programmatically. DOIs are resolved via CrossRef and PMIDs via PubMed to retrieve the real paper. An independent AI verifier then compares the resolved paper against what was claimed — checking whether the topic, authors, and study match. If the DOI returns a 404 (paper does not exist) or the resolved paper is on a different topic, the citation is marked as hallucinated. Knitify citations are verified by the model's built-in quality assurance layer.

Citation Fidelity by Query Difficulty

2. Reference Density

Average number of unique peer-reviewed sources cited per response.

3. Reference Quality & Recency

Breakdown of citation volume, recency, and journal breadth across all 7 systems.

Journal Quality

Knitify top journals (verified via PubMed): Journal of Veterinary Internal Medicine (18), Journal of Feline Medicine and Surgery (12), Veterinary Record (8), Journal of the American Veterinary Medical Association (7).

Neither Knitify nor Gemini cited NEJM for this domain — veterinary queries draw from specialized journals like JVIM and JFMS rather than general medical journals.

4. Clinical Safety

Binary metric: no wrong dose >50%, no missed major contraindication, no wrong interaction severity.

5. Speed of Answer

6. Knitify Tier Comparison

7. Head-to-Head Comparisons

Direct comparisons between matched tiers — same-class models, all metrics.

Knitify Fast vs Gemini 3.0 Flash

Knitify Premium vs Gemini 3.0 Pro

Knitify Fast vs Gemini 3.0 Flash + Google Search

Knitify Premium vs Gemini 3.0 Pro + Google Search

8. Summary

Species-specific accuracy matters. Drug dosing, toxicity thresholds, and metabolism differ between dogs, cats, and humans. Gemini 2.5 Flash makes dangerous errors (wrong doses, missed species contraindications) on 25% of complex pharmacotherapy queries. Knitify Fast/Premium achieve 100% clinical safety.

Verified references for client communication. Every Knitify citation links to a real PubMed paper that veterinarians can share with pet owners or referring specialists — no risk of linking to non-existent papers.

17 verified references per Premium response. Deep coverage of veterinary literature for case workup, client education, and continuing education.

Appendix A: Evaluation Setup

A.1 Benchmark Design

Seven systems were evaluated: three Knitify tiers (Fast, Standard, Premium) and four Gemini configurations (2.5 Flash, 2.5 Pro, 3.0 Flash, 3.0 Pro). All systems received identical queries. Gemini systems were prompted to cite veterinary peer-reviewed sources with author, title, journal, year, and DOI or PubMed link.

A.2 Evaluation Judge

Clinical safety scored by an independent AI judge. Citation fidelity verified by checking each reference against PubMed — independent of the AI judge.

A.3 Citation Verification

Each cited reference is checked against PubMed to confirm it is a real, on-topic paper. A citation passes if (a) the identifier resolves to an existing paper and (b) the paper is relevant to the claim. Knitify citations are verified by the model's built-in quality assurance layer.

A.4 Query Tiers

• Common (8 queries): Core veterinary internal medicine (e.g., feline hyperthyroidism, canine epilepsy)
• Complex (8 queries): Species-specific pharmacotherapy and drug interactions (e.g., NSAID GI toxicity in dogs, tramadol metabolism differences)
• Niche (7 queries): Therapeutic nutrition (e.g., renal diets for CKD, MCT diets for epilepsy)
• Emerging (7 queries): Novel veterinary treatments (e.g., GS-441524 for FIP, stem cell therapy for OA)

A.5 Gemini Prompt

All Gemini systems received the following prompt template for each query:

You are a medical research assistant. Answer the following research question thoroughly.
Support every claim with citations to peer-reviewed sources. For each citation, include:
- First author et al.
- Paper title
- Journal name
- Year of publication
- DOI or PubMed link if available

Target approximately [TARGET_WORDS] words for the main answer (excluding references).
Format your references in a numbered list at the end.

Research question: [QUERY]

Gemini models were called via the Gemini API (generativelanguage.googleapis.com) with the prompt above. No search grounding or retrieval tools were enabled — responses are generated entirely from model parameters.

Target word counts were matched to the corresponding Knitify tier to ensure comparable output length.

Appendix B: Test Queries

B.1 Common — Internal Medicine (8 queries)

B.2 Complex — Pharmacotherapy (8 queries)

B.3 Niche — Therapeutic Nutrition (7 queries)

B.4 Emerging — Novel Treatments (7 queries)