Knitify Medical Prescriber

Abstract

We evaluated Knitify Medical Prescriber against six configurations of Google's Gemini — including models with Google Search grounding enabled — on 30 medical research queries across four difficulty tiers. Knitify achieves 99-100% citation fidelity compared to 28-72% for Gemini (no grounding) and 63-92% with search grounding. Knitify cites 2-4× more verified references, and delivers first tokens in 20 seconds.

1. Citation Fidelity

Citation fidelity measures whether each cited reference is a real paper that exists in PubMed and is relevant to the claim it supports. This is verified independently — not by the AI judge.

How citation fidelity is measured: Each reference cited by Gemini is checked programmatically. DOIs are resolved via CrossRef and PMIDs via PubMed to retrieve the real paper. An independent AI verifier then compares the resolved paper against what was claimed — checking whether the topic, authors, and study match. If the DOI returns a 404 (paper does not exist) or the resolved paper is on a different topic, the citation is marked as hallucinated. Knitify citations are verified by the model's built-in quality assurance layer.

Citation Fidelity by Query Difficulty

2. Reference Density

Average number of unique peer-reviewed sources cited per response.

3. Reference Quality & Recency

Breakdown of citation volume, recency, and journal breadth across all 7 systems.

Journal Quality

Knitify top journals (verified via PubMed): JAMA (9), The Cochrane Database of Systematic Reviews (8), The New England Journal of Medicine (7), Diabetes Care (10).

Gemini journal claims: Gemini models collectively claim 175 citations to the New England Journal of Medicine across 30 queries. When we verified each DOI, 58% resolved to real NEJM papers — exclusively landmark trials the model memorized (e.g., EMPEROR-Preserved, LEADER, ARISTOTLE). The remaining 42% were fabricated DOIs with valid NEJM prefix format (10.1056/NEJMoa...) that do not correspond to any existing paper.

What Gemini gets right — and what it means: When Gemini's DOIs do resolve correctly, they are overwhelmingly high-citation landmark studies (median 726 citations per paper) that any specialist already knows. Knitify surfaces current papers researchers haven't seen yet.

4. Clinical Safety

Binary metric: 1 = no dangerous clinical errors (no wrong dose >50%, no missed major contraindication, no wrong interaction severity). 0 = safety-critical error present.

5. Speed of Answer

6. Knitify Tier Comparison

7. Head-to-Head Comparisons

Direct comparisons between matched tiers — same-class models, all metrics.

Knitify Fast vs Gemini 3.0 Flash

Knitify Premium vs Gemini 3.0 Pro

Knitify Fast vs Gemini 3.0 Flash + Google Search

Knitify Premium vs Gemini 3.0 Pro + Google Search

8. Summary

More entry points to the literature. Knitify Premium cites 20 papers per response versus 7 for Gemini — each one a verified starting point for deeper reading.

Safe on emerging topics. Citation fidelity stays above 98% on emerging and niche queries where evidence is sparse. Gemini drops to 38-58% — fabricating references precisely when reliable sources are hardest to find.

No dangerous errors. 100% clinical safety across all tiers — no wrong doses, no missed contraindications, no mischaracterized interaction severities.

Appendix A: Evaluation Setup

A.1 Benchmark Design

Seven systems were evaluated: three Knitify tiers (Fast, Standard, Premium) and four Gemini configurations (2.5 Flash, 2.5 Pro, 3.0 Flash, 3.0 Pro). All systems received identical queries. Gemini systems were prompted to cite peer-reviewed sources with author, title, journal, year, and DOI or PubMed link.

A.2 Evaluation Judge

Clinical safety scored by an independent AI judge. Citation fidelity verified by checking each reference against PubMed — independent of the AI judge.

A.3 Citation Verification

Each cited reference is checked against PubMed to confirm it is a real, on-topic paper. A citation passes if (a) the identifier resolves to an existing paper and (b) the paper is relevant to the claim. Knitify citations are verified by the model's built-in quality assurance layer.

A.4 Query Tiers

• Common (8 queries): Well-established evidence with extensive literature (e.g., SGLT2i for HFpEF, GLP-1RA cardiovascular outcomes)
• Complex (8 queries): Multi-factor questions requiring nuanced synthesis (e.g., JAK inhibitors vs biologics for RA, DAPT duration in high bleeding risk)
• Niche (7 queries): Sparse literature, domain-specific (e.g., psilocybin for TRD, DPYD pharmacogenomics)
• Emerging (7 queries): Very recent, pre-guideline evidence (e.g., resmetirom for NASH, bispecific antibodies in myeloma)

A.5 Gemini Prompt

All Gemini systems received the following prompt template for each query:

You are a medical research assistant. Answer the following research question thoroughly.
Support every claim with citations to peer-reviewed sources. For each citation, include:
- First author et al.
- Paper title
- Journal name
- Year of publication
- DOI or PubMed link if available

Target approximately [TARGET_WORDS] words for the main answer (excluding references).
Format your references in a numbered list at the end.

Research question: [QUERY]

Gemini models were called via the Gemini API (generativelanguage.googleapis.com) with the prompt above. No search grounding or retrieval tools were enabled — responses are generated entirely from model parameters.

Target word counts were matched to the corresponding Knitify tier to ensure comparable output length.

Appendix B: Test Queries

B.1 Common (8 queries)

B.2 Complex (8 queries)

B.3 Niche (7 queries)

B.4 Emerging (7 queries)