Knitify Scientific Advanced

Abstract

We evaluated Knitify Scientific Advanced against four Gemini configurations and two OpenAI GPT-5.5 configurations on 30 scientific queries spanning drug mechanisms, PK/PD interactions, structure-activity relationships, and emerging therapeutics. Knitify achieves 96-98% citation fidelity compared to 20-66% for plain Gemini and 76-92% for GPT-5.5. On complex PK queries, Gemini 2.5 Flash drops to 8%. Knitify Fast and Standard deliver first tokens in 13 seconds — vs 110-125 seconds for GPT-5.5.

How citation fidelity is measured: Each reference cited by Gemini is checked programmatically. DOIs are resolved via CrossRef and PMIDs via PubMed to retrieve the real paper. An independent AI verifier then compares the resolved paper against what was claimed — checking whether the topic, authors, and study match. If the DOI returns a 404 (paper does not exist) or the resolved paper is on a different topic, the citation is marked as hallucinated. Knitify citations are verified by the model's built-in quality assurance layer.

1. Citation Fidelity

2. Reference Density

3. Reference Quality & Recency

Year Distribution

Per-System Breakdown

Citation volume, recency, and journal breadth across all 7 systems.

Journal Quality

Knitify top journals (verified via PubMed): The New England Journal of Medicine (6), Clinical Pharmacology & Therapeutics (5), Journal of Medicinal Chemistry (4), Pharmacology & Therapeutics (4).

Gemini journal claims: Gemini models collectively claim 131 citations to the New England Journal of Medicine across 30 queries. When we verified each DOI, 58% resolved to real NEJM papers — exclusively landmark trials the model memorized. The remaining 42% were fabricated DOIs with valid NEJM prefix format (10.1056/NEJMoa...) that do not correspond to any existing paper.

GPT-5.5 recency cliff: Plain GPT-5.5 cites 0 papers from 2025-26 across all 30 queries. Even with the web_search tool enabled, only 1% of citations are recent (4 of 360). GPT-5.5's January 2026 training cutoff is structural — web grounding rarely promotes new literature into the response. Knitify Standard cites 43% from 2025-26 on the same queries.

4. Speed of Answer

5. Knitify Tier Comparison

6. Head-to-Head Comparisons

Direct comparisons between matched tiers — same-class models, all metrics.

Knitify Fast vs Gemini 3.0 Flash

Knitify Premium vs Gemini 3.0 Pro

Knitify Fast vs Gemini 3.0 Flash + Google Search

Knitify Premium vs Gemini 3.0 Pro + Google Search

Knitify Standard vs GPT-5.5

Knitify Premium vs GPT-5.5 + Web Search

7. Summary

Complex queries are where it matters most. On PK/drug interaction questions, Gemini 2.5 Flash drops to 8% CF. Knitify stays at 99-100%.

2.4× more references. Knitify Premium cites 18 verified papers per response versus 8 for Gemini.

GPT-5.5 findings (added May 2026)

GPT-5.5 with web_search beats Gemini 3.0 Pro + Search (92% vs 89% CF) and lifts the verified reference count to 15.5 per response — close to Knitify Premium's 18.3. Without the web tool, plain GPT-5.5 falls to 76% CF, still ahead of every plain Gemini variant but well below Knitify.

Recency is the structural ceiling. GPT-5.5's January 2026 training cutoff means plain calls cite zero papers from 2025-26. Even web_search grounding lifts that to only 1%. Knitify Standard cites 43% recent literature on the same queries.

Latency penalty is severe. GPT-5.5 takes 110-125s per query because ~65% of output tokens are invisible reasoning. Knitify Standard finishes the same query in 12.8s.

Failure mode for plain GPT-5.5: fabricated identifiers. The model often gets author, journal, year, and title correct but invents the matching PMID. Example from this evaluation: GPT-5.5 cited PMID 18315556 as Wong et al.'s apixaban Factor Xa paper — but that PMID actually resolves to a von Willebrand factor study (the real Wong PMID is 18315548). The DOI for the same reference was correct. This is a structural risk for any clinical workflow that relies on the PMID for traceability.

Appendix A: Evaluation Setup

Eleven systems evaluated: three Knitify tiers, four plain Gemini configurations, two Gemini + Google Search configurations, and two OpenAI GPT-5.5 configurations (plain and with the web_search tool). All received identical queries. Non-Knitify systems were prompted to cite sources with DOI/PubMed links. Clinical safety scored by Gemini 3.0 Pro (temp=0, batch). Citation fidelity verified by resolving each DOI via CrossRef and each PMID via PubMed esummary, then matching the resolved paper against the cited claim with an independent LLM verifier (Gemini Flash-Lite, temp=0).

GPT-5.5 calls used OpenAI's /v1/responses endpoint (Chat Completions rejects the web_search tool). GPT-5.5 was added to this report on May 15, 2026, three weeks after its April 24 release.

A.5 Gemini Prompt

All Gemini systems received the following prompt template for each query:

You are a medical research assistant. Answer the following research question thoroughly.
Support every claim with citations to peer-reviewed sources. For each citation, include:
- First author et al.
- Paper title
- Journal name
- Year of publication
- DOI or PubMed link if available

Target approximately [TARGET_WORDS] words for the main answer (excluding references).
Format your references in a numbered list at the end.

Research question: [QUERY]

Gemini models were called via the Gemini API (generativelanguage.googleapis.com) with the prompt above. No search grounding or retrieval tools were enabled — responses are generated entirely from model parameters.

Target word counts were matched to the corresponding Knitify tier to ensure comparable output length.

Appendix B: Test Queries

B.1 Common — Drug Mechanisms (8)

B.2 Complex — PK/Drug Interactions (8)

B.3 Niche — SAR/Prodrug Design (7)

B.4 Emerging — Novel Targets (7)