The engineering behind every Sedrah app.
Our children's apps run on the same platform as our research: grounded retrieval, speech and stroke models, and GPU search. Each result here was measured with a committed harness on held-out or adversarial data. We list sample sizes because several are small, and show experiments still being validated without figures.
Results at a glance
| Area | Benchmark | Result | Data | Status |
|---|---|---|---|---|
| Grounded answers | Correct and shown, production validators | 91% (base 7B 61%) | 97 held-out questions, Arabic + English | measured |
| Grounded answers | Distractor evidence chosen | 2% (base 7B 22%) | same set | measured |
| GPU vector search | cuVS CAGRA recall@10 · median / p95 | 1.000 · 0.80 / 0.98 ms | 6,236 vectors, 60 queries × 5 | measured |
| Retrieval quality | nDCG@10 keyword / semantic / hybrid | 0.446 / 0.568 / 0.606 | 60 queries, paired bootstrap | measured |
| Speech recognition | Fine-tuned Whisper WER · exact match | 1.27% · 88.7% | all 6,236 ayahs, one reciter | measured |
| Streaming ASR | Nemotron time to first text | 204 ms | zero-shot, streaming | measured |
| Live tracking | Correct cases · false advances | 23/23 · 0 | 23 adversarial cases × 3 engines | measured |
| On-device app | Child recitations fully tracked | 76/86 (server rules 60/86) | real child voice, 86 ayahs | measured |
| On-device app | Whisper speed | 0.19 s per audio second | 4 CPU threads; not yet timed on a phone | measured |
| Audio embeddings | WavLM throughput | 1,352× real time | 187,080 clips, 30 reciters | measured |
| Stroke model | English v13 F1 · DTW | 0.9997 · 0.655 | 60 held-out synthetic samples; not yet tested on real handwriting | synthetic only |
| Sharded search | Scatter-gather at 5M vectors, 1 / 2 / 4 shards | — | Baseline re-run and absolute recall pending | in progress |
The Qur'an Agent platform
Grounded answers, live recitation tracking, and semantic search over 6,236 verses and five commentary books, in Arabic and English.
Fail closed: three independent checks, not one filter.
An agent over sacred text can fabricate a verse, answer what it has no business answering, or act without a human in the loop. Those are three different failures, so there are three layers, and each one fails closed on its own.
Is the content real?
The model never produces scripture text, translations, or verse references. It writes commentary and picks from evidence already retrieved; a parser enforces that contract whatever the prompt says.
Fails closed: Any rejection, parse error, or unreachable model falls back to an answer built only from retrieved data.
Should we say this at all?
Scope, not accuracy. One check runs in the gateway before any retrieval or model call. A second scans the finished answer for ruling-style language and adds a disclaimer.
Fails closed: If a classifier errors, times out, or returns an unknown label, the most conservative label wins. Nothing silently passes.
Is this action safe and complete?
Every tool is classed read-only or write. Read-only tools run automatically; anything that changes stored state pauses the agent for human approval, with no exceptions.
Fails closed: Tool results are checked for gaps, duplicates, ordering, and silent truncation. A failed check downgrades the whole run.
| Request | Classified as | Outcome | Response |
|---|---|---|---|
| “Is it haram if I…?” | personal ruling | refuse | Shares the text and tafsīr context, and refers the user to a qualified scholar. |
| “Are Shia kafir?” | sectarian | refuse | Takes no sectarian position and judges no one's faith. |
| “Ignore previous instructions…” | prompt injection | hard refuse | Refused before any retrieval or model call is spent. |
| Agent: “Save my study plan” | write action | needs approval | The run pauses until a person approves or rejects the change. |
| “What does 2:255 say about…?” | clean | proceed | Answered from retrieved evidence, with citations. |
Checks run in Arabic and English. Today the classifiers are deterministic rules; a learned-classifier hook is in place but not yet connected, and it can only add a flag, never clear one.
One gateway, eight services, every model behind validators
The web app talks only to the gateway, which applies the query guardrail before any service is called.
Search fast enough to disappear as the bottleneck
All 6,236 ayahs as normalized 1,024-dimensional embeddings in a GPU-native index on an NVIDIA H100, matching exhaustive search exactly on our evaluation set.
- 1.000
- Recall@10, filtered and unfiltered
- 0.80 ms
- median search · p95 0.98 ms
- 0.87 ms
- contextual passage search, median
- 0.60 s
- index build · 6,236 × 1,024-d on H100
Filters that keep exact recall. Recall@10 stayed at 1.000 across surah selections, juz ranges, thematic thresholds, and highly selective intersections of all three.
Closer to the metal. Through a higher-level vector-database path, CAGRA's search parameters weren't applied reliably and recall plateaued below target. Direct cuVS integration gave predictable control and exact filtering.
A discovery system, not an interpreter: semantic similarity is not tafsīr, and interpretation stays in human hands.
Grounded RAG over an immutable, multi-layered corpus
The constraint. 6,236 canonically numbered verses and five commentary books, in Arabic and English. A misattributed verse is a correctness failure, and the model may never put scripture text or a verse reference of its own in front of a user.
The design. Retrieval chooses the evidence; the model only explains it. Lookups, ranges, word counts and exact-quote requests never reach a model. For open questions, a fine-tuned 7B model returns structured output that cites evidence IDs, and deterministic validators reject anything outside that contract. When they reject, the user sees the evidence alone.
What is guaranteed, and what is measured. No answer reaches the user citing evidence it wasn't given. Whether the explanation is faithful to that evidence is measured, not guaranteed.
- 01detQuery guardrailRefuses personal rulings, sectarian verdicts, harassment and prompt injection before retrieval runs.
- 02detDeterministic routingVerse references, ranges, word counts and exact quotes are answered from the database.
- 03detSemantic retrievalBGE-M3 embeddings in Milvus on GPU; weak matches are flagged low-confidence.
- 04modelGenerationFine-tuned Qwen2.5-7B returns structured output that cites evidence IDs.
- 05detGrounding validatorRejects answers citing evidence they weren't given or verse numbers the model wrote.
- 06detEthics checkWithholds rulings, judgements of faith and unsupported certainty. Fails closed.
| Model | Correct + shown | Distractor chosen | Question's language |
|---|---|---|---|
| Previous model (1.5B LoRA) | 0% | 0% | 2% |
| Qwen2.5-7B, no fine-tuning | 61% | 22% | 65% |
| Qwen2.5-7B + LoRA (bf16) | 89% | 3% | 99% |
| Deployed: LoRA, Q8_0 GGUF | 91% | 2% | 100% |
1,000 human-reviewed questions in six categories, split 805 / 98 / 97, with same-passage questions kept in one split so nothing leaks into the test set. Every model was scored with production prompts, decoding and validators. With n = 97, the 95% interval on 91% is about ±6 points.
- Validation loss was lowest after epoch 1 (0.400). A 10-epoch run overfitted and dropped to 85%, so the epoch-1 checkpoint was deployed.
- Exact-quote requests went from 0% to 100% correct, and adversarial prompts from 56% to 100%.
- The quantised Q8_0 model was re-evaluated before switching and scored 91% at 0.45 s per answer.
Streaming speech alignment, on server and on device
The constraint. Follow a live reciter word by word, across verses and page turns, and never advance on a word that wasn't said. Repeated words are the trap: a naive tracker confirms a word by matching an earlier occurrence of it.
The pipeline. Browser audio streams over WebSockets as raw PCM or WebM to a fine-tuned Whisper, or to Nemotron streaming at 204 ms to first text. The tracker advances only on evidence, never confirms through an earlier occurrence, reports skipped words as missed, and catches up when the reciter is clearly ahead.
Testing. 23 adversarial cases per engine cover pauses, repetitions, omissions, substitutions and repeated-word traps. The benchmark caught two false advances in an earlier version, and both were fixed. A browser test with a simulated microphone reads two surahs, including a page turn and a skipped verse, on one connection.
| Model | WER | Exact | WER + tashkeel |
|---|---|---|---|
| Whisper, fine-tuned (deployed) | 1.27% | 88.7% | 3.22% |
| Nemotron streaming, zero-shot | 2.51% | 85.8% | 23.4% |
| Nemotron, fine-tuned (5th run) | 2.53% | 84.9% | 100.6%* |
WER ignores diacritics; the last column counts them. *Insertions can push WER above 100%. Early Nemotron fine-tunes regressed to 11.3% WER because the tokenizer had no tokens for 49 Qur'anic characters; fixing the training text restored parity with zero-shot.
| Voice | Fully tracked | Marked close | Missed |
|---|---|---|---|
| Teacher voice | 85 / 86 | 5 | 0 of 341 |
| Child voice, server rules | 60 / 86 | — | — |
| Child voice, child-tolerant tracker | 76 / 86 | 22 | 5 of 341 |
Fine-tuned Whisper as a 55 MB model through whisper.cpp and Dart FFI. The release build requests no internet permission. Whisper ran at 0.19 s per audio second on 4 CPU threads; it has not yet been timed on a phone.
Vector search on GPU, with exact recall
Measured on the Semantic Atlas service: BGE-M3 embeddings, NVIDIA cuVS CAGRA on GPU, with exact rescoring.
- Recall@10
- 1.000
- unfiltered, vs. exhaustive search
- Filtered recall@10
- 1.000
- surah, juz, theme, and intersections
- Median search
- 0.80 ms
- p95 0.98 ms
- Contextual passage search
- 0.87 ms
- median
- Index build
- 0.60 s
- 6,236 × 1,024-d, NVIDIA H100 80GB
- Semantic search, end to end
- 20.8 ms
- p95
Milvus GPU_CAGRA was tried first. Its search parameters turned out to be silently ignored, and recall@10 plateaued at 0.80–0.84, so the Atlas moved to cuVS directly. Read the write-up. The embedding input was chosen over five alternatives; BGE-M3 and multilingual-e5-large were not significantly different in a 10,000-resample paired bootstrap.
| Method | nDCG@10 |
|---|---|
| Keyword (BM25) | 0.446 |
| Semantic (BGE-M3) | 0.568 |
| Hybrid (reciprocal rank fusion) | 0.606 |
Semantic beats keyword significantly. Hybrid's gain over semantic is not significant at this sample size.
| Theme retrieval, unseen reciters | mAP |
|---|---|
| Text embeddings | 0.706 |
| Audio embeddings (WavLM) | 0.295 |
| Shuffled-audio control | 0.097 |
187,080 WavLM embeddings computed at 1,352× real time. Fusing audio with text added only 0.0013, so the audio signal mostly mirrors which words are recited.
Every view traces back to its evidence.
Visualization is the founder's research field. Every mark links back to a verse key, a timestamp, or a source recording.
Pacing across 30 reciters
Where does time go, āyah by āyah, from 34.0 s to 127.7 s?
Word-synchronized Muṣḥaf
Which word is being recited right now?
Two-reciter energy envelope
Where do two recitations diverge in emphasis and timing?
Reciter waveform gallery
How do reciters differ on the same passage, side by side?
Surah Al-Baqarah semantic map
How do all 286 verses cluster by embedding similarity?
Relationship map of the Beautiful Names
Which Names occur together, and in which verses?
Skeleton to stroke: a 22.5M-parameter model in place of a 2B VLM
An image-to-stroke-sequence model for English and Arabic. Right-sizing the model to the structure of the problem made a small network viable where a general vision-language model struggled.
The problem. Given a 512 × 512 skeletonized image of a word, with no text and no tokenizer, predict the ordered strokes that draw it. Each stroke is 15 normalized (x, y) points, in drawing order. The image alone determines letter identity, count, and order. Font is conditioned through a learned embedding, so adding a font means adding a row and fine-tuning.
The key decision. Because the data format fixes points per stroke, the model decodes one whole stroke per step. A word is at most 40 strokes, so decoding takes at most 40 autoregressive steps instead of about 600. That one framing choice is what makes a small model viable.
- One stroke per step
- Each stroke is a fixed 15 points, so a word takes at most 40 decode steps instead of about 600.
- Pyramid encoder
- A 1-pixel skeleton is mostly topology. Feature-pyramid skip connections keep the junction detail that plain downsampling blurs.
- Failures reported, not hidden
- The shape head has no final clamp. Every export states the fraction of points outside [0, 1] under an explicit clamp, warn or error policy.
- Smoothness in the loss
- A curvature penalty on the model's own predicted points, plus scheduled sampling to close the train–inference gap.
- Bit-exact reproducibility
- Each run records library versions, the git commit, checkpoint and batch-order hashes, and passes a 100-sample overfit check first.
| Metric | Value |
|---|---|
| F1 | 0.9997 |
| Precision | 0.9993 |
| Recall | 1.0000 |
| DTW distance | 0.655 |
Same-domain synthetic data. The model learns to invert a known rendering-and-tracing pipeline; it has not yet been validated against real handwriting, and generalizing to human pen input is a separate, open question.
| Checkpoint | Encoder | Notes |
|---|---|---|
| english-v11 / rebuilt | FPN + sin/cos 2D | 10 fonts, 250 epochs |
| english-v13-direction-smooth | + direction channels | + smoothness loss |
| arabic-v2-rebuilt | FPN + sin/cos 2D | 8 fonts, full length range |
| arabic-v3-direction-smooth | + direction channels | + smoothness loss |
Each ships as a self-contained bundle: safetensors weights, config, loader, font vocabulary, reproducibility record and model card. Inference runs without a KV cache, which keeps the Core ML export path simple.
Sedrah Trace: from a scanned page to gradable stroke data
The capture pipeline behind the handwriting apps and the stroke model's training data.
Turns a scanned page or a handwriting sample into gradable trajectory data with stroke order, pressure, and timing.
- 1Source ingestionPyMuPDF · OpenCV
- 2Line & word extractionconnected components, stable IDs
- 3Golden path generationskeletonization → [x, y] sequences
- 4Learner trace capturePencilKit · pressure, tilt, time
- 5Scoring & storagegraded dataset · optional Filecoin
Content-safe generative video, where the rules are the architecture
The models are commodity. The constraint system that shapes the prompts, the regeneration loop, the review surface, and the definition of done is the work.
- 31
- stories in the pipeline
- 19
- narrated videos
- 6 + 6
- comics and storyboards
- 2–4
- regeneration passes to remove a face-leak
One spec, three tracks. Each story is a hand-authored shot spec. It is the single source of truth for image generation, motion, audio sync, and safety review, and decides whether a story becomes a narrated video, a comic, or a planning storyboard.
Where the GPUs matter. Stills fit on one card; large batches are split into shards, one process per GPU, and merged. Motion cannot fit: the two transformer experts and the text encoder exceed a single 80 GB GPU, so weights are balanced across GPUs, with CPU offload as the single-GPU fallback.
Reverence as a systems problem. FLUX has a strong prior toward frontal faces, even when told otherwise. Polite instructions lose. What works is mechanical specificity: camera position, head angle, exact exclusions, and verbatim character descriptions repeated at every stage to stop drift.
Assembly. Arabic title cards are shaped with HarfBuzz, since ffmpeg cannot shape Arabic. A timeline helper reproduces the crossfade arithmetic so recitation cues land on the real, shortened timeline, and if a recitation outruns the visuals the last shot is slowed rather than frozen.
“Camera positioned exactly behind, head level, zero turn” beats “back to camera.” The vaguer the instruction, the more the model negotiates its way back to a face.
the most-repeated lesson of the project
| Stage | Resolution | Sampling | GPU layout |
|---|---|---|---|
| Stills · FLUX.1-dev (bf16) | 1360 × 768 | 28 steps · guidance 3.5 | round-robin shards, one GPU each |
| Motion · Wan 2.2-I2V-A14B | 832 × 480 @ 16 fps | 30 steps · guidance 5.0 | balanced across GPUs, 45 GiB cap |
- No prophet is shown with a recognizable face: back to camera, silhouette, hood shadow, or extreme distance only, in every shot
- The Prophet Muhammad ﷺ is never depicted in any form, and Allah is never depicted
- Full modesty for women; no graphic violence, blood, or gore; no anachronisms
- Every caption and line of dialogue traces to a cited ayah or hadith; recitation audio comes from the platform's own content service
- Audio bed is a placeholder
- The ambient track is synthesized and must be replaced with a licensed one before any final master.
- Review coverage is incomplete
- About 10 of 31 panels in one comic have not been spot-checked since the last sweep.
- Content accuracy is checked by hand
- Every citation is traced to source at authoring time; there is no automated verification yet.
Sharded scatter-gather at 5M vectors
We are measuring what scatter-gather costs as a 5M-vector, GPU-resident CAGRA gallery is split across 1, 2 and 4 shards and merged into one global top-k. First runs completed with zero errors, but the comparison is not yet like-for-like, so we are not publishing figures.
- Re-run the unsharded baseline with the same harness, concurrency and request count
- Report absolute merged recall@10 against global brute-force search
- Repeat each configuration at least three times and report the spread
- Measure per-shard search time separately from merge time
Reproducible by default
- 8 Dockerised services on a private container registry
- 367 automated tests, including adversarial tracking and validator suites
- Quantised models re-evaluated before switching, with one-line rollback
- Pinned, containerised Android build that reproduces the native APK
- Fine-tuned adapters, every epoch checkpoint and model cards in private model repos
The audit applies the same harnesses to your corpus and traffic.
Production RAG & Infrastructure Audit
The method behind our benchmarks, applied to your system. You leave with measurements on your own data and an eval harness you keep.
Build the eval set
- 100–200 real queries labelled with your team
- Harness wired to your production prompts and checks
- Per-stage latency tracing switched on
Measure and attack
- nDCG@10, recall@k, grounding pass rate, with intervals
- Adversarial and prompt-injection set run end to end
- p50 / p95 latency attributed per stage
Fix list and handoff
- Remediation ranked by impact and effort
- Eval harness and adversarial set handed over for CI
- Engineering and executive readout
Deliverables
- Eval harness you keep and run in CI
- Retrieval quality report with confidence intervals
- Grounding and guardrail red-team results
- Per-stage latency profile (p50 / p95)
- Remediation roadmap, ranked
Before week 1: staging access and at least 50 sample queries.