Computer Vision · NLP · Healthcare AI

Donut
Med
Decoder.

A fine-tuned Donut transformer that reads handwritten Bangladeshi prescription images and extracts structured medical data — no OCR pipeline, no language preprocessing, end-to-end.

View on GitHub See Results
Training Results — Best Checkpoint ✓ Milestone 3
0.134 Best CER (Character Error Rate)
Donut Base Model — naver-clova-ix/donut-base
Bengali Handwritten prescription images
CV Eng. Role — AIG 200 Capstone Team
Context

A real problem in healthcare.

The Problem

In Bangladesh and across South Asia, millions of prescriptions are handwritten in a mix of Bengali script and English abbreviations — often illegible to patients and pharmacists alike.

  • Handwriting varies drastically between physicians
  • Mixed Bengali and Latin scripts in the same document
  • Medication errors from misread prescriptions are a serious patient safety risk
  • No scalable, automated solution existed for this script and domain

The Approach

Fine-tune the Donut (Document Understanding Transformer) on a dataset of handwritten Bangladeshi prescription images — training the model to output structured JSON directly from raw image pixels.

  • No separate OCR step — the model reads image to text end-to-end
  • Donut's vision encoder handles layout and script variation natively
  • Structured output: drug name, dosage, frequency, patient info
  • CER as the training metric — character-level accuracy on extracted text
Architecture

How Donut works.

Donut — Document Understanding Transformer ───────────────────────────────────────────────────────────── Input └── Prescription image (handwritten, Bengali + English mixed script) Vision Encoder (Swin Transformer) ├── Image split into patches → patch embeddings ├── Hierarchical feature extraction across 4 stages └── Output: rich 2D visual feature map │ no OCR, no text region detection, no language ID ▼ Text Decoder (BART-style autoregressive decoder) ├── Cross-attends to visual features from encoder ├── Generates tokens one at a time, left to right └── Prompted with a task token → <s_rx> │ ▼ Output └── Structured JSON { "patient": "...", "drug": "...", "dosage": "...", "frequency": "...", "notes": "..." } ───────────────────────────────────────────────────────────── Fine-tuned from: naver-clova-ix/donut-base Metric: CER (Character Error Rate — lower is better) Best checkpoint: CER 0.134
👁️
Vision Encoder

Swin Transformer

Hierarchical vision transformer that processes the prescription image as a grid of patches. Captures both fine-grained character strokes and broader document layout without any explicit region proposals.

Patch embeddings 4-stage hierarchy
🧠
Text Decoder

BART-style Autoregressive

Cross-attends to the visual feature map from the encoder and generates structured text token by token. A task prompt token tells the model what schema to produce — no separate classification head needed.

Cross-attention Task prompting
📊
Training Objective

Character Error Rate

CER measures edit distance between predicted and ground-truth text at the character level — ideal for a handwriting task where partial credit matters and individual character accuracy is clinically significant.

Lower = better Edit distance
Training

From baseline to best checkpoint.

Phase 1
Dataset preparation
Collected and preprocessed handwritten Bangladeshi prescription images. Annotated with structured ground-truth JSON labels covering drug name, dosage, frequency, and patient metadata. Applied augmentation to handle varied lighting, scan quality, and handwriting styles.
Phase 2
Donut base fine-tuning
Fine-tuned naver-clova-ix/donut-base on the prescription dataset. The model was prompted with a task token to trigger structured extraction mode. Training used cross-entropy loss on the token sequence outputs.
Phase 3 — Best checkpoint
CER 0.134 achieved
Best validation CER of 0.134 — meaning the model correctly predicted ~86.6% of characters in extracted prescription text. Checkpoint saved and evaluated against held-out test images to confirm generalization.
CER 0.13486.6% char accuracy
Phase 4
Milestone 3 — Report & cue cards
Prepared the Computer Vision Engineer section of the Milestone 3 capstone report. Documented the Donut architecture, training pipeline, CER results, and evaluation methodology. Built cue cards summarizing model design decisions for the presentation.
Results

What the model achieved.

0.134 Best CER // character error rate
86.6% Character accuracy // 1 - CER
0 OCR preprocessing steps // end-to-end
JSON Structured output format // directly from image
~ python evaluate.py --checkpoint best_model.pt
Loading checkpoint: epoch_12_cer0.134.pt
Running inference on test set...
 
Sample prediction:
{
"patient": "Mohammad R., 54M",
"drug": "Metformin 500mg",
"dosage": "1 tablet",
"frequency": "twice daily with meals",
"notes": "3 month supply"
}
 
Validation CER: 0.134 ✓ best checkpoint
Test CER: 0.141
~
Stack

Built with.

CategoryTool / LibraryPurpose
Model Donut naver-clova-ix Document Understanding Transformer — vision encoder + text decoder
Framework PyTorch fine-tuning Model training, gradient updates, checkpoint saving
Library Hugging Face Transformers DonutProcessor, VisionEncoderDecoderModel, training utilities
Metric CER character error rate Edit distance / ground truth length — primary eval metric
Data Bangladeshi Rx Images Handwritten prescriptions, Bengali + English mixed script
Role Computer Vision Engineer AIG 200 Training pipeline, architecture decisions, eval, Milestone 3 report