Transformer-Based Handwriting Recognition

Early fusion of online stroke and offline image features

A transformer-based handwriting recognition system that jointly uses online (pen trajectory) and offline (rasterized image) features. Published at ACPR 2025.

Problem

Handwriting carries complementary cues in the rasterized glyph and in the pen’s trajectory, yet most recognition systems exploit only one modality — losing information the other captures.

Input modalities for handwritten text recognition: (A) image-only, (B) stroke-only, (C) late fusion, and (D) early fusion of both — this work.

Approach

  • Patch encoder converts the grayscale crop into fixed-length visual tokens
  • Lightweight transformer embeds the (x, y, pen) stroke sequence
  • Learnable latent queries attend jointly to both token streams, yielding context-enhanced stroke embeddings that are pooled and decoded under a cross-entropy objective
  • Early fusion in a shared latent space, before any high-level classification, so temporal and visual cues reinforce each other — producing stronger writer independence
The proposed pipeline: image and stroke encoders, cross-modal fusion via learnable latent queries, and a classification head.

Results

  • State-of-the-art accuracy on IAMOn-DB and VNOn-DB, exceeding previous bests by up to 1%
  • Pipeline adapted with gesturification to the ISI-Air dataset
Qualitative results on (A) IAM-OnDB and (B) VN-OnDB: baselines in red misrecognize the characters that our model, in green, recognizes correctly.

Publication

Lodh, A., Chakraborty, R., Shivakumara, P., Pal, U. A Transformer Based Handwriting Recognition System Jointly Using Online and Offline Features. ACPR 2025, LNCS, pp. 250–264. DOI · arXiv · Code (HATChar-Classifier)

Work done at the Computer Vision and Pattern Recognition Unit, Indian Statistical Institute, Kolkata, advised by Prof. Umapada Pal.