Historical Texts contain accounts of past events that provide insight into the ideologies, culture, and innovations that have shaped our world. This emphasizes the need to preserve and digitize them for the future. Simply taking photos is infeasible for a large volume of data and does not produce searchable text. Deep learning-based approaches such as Optical Character Recognition (OCR) produce searchable text quickly however, there is room for improvement as they are error-prone. A model solely based on pre-trained Transformers is seen as a viable alternative to the overused combination of Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) models.

The aim of this project is to implement deep learning-based text recognition which is trained (fine-tuned) to scan historical texts. The model makes use of the Transformer architecture (Transformer-based Optical Character Recognition (TrOCR) [1]) instead of the traditional CNNs and RNNs. The performance of the model has been evaluated and compared with other deep-learning models that have also been employed for scanning historical books.

The TROCR model developed by researchers from Microsoft is an end-to-end Transformer-based text recognition model. Transformers are encoder-decoder models that harness self-attention and are alternatives to RNNs for sequence-to-sequence (e.g. text->text) problems. Words are encoded into numbers and then positional information is embedded. Multi-head attention produces a new sequence from the attention weights (contribution of each word in the input to the output) calculated for the Feed Forward network to generate the output feature vector which is used by the decoder to produce the text. The decoder uses masking to ensure only previous text is looked at (auto-regressive) for token generation and selects the word with the highest probability score from the SoftMax function. TrOCR [1] is an end-to-end Transformer-based text recognition model. ViT [2] image processor and RoBERTa [3] tokenizer are used.

The model is imported from its HuggingFace page [6] and is trained using PyTorch on the Washington Database [5] with images from George Washington's papers and those from the ARDIS Dataset [4] that contain dates from Swedish Church records. The data is cleaned and formatted to be used as a custom PyTorch Dataset and then a DataLoader; these are used to store and load the data efficiently, respectively. Pixel values of the image are squeezed to a suitable dimension for the processor. The tokenizer pads the ground truth so that all sequences are of the same length for being used in the decoder. This data is split into a train (1981 samples), validation, and test sets (661 samples each) for the purposes as suggested by their names. The model is trained with ADAM optimizer with weight decay to penalize large weights and avoid overfitting (memorizing training data but underperforming with unseen data) and its performance is evaluated in each iteration/epoch.

Evaluation metrics: Precision- the ratio of the number of accurately predicted words out of all the detected words
Recall- the ratio of correct matches to the number of ground truths
F1 score- the weighted average of precision and recall
CER (Character Error Rate)- the percentage of incorrectly predicted characters

The results of testing the finetuned model are given below. It can be inferred that the model performs with ~88% of precision, recall, and F1 score, indicating that the model accurately predicted 88% of detected and ground truth words and that these metrics are well-balanced. Thus, the model performs well on the combination of the two datasets used.

Hurdles such as an incompatible dataset and insufficient VRAM on Goole Colab were managed to produce a model that is only compatible with words and one-line text, with a competitive CER of 0.06. The Hist-TrOCR was made to fulfill the objectives of this project and meet its requirements. To make the model work with longer text lines, a text detection module that detects the text regions from the image and segments them could be integrated with the fine-tuned model.