Bengali Document Layout Analysis - A YOLOV8 Based Ensembling Approach

We finetuned our model on the whole dataset to detect and segment all of the given instance classes. We fine-tuned this model for 50 epochs. Our experiments showed that providing more training epochs than 50 degrades results.

A specialized YOLOv8m-seg model was trained on our dataset with the singular purpose of accurately recognizing and segmenting images. This deliberate focus on image segmentation emerged as a strategic response to address a critical shortcoming encountered when utilizing the general model for this task.

The image masks generated by the general model exhibited occasional deficiencies, most notably manifesting as gaps within the mask. This phenomenon became particularly apparent when images contained additional classes coexisting within the image mask. These gaps compromised the integrity and precision of the segmentation process, impeding our goal of obtaining meticulous image masks. We trained this model for 100 epochs on the single class image dataset.

Models underwent rigorous evaluation on the validation set, utilizing diverse metrics including class-wise DICE[10] scores, instances prediction, correct predictions, predicted area, intersection area, and mAP50 [11]. Metrics were computed across 20 image size and confidence threshold combinations for each of the 10 models, yielding over 200 scores.

These scores revealed optimal image size and confidence thresholds for each class. Notably, image sizes of 640px generally excelled, with different confidence thresholds impacting distinct classes. Lower confidences (e.g., 0.25) enhanced paragraph and textbox performance, while higher confidences (e.g., 0.45) benefited tables. Images performed best at around 0.35 confidence. This knowledge prompted separate inferences for each class with its optimal threshold. Validation scores also illuminated post-processing effects.

Integrating validation and submission scores pinpointed overfitting onset, guiding dataset-wide model replication to address overfitting concerns.

In addition to the quantitative evaluation, a qualitative manual testing procedure was executed on select test set images. This manual scrutiny provided a nuanced understanding of the models’ strengths and weaknesses. Particularly, it unveiled issues within image and table masks that emerged when these entities intersected with other classes. The resulting gaps or inaccuracies in image and table masks negatively impacted their respective scores. This observation prompted the formulation of strategies to address such challenges.

Two distinct prediction strategies were employed on the general model, each tailored to the characteristics of specific document elements. And for the we used the image model on top of the general model to enhance image prediction

Paragraph & Textbox Prediction

For the identification of paragraphs and textboxes, predictions were made with a confidence threshold of 0.25. This threshold ensured a balanced approach, effectively capturing instances of these entities within documents.

Image & Table Prediction

Predictions for image and table classes were conducted with a higher confidence threshold of 0.35. This elevated threshold optimized the model’s ability to precisely discern image and table elements within the document.

Key Parameters

Several critical parameters guided the predictive process:

• •

  retina_masks: Enabled retina masks [12] for enhanced segmentation precision. Enabling this feature makes the model use high resolution segmentation masks.

• •

  conf: Confidence thresholds in the general model were set at 0.25 for paragraph and textbox prediction, and 0.35 for image and table prediction, optimizing sensitivity and specificity.

• •

  imgsz: Uniformly set at 640 for consistent image processing.

• •

  stream: Activated streaming mode for efficient prediction operations.

This multifaceted predictive approach underlined our commitment to capturing nuanced document elements, contributing to an advanced and precise document analysis framework.

To further enhance the prediction of the models we applied the various post processing on the predictions of the two models.

Table mask refinement using convex-hull filling

Predicted table masks frequently exhibited gaps, at intersections with other classes as most tables have other elements in them. To address this, a technique involving the use of convex hull filling [5] was employed. This method effectively mitigated the gaps in table masks, enhancing the overall accuracy and completeness of document element segmentation.

Image Mask Prediction

A second prediction phase segments image masks using a specialized model when the general model detects an image instance in a document. This addresses gaps often found in general model’s image masks, especially at intersections with other classes. Firstly, the documents in which the general model detect images are given to the image model. The image model then gives refined prediction of image instances on documents. Image instance detection by the general model is used to avoid redundant inference through the specialized model, optimizing efficiency. The confidence threshold of this model was 0.35.

During prediction, certain images led to CUDA out-of-memory (OOM) exceptions, disrupting normal processing and giving empty predictions [13]. To address this, a systematic approach was adopted. When a CUDA OOM exception occurred, the ”retina_masks” parameter was dynamically disabled during prediction. This strategic decision prevented further disruptions by alleviating the additional memory burden imposed by retina masks. Images triggering CUDA OOM errors were processed with ”retina_masks” set to ”False.” This resulted in getting high resolution masks in as many images as possible.