Abstract: To explore the application value of the deep learning-based Mask R-CNN instance segmentation model in the diagnosis of lung cancer using pathological images, we retrospectively selected 2262 lung cancer pathological images and obtained a diagnostic model through training. We evaluated the performance of the model using Average Precision (AP) and Average Recall (AR). The results revealed that when the pre network of the model was ResNet50 and the Intersection over Union (IoU) was 0.5 and 0.75, respectively, the AP was 0.8152 and 0.5109, and the AR was 0.6051 and 0.5036, respectively. When the pre network was ResNet101, the AP was 0.8804 and 0.6304, and the AR was 0.6268 and 0.5435, respectively, indicating good performance of the model. Meanwhile, we verified that the Mask R-CNN-based instance segmentation model exhibited high accuracy and performance in the diagnosis of lung cancer using pathological images, which could be a useful tool for clinical diagnosis.