As the economy advances and railtransittechnology progresses, railtransit plays an important role in alleviating urban traffic pressure and promoting regional economic development. However, railway safety faces potential hazards such as foreign object intrusion in the trackside environment, seriously threatening the safety of trains and passengers. Therefore, to achieve high-performance recognition of obstacle target images in rail transit, a small sample object detection algorithm YOLOv5-RTO, which uses the EVC (Explicit Visual Center) attention mechanism and lightweight up-sampling operator CARAFE (Content-Aware ReAssembly of FEatures) to improve YOLOv5, has been proposed. Based on the self-collected image data of rail transit obstacles, a dense and time-varying dataset was created within the rail transit system, and data augmentation methods were used to reduce the impact of small sample datasets on the detection performance of the subsequent algorithm. Using this dataset, target detection tests were conducted on rail transit obstacle images to analyse the actual performance of the training model. The experiment results indicate that the enhanced YOLOv5- RTO algorithm integrates the advantages of EVC and CARAFE while preserving the lightweight and accurate characteristics of YOLOv5. The improved algorithm scheme has an average inference time of 1.3 ms per frame in the field of rail transit images, achieving an overall mAP of 96.5%, an increase of 1.8% compared to the original algorithm. It has a certain engineering application value and provides reference for further optimizing the safety system of urban rail transit.