Image Capturing for Visually Impaired People using Machine Learning

Over the last decade, machine learning and computer vision have become widely recognized as effective tools for creating intelligent assistive devices for enhancing accessibility for the blind or visually impaired. This paper describes a camera-driven capture-and- interpret process using machine learning to turn visual data into useful audio information for a visually impaired person. The proposed capture-and-interpret process will capture live photographs using a camera, process the photographs using machine learning techniques for object detection and optical character recognition, and convert the results into spoken words for helping users understand their environments. To increase situational awareness, we identify and prioritize contextual visual information (objects, text, obstacles, and scene context) that is relevant to the user. Rather than providing pre- defined/static data, the proposed capture-and-interpret process is designed to adjust dynamically based on any changes in the environment so that the visually impaired user will receive real-time, relevant feedback about what they see. The system architecture consists of the following components: 1) a camera interface for image- capture; 2) a machine-learning processing module written in Python; and 3) a text-to-speech engine for providing audio descriptions of the environment. Experimental evaluations of the system produced significant improvements in object recognition accuracy, response time, and user experience as compared to existing vision-assistance devices. Our results demonstrate that combining machine learning, computer vision, and audio interaction provides a viable and scalable method for improving independent mobility and day-to-day assistance for visually impaired users.