This paper presents a way to design a lateral controller for an automated vehicle using information gained through image processing with the control objective being to stay on a desired path. Two possible ways to obtain the information necessary for lateral control by image processing are presented, one based on pixel intensity summation and the other on vanishing point calculations. The paper also describes two algorithms for the actual lateral control, one based on classical control theory and the other on modern H∞ control. The resulting control algorithms were implemented on a scaled autonomous vehicle system