Infrastructure degradation is becoming a wide-reaching problem in the United States, and there is a need to determine ways to intelligently distribute taxpayer money when addressing the issues. This paper investigates the use of smartphones to classify various road anomalies by using on-board sensors, including accelerometers, gyroscopes, and a cameras. Having a relatively robust sensor array in a ubiquitous device allows for crowdsourcing of data collection, and makes mapping large road networks that are prevalent in the US much more feasible. Specifically, this paper will propose a novel machine learning algorithm that can identify and differentiate between four different classifications of road anomalies, as opposed to the binary approach (using thresholding) that has been employed in similar studies. Additionally, this approach will be able to classify anomalies by severity, as well as provide an estimate of overall road roughness using the International Roughness Index (IRI). This data will allow for more accurate evaluations of overall road conditions than similar methods, and will allow preventive maintenance to be performed, potentially saving time and money