In this paper we propose a system which recognizes and interprets the semantics of handwritten annotations on printed documents. The semantic information will be sent to the Semantic Desktop, the personal Semantic Web on the desktop computer, which supports users in their information management. This allows a seamless integration of interactive paper into the individual knowledge work. The current implementation of the proposed system works with OpenOffice documents printed on Anoto paper. However, our system generally works on any kind of interactive paper documents and the support of other document formats is planned in near future.