In this paper, we recall how the syntax of diagrams is captured by the diagram editor generator DIAGEN, and outline a visual, rule-based, and object-oriented programming language based on graph transformation by which DIAGEN can be extended to model the semantics of diagrams as well. This language is generic w.r.t. the diagram notation to be used in the programmed visual system and may thus be used for implementing arbitrary systems based on diagram-manipulation