In this paper, we show that quantum memory for qudit states encoded in a single photon pulsed optical field has a conceptually simple modular realization using only passive linear optics and coherent feedback. We exploit the idea that two decaying optical cavities can be coupled in a coherent feedback configuration to create an internal mode of the coupled system which is isolated and decoherence-free for the purpose of qubit storage. The qubit memory can then be switched between writing/read-out mode and storage mode simply by varying the routing of certain freely propagating optical fields in the network. It is then shown that the qubit memories can be interconnected with one another to form a qudit quantum memory. We explain each of the phase of writing, storage, and read-out for this modular quantum memory scheme. The results point a way towards modular architectures for complex compound quantum memories
