In this paper we develop two types of tools to deal with differentiability
properties of vectors in continuous representations \pi \: G \to \GL(V) of an
infinite dimensional Lie group G on a locally convex space V. The first
class of results concerns the space V∞ of smooth vectors. If G is a
Banach--Lie group, we define a topology on the space V∞ of smooth
vectors for which the action of G on this space is smooth. If V is a Banach
space, then V∞ is a Fr\'echet space. This applies in particular to
C∗-dynamical systems (\cA,G, \alpha), where G is a Banach--Lie group.
For unitary representations we show that a vector v is smooth if the
corresponding positive definite function \la \pi(g)v,v\ra is smooth.
The second class of results concerns criteria for Ck-vectors in terms of
operators of the derived representation for a Banach--Lie group G acting on a
Banach space V. In particular, we provide for each k∈N examples of
continuous unitary representations for which the space of Ck+1-vectors is
trivial and the space of Ck-vectors is dense.Comment: 44 pages, Lemma 5.2 and some typos correcte