4,229 research outputs found
f-Divergence constrained policy improvement
To ensure stability of learning, state-of-the-art generalized policy
iteration algorithms augment the policy improvement step with a trust region
constraint bounding the information loss. The size of the trust region is
commonly determined by the Kullback-Leibler (KL) divergence, which not only
captures the notion of distance well but also yields closed-form solutions. In
this paper, we consider a more general class of f-divergences and derive the
corresponding policy update rules. The generic solution is expressed through
the derivative of the convex conjugate function to f and includes the KL
solution as a special case. Within the class of f-divergences, we further focus
on a one-parameter family of -divergences to study effects of the
choice of divergence on policy improvement. Previously known as well as new
policy updates emerge for different values of . We show that every type
of policy update comes with a compatible policy evaluation resulting from the
chosen f-divergence. Interestingly, the mean-squared Bellman error minimization
is closely related to policy evaluation with the Pearson -divergence
penalty, while the KL divergence results in the soft-max policy update and a
log-sum-exp critic. We carry out asymptotic analysis of the solutions for
different values of and demonstrate the effects of using different
divergence functions on a multi-armed bandit problem and on common standard
reinforcement learning problems
On irreducible n-ary quasigroups with reducible retracts
An n-ary operation q:A^n->A is called an n-ary quasigroup of order |A| if in
x_0=q(x_1,...,x_n) knowledge of any n elements of x_0,...,x_n uniquely
specifies the remaining one. An n-ary quasigroup q is permutably reducible if
q(x_1,...,x_n)=p(r(x_{s(1)},...,x_{s(k)}),x_{s(k+1)},...,x_{s(n)}) where p and
r are (n-k+1)-ary and k-ary quasigroups, s is a permutation, and 1<k<n. For
even n we construct a permutably irreducible n-ary quasigroup of order 4r such
that all its retracts obtained by fixing one variable are permutably reducible.
We use a partial Boolean function that satisfies similar properties. For odd n
the existence of a permutably irreducible n-ary quasigroup such that all its
(n-1)-ary retracts are permutably reducible is an open question; however, there
are nonexistence results for 5-ary and 7-ary quasigroups of order 4.
Keywords:n-ary quasigroups, n-quasigroups, reducibility, Seidel switching,
two-graphsComment: 8 p., 1 fig., ACCT-10. v2: revised, the figure improve
Mesoscopic and microscopic dipole clusters: Structure and phase transitions
Two dimensional (2D) classical system of dipole particles confined by a
quadratic potential is studied. For clusters of N < 81 particles ground state
configurations and appropriate eigenfrequencies and eigenvectors for the normal
modes are found. Monte Carlo and molecular dynamic methods are used to study in
detail the order - disorder transition (the "melting" of clusters). In
mesoscopic clusters (N < 37) there is a hierarchy of transitions: at lower
temperatures an intershell orientational disordering of pairs of shells takes
place; at higher temperatures the intershell diffusion sets in and the shell
structure disappears. In "macroscopic" clusters (N > 37) an orientational
"melting" of only the outer shell is possible. The most stable clusters (having
both maximal lowest nonzero eigenfrequencies and maximal temperatures of total
melting) are that of completed crystal shells which are concentric groups of
nodes of 2D hexagonal lattice with a number of nodes placed in the center of
them. The study of different quantities shows that the melting temperature is a
nonmonotonic function of the number of particles in the system. The dynamical
equilibrium between "solidlike" and "orientationally disordered" forms of
clusters is considered.Comment: 12 pages, 16 Postscript figures. Submitted to Phys. Rev.
- …