We present a possible extension of the random-matrix theory, which is widely
used to describe spectral fluctuations of chaotic systems. By considering the
Kaniadakis non-Gaussian statistics, characterized by the index {\kappa}
(Boltzmann-Gibbs entropy is recovered in the limit {\kappa}\rightarrow0), we
propose the non-Gaussian deformations ({\kappa} \neq 0) of the conventional
orthogonal and unitary ensembles of random matrices. The joint eigenvalue
distributions for the {\kappa}-deformed ensembles are derived by applying the
principle maximum entropy to Kaniadakis entropy. The resulting distribution
functions are base invarient as they depend on the matrix elements in a trace
form. Using these expressions, we introduce a new generalized form of the
Wigner surmise valid for nearly-chaotic mixed systems, where a
basis-independent description is still expected to hold. We motivate the
necessity of such generalization by the need to describe the transition of the
spacing distribution from chaos to order, at least in the initial stage. We
show several examples about the use of the generalized Wigner surmise to the
analysis of the results of a number of previous experiments and numerical
experiments. Our results suggest the entropic index {\kappa} as a measure for
deviation from the state of chaos. We also introduce a {\kappa}-deformed
Porter-Thomas distribution of transition intensities, which fits the
experimental data for mixed systems better than the commonly-used
gamma-distribution.Comment: 18 pages, 8 figure