We bring in here a novel algebraic approach for attacking the McEliece
cryptosystem. It consists in introducing a subspace of matrices representing
quadratic forms. Those are associated with quadratic relationships for the
component-wise product in the dual of the code used in the cryptosystem.
Depending on the characteristic of the code field, this space of matrices
consists only of symmetric matrices or skew-symmetric matrices. This matrix
space is shown to contain unusually low-rank matrices (rank 2 or 3
depending on the characteristic) which reveal the secret polynomial structure
of the code. Finding such matrices can then be used to recover the secret key
of the scheme. We devise a dedicated approach in characteristic 2 consisting
in using a Gr\"obner basis modeling that a skew-symmetric matrix is of rank
2. This allows to analyze the complexity of solving the corresponding
algebraic system with Gr\"obner bases techniques. This computation behaves
differently when applied to the skew-symmetric matrix space associated with a
random code rather than with a Goppa or an alternant code. This gives a
distinguisher of the latter code family. We give a bound on its complexity
which turns out to interpolate nicely between polynomial and exponential
depending on the code parameters. A distinguisher for alternant/Goppa codes was
already known [FGO+11]. It is of polynomial complexity but works only in a
narrow parameter regime. This new distinguisher is also polynomial for the
parameter regime necessary for [FGO+11] but contrarily to the previous one is
able to operate for virtually all code parameters relevant to cryptography.
Moreover, we use this matrix space to find a polynomial time attack of the
McEliece cryptosystem provided that the Goppa code is distinguishable by the
method of [FGO+11] and its degree is less than qβ1, where q is the alphabet
size of the code.Comment: 61 page