Secure communication is of paramount importance in modern society. Asymmetric cryptography methods such as the widely used RSA method allow secure exchange of information between parties who have not shared secret keys. However, the existing asymmetric cryptographic schemes rely on unproven mathematical assumptions for security. Further, the digital keys used in their implementation are susceptible to copying that might remain unnoticed. Here we introduce a secure communication method that overcomes these two limitations by employing Physical Unclonable Keys (PUKs). Using optical PUKs realized in opaque scattering materials and employing off-the-shelf equipment, we transmit messages in an error-corrected way. Information is transmitted as patterned wavefronts of few-photon wavepackets which can be successfully decrypted only with the receiver's PUK. The security of PUK-Enabled Asymmetric Communication (PEAC) is not based on any stored secret but on the hardness of distinguishing between different few-photon wavefronts