Modern deep learning approaches usually transform inputs into a
modality-specific form. For example, the most common deep learning approach to
image classification involves decoding image file bytes into an RGB tensor
which is passed into a neural network. Instead, we investigate performing
classification directly on file bytes, without the need for decoding files at
inference time. Using file bytes as model inputs enables the development of
models which can operate on multiple input modalities. Our model,
\emph{ByteFormer}, achieves an ImageNet Top-1 classification accuracy of
77.33% when training and testing directly on TIFF file bytes using a
transformer backbone with configuration similar to DeiT-Ti (72.2% accuracy
when operating on RGB images). Without modifications or hyperparameter tuning,
ByteFormer achieves 95.42% classification accuracy when operating on WAV
files from the Speech Commands v2 dataset (compared to state-of-the-art
accuracy of 98.7%). Additionally, we demonstrate that ByteFormer has
applications in privacy-preserving inference. ByteFormer is capable of
performing inference on particular obfuscated input representations with no
loss of accuracy. We also demonstrate ByteFormer's ability to perform inference
with a hypothetical privacy-preserving camera which avoids forming full images
by consistently masking 90% of pixel channels, while still achieving
71.35% accuracy on ImageNet. Our code will be made available at
https://github.com/apple/ml-cvnets/tree/main/examples/byteformer