We investigate the potential of GPT-4~\cite{gpt4} to perform Neural
Architecture Search (NAS) -- the task of designing effective neural
architectures. Our proposed approach, \textbf{G}PT-4 \textbf{E}nhanced
\textbf{N}eural arch\textbf{I}tect\textbf{U}re \textbf{S}earch (GENIUS),
leverages the generative capabilities of GPT-4 as a black-box optimiser to
quickly navigate the architecture search space, pinpoint promising candidates,
and iteratively refine these candidates to improve performance. We assess
GENIUS across several benchmarks, comparing it with existing state-of-the-art
NAS techniques to illustrate its effectiveness. Rather than targeting
state-of-the-art performance, our objective is to highlight GPT-4's potential
to assist research on a challenging technical problem through a simple
prompting scheme that requires relatively limited domain
expertise\footnote{Code available at
\href{https://github.com/mingkai-zheng/GENIUS}{https://github.com/mingkai-zheng/GENIUS}.}.
More broadly, we believe our preliminary results point to future research that
harnesses general purpose language models for diverse optimisation tasks. We
also highlight important limitations to our study, and note implications for AI
safety