Light and shadows of smart contract development with LLMs

Abstract

Smart contract development remains almost inaccessible to non-experts developers despite the growing adoption of blockchain technology across industries. This paper evaluates the potential of Large Language Models (LLMs) for automated smart contract generation from legal agreements. The work systematically assesses the capabilities of four leading commercial LLMs – gpt-4-turbo (OpenAI), claude-3.5-sonnet (Anthropic), mistral-large (MistralAI), and gemini-1.5-pro (Google) – across a diverse range of legal agreements with varying complexity. The evaluation framework consists of a in-depth evaluation of structured code patterns – typical to smart contracts – to provide nuanced insights into model performances. The results reveal a performance hierarchy with claude-3.5-sonnet and gpt-4-turbo consistently outperforming mistral-large and gemini-1.5-pro, particularly when handling complex agreements such as mortgage note agreement and property sales agreement. A nonlinear relationship has been observed between contract complexity and model performance, with even top-performing models showing significant degradation when processing intricate legal structures. Although achieving syntactic correctness has become increasingly feasible, ensuring functional completeness and security remains challenging, as evidenced by high-impact vulnerabilities detected across all generated smart contracts. This work contributes to the growing discourse on LLM applications in blockchain technology by providing empirical evidence of current capabilities and limitations, establishing a robust foundation for future research in AI-assisted smart contract development