From hyperbole to neutrality: A linguistic analysis of planner- and LLM-generated summaries of public submissions

Source abstract

Large language models (LLMs) are increasingly applied to text summarisation tasks in many professional domains, yet their use in urban planning remains underexplored. This paper investigates how planner-written summaries of public submissions compare with LLM-generated summaries, with a focus on linguistic characteristics, tone and semantic similarity. Drawing on 7932 public submissions to a housing reform discussion paper, we analysed a planner-written summary, and 25 summaries generated using ChatPDF (OpenAI GPT-4o). Five prompt engineering methods were used to generate the summaries. Evaluation methods combined syntactic and semantic analysis and automated evaluation metrics. Semi-structured reflections from planning officers supplemented the analysis. Results demonstrate that LLM summaries are more difficult to read and lexical diversity is significantly more complex than planner-written summaries. LLMs tend to incorporate emotive sentiment from submissions, whereas planners use more neutral language focused on planning impacts. Automated metrics revealed poor n-gram overlap but good semantic similarity. The findings suggest that LLMs can effectively capture meaning from text but struggle with interpretive neutrality and planning contexts.