The paper evaluates GPT-3.5-turbo-instruct on predicting the next human chess move by feeding it a PGN-formatted prompt of the game state and using greedy decoding. Across the full Lichess 2022 evaluation set, the model's top-1 predicted move matches the human's actual move 53.7% of the time. For special moves it reaches 72.4% on castling, 71.4% on en passant, 86.0% on pawn promotion, and 92.8% on threefold repetition. The authors note the comparison is not entirely fair because GPT-3.5 was never trained for chess and has far more parameters than the purpose-built baselines.
it is difficult to make a fair comparison between allie and gpt-3.5 because on the one hand, gpt-3.5 has many more parameters and potentially observed much more chess data during pre-training. on the other hand, gpt-3.5 was never intended to play chess, and the fact that it can play chess is somewhat remarkable.