IC-383GPT-4-1106, GPT-3.5-1106, and unfine-tuned CodeLlama-13b achieve 44.3%, 39.5%, and 38.5% API call accuracy respectively on unseen APIs (Level 3) with 3-shot retrieved prompting in the API Pack evaluation
Zhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen, Rameswar Panda
The paper evaluates three released, non-fine-tuned models on the API Pack Level 3 benchmark, which tests generation of API calls for entirely unseen APIs. Using 3-shot retrieved prompting (three semantically relevant API examples provided in context), GPT-4-1106 reaches 44.3% API call accuracy, GPT-3.5-1106 reaches 39.5%, and the unfine-tuned CodeLlama-13b baseline reaches 38.5%. In the 0-shot setting (no examples), all three models drop to near-zero accuracy: GPT-4-1106 0.1%, GPT-3.5-1106 0.7%, CodeLlama-13b 0.0%. The evaluation uses a SequenceMatcher similarity ratio threshold of 0.9 to judge correctness.
GPT-3.5 and GPT-4 were evaluated only on Level 3 due to proprietary fine-tuning cost constraints; Levels 1 and 2 are not reported for them. The evaluation uses a similarity-ratio threshold (0.9) rather than execution-based metrics, which the authors acknowledge as a limitation.