Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Technical Specifications 2 Technical Specifications 3 Academic benchmarks 3 Academic benchmarks 4 Safety 4 Safety 5 Weakness 5 Weakness 6 Phi-3-Vision 6.1 Technical Specifications 6.1 Technical Specifications 6.2 Academic benchmarks 6.2 Academic benchmarks 6.3 Safety 6.3 Safety 6.4 Weakness 6.4 Weakness References References A Example prompt for benchmarks A Example prompt for benchmarks B Authors (alphabetical) B Authors (alphabetical) C Acknowledgements C Acknowledgements 3 Academic benchmarks On the next page we report the results for phi-3-mini on standard open-source benchmarks measuring the model’s reasoning ability (both common sense reasoning and logical reasoning). We compare to phi-2 [JBA+ 23], Mistral-7b-v0.1 [JSM+ 23], Mixtral-8x7b [JSR+ 24], Gemma 7B [TMH+ 24], Llama-3-instruct8b [AI], and GPT-3.5. All the reported numbers are produced with the exact same pipeline to ensure that the numbers are comparable. These numbers might differ from other published numbers due to slightly different choices in the evaluation. As is now standard, we use few-shot prompts to evaluate the models, at temperature 0. The prompts and number of shots are part of a Microsoft internal tool to evaluate language models, and in particular we did no optimization to the pipeline for the phi-3 models.[4] The number of k–shot examples is listed per-benchmark. An example of a 2-shot prompt is described in Appendix A. Authors:
(1) Marah Abdin;
(2) Sam Ade Jacobs;
(3) Ammar Ahmad Awan;
(4) Jyoti Aneja;
(5) Ahmed Awadallah;
(6) Hany Awadalla;
(7) Nguyen Bach;
(8) Amit Bahree;
(9) Arash Bakhtiari;
(10) Jianmin Bao;
(11) Harkirat Behl;
(12) Alon Benhaim;
(13) Misha Bilenko;
(14) Johan Bjorck;
(15) Sébastien Bubeck;
(16) Qin Cai;
(17) Martin Cai;
(18) Caio César Teodoro Mendes;
(19) Weizhu Chen;
(20) Vishrav Chaudhary;
(21) Dong Chen;
(22) Dongdong Chen;
(23) Yen-Chun Chen;
(24) Yi-Ling Chen;
(25) Parul Chopra;
(26) Xiyang Dai;
(27) Allie Del Giorno;
(28) Gustavo de Rosa;
(29) Matthew Dixon;
(30) Ronen Eldan;
(31) Victor Fragoso;
(32) Dan Iter;
(33) Mei Gao;
(34) Min Gao;
(35) Jianfeng Gao;
(36) Amit Garg;
(37) Abhishek Goswami;
(38) Suriya Gunasekar;
(39) Emman Haider;
(40) Junheng Hao;
(41) Russell J. Hewett;
(42) Jamie Huynh;
(43) Mojan Javaheripi;
(44) Xin Jin;
(45) Piero Kauffmann;
(46) Nikos Karampatziakis;
(47) Dongwoo Kim;
(48) Mahoud Khademi;
(49) Lev Kurilenko;
(50) James R. Lee;
(51) Yin Tat Lee;
(52) Yuanzhi Li;
(53) Yunsheng Li;
(54) Chen Liang;
(55) Lars Liden;
(56) Ce Liu;
(57) Mengchen Liu;
(58) Weishung Liu;
(59) Eric Lin;
(60) Zeqi Lin;
(61) Chong Luo;
(62) Piyush Madan;
(63) Matt Mazzola;
(64) Arindam Mitra;
(65) Hardik Modi;
(66) Anh Nguyen;
(67) Brandon Norick;
(68) Barun Patra;
(69) Daniel Perez-Becker;
(70) Thomas Portet;
(71) Reid Pryzant;
(72) Heyang Qin;
(73) Marko Radmilac;
(74) Corby Rosset;
(75) Sambudha Roy;
(76) Olatunji Ruwase;
(77) Olli Saarikivi;
(78) Amin Saied;
(79) Adil Salim;
(80) Michael Santacroce;
(81) Shital Shah;
(82) Ning Shang;
(83) Hiteshi Sharma;
(84) Swadheen Shukla;
(85) Xia Song;
(86) Masahiro Tanaka;
(87) Andrea Tupini;
(88) Xin Wang;
(89) Lijuan Wang;
(90) Chunyu Wang;
(91) Yu Wang;
(92) Rachel Ward;
(93) Guanhua Wang;
(94) Philipp Witte;
(95) Haiping Wu;
(96) Michael Wyatt;
(97) Bin Xiao;
(98) Can Xu;
(99) Jiahang Xu;
(100) Weijian Xu;
(101) Sonali Yadav;
(102) Fan Yang;
(103) Jianwei Yang;
(104) Ziyi Yang;
(105) Yifan Yang;
(106) Donghan Yu;
(107) Lu Yuan;
(108) Chengruidong Zhang;
(109) Cyril Zhang;
(110) Jianwen Zhang;
(111) Li Lyna Zhang;
(112) Yi Zhang;
(113) Yue Zhang;
(114) Yunan Zhang;
(115) Xiren Zhou. Authors: Authors: (1) Marah Abdin; (2) Sam Ade Jacobs; (3) Ammar Ahmad Awan; (4) Jyoti Aneja; (5) Ahmed Awadallah; (6) Hany Awadalla; (7) Nguyen Bach; (8) Amit Bahree; (9) Arash Bakhtiari; (10) Jianmin Bao; (11) Harkirat Behl; (12) Alon Benhaim; (13) Misha Bilenko; (14) Johan Bjorck; (15) Sébastien Bubeck; (16) Qin Cai; (17) Martin Cai; (18) Caio César Teodoro Mendes; (19) Weizhu Chen; (20) Vishrav Chaudhary; (21) Dong Chen; (22) Dongdong Chen; (23) Yen-Chun Chen; (24) Yi-Ling Chen; (25) Parul Chopra; (26) Xiyang Dai; (27) Allie Del Giorno; (28) Gustavo de Rosa; (29) Matthew Dixon; (30) Ronen Eldan; (31) Victor Fragoso; (32) Dan Iter; (33) Mei Gao; (34) Min Gao; (35) Jianfeng Gao; (36) Amit Garg; (37) Abhishek Goswami; (38) Suriya Gunasekar; (39) Emman Haider; (40) Junheng Hao; (41) Russell J. Hewett; (42) Jamie Huynh; (43) Mojan Javaheripi; (44) Xin Jin; (45) Piero Kauffmann; (46) Nikos Karampatziakis; (47) Dongwoo Kim; (48) Mahoud Khademi; (49) Lev Kurilenko; (50) James R. Lee; (51) Yin Tat Lee; (52) Yuanzhi Li; (53) Yunsheng Li; (54) Chen Liang; (55) Lars Liden; (56) Ce Liu; (57) Mengchen Liu; (58) Weishung Liu; (59) Eric Lin; (60) Zeqi Lin; (61) Chong Luo; (62) Piyush Madan; (63) Matt Mazzola; (64) Arindam Mitra; (65) Hardik Modi; (66) Anh Nguyen; (67) Brandon Norick; (68) Barun Patra; (69) Daniel Perez-Becker; (70) Thomas Portet; (71) Reid Pryzant; (72) Heyang Qin; (73) Marko Radmilac; (74) Corby Rosset; (75) Sambudha Roy; (76) Olatunji Ruwase; (77) Olli Saarikivi; (78) Amin Saied; (79) Adil Salim; (80) Michael Santacroce; (81) Shital Shah; (82) Ning Shang; (83) Hiteshi Sharma; (84) Swadheen Shukla; (85) Xia Song; (86) Masahiro Tanaka; (87) Andrea Tupini; (88) Xin Wang; (89) Lijuan Wang; (90) Chunyu Wang; (91) Yu Wang; (92) Rachel Ward; (93) Guanhua Wang; (94) Philipp Witte; (95) Haiping Wu; (96) Michael Wyatt; (97) Bin Xiao; (98) Can Xu; (99) Jiahang Xu; (100) Weijian Xu; (101) Sonali Yadav; (102) Fan Yang; (103) Jianwei Yang; (104) Ziyi Yang; (105) Yifan Yang; (106) Donghan Yu; (107) Lu Yuan; (108) Chengruidong Zhang; (109) Cyril Zhang; (110) Jianwen Zhang; (111) Li Lyna Zhang; (112) Yi Zhang; (113) Yue Zhang; (114) Yunan Zhang; (115) Xiren Zhou. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv [4] For example, we found that using ## before the Question can lead to a noticeable improvement to phi-3-mini’s results across many benchmarks, but we did not do such changes in the prompts.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Microsoft

phi-3-mini's Triumph: Redefining Performance on Academic LLM Benchmarks

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

How to Deploy Large Language Models on Android with TensorFlow Lite

phi-3-mini: The 3.8B Powerhouse Reshaping LLM Performance on Your Phone

Unpacking phi-3-mini: Architecture Driving Phone-Deployable LLM Power

Fortifying LLM Safety: phi-3's Responsible AI Alignment

Navigating LLM Frontiers: phi-3's Weaknesses and Augmentation Pathways

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

How to Deploy Large Language Models on Android with TensorFlow Lite

phi-3-mini: The 3.8B Powerhouse Reshaping LLM Performance on Your Phone

Unpacking phi-3-mini: Architecture Driving Phone-Deployable LLM Power

Fortifying LLM Safety: phi-3's Responsible AI Alignment

Navigating LLM Frontiers: phi-3's Weaknesses and Augmentation Pathways

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps