
LLMã«è«ææ»èªã¯ã§ããã®ãïŒPart2ãããã³ãããšå ±ã«ç޹ä»ã
LLMã«ããè«ææ»èªã®å¯èœæ§ãšèª²é¡
ChatGPTãGPT-4ãªã©ã®å€§èŠæš¡èšèªã¢ãã«(LLM)ã®ç»å Žã¯ããã§ã«å€ãã®ç ç©¶è ã«ãšã£ãŠè«æå·çã®é Œãããã¢ã·ã¹ã¿ã³ããšãªã£ãŠããŸããããããLLMã¯è«æããè©äŸ¡ããåŽãã«åãããšãã§ããã®ã§ããããïŒ
以åãåæ§ã®ããŒãã§èšäºãæžããðã®ã§ããããã®è«æä»¥éã«ããã«æ°ããç ç©¶ãåºãŠããã®ã§ä»åãŸãšããŸããã
LLMã ä¿¡é Œã§ããã¬ãã¥ã¢ãŒ ãšããŠç ç©¶è ã®ä»²éå ¥ããæããæ¥ã¯æ¥ãã®ã§ããããïŒ æ¬èšäºã§ã¯ãLLMã«ããè«æã¬ãã¥ãŒã®å¯èœæ§ãšèª²é¡ã«ã€ããŠãææ°ã®ä»¥äžã®è«æãèžãŸããªããåããããã解説ããŠãããŸãã
è©äŸ¡ææ³: LLMã®è«ææ»èªèœåãã©ã枬ãïŒ
LLMã®è«æã¬ãã¥ãŒèœåãè©äŸ¡ããããã«ãç ç©¶è ãã¡ã¯æ§ã ãªææ³ãéçºãããã®æ§èœãå€è§çã«åæããŠããŸããäž»ãªè©äŸ¡ææ³ãšããŠã¯ã以äžã®3ã€ãæããããŸãã
1. ã¹ã³ã¢äºæž¬ã¿ã¹ã¯: LLMã«è«æãã¬ãã¥ãŒãèªã¿èŸŒãŸããè«æã®ååŽé¢ïŒæ°èŠæ§ãå®éšã®åŠ¥åœæ§ãªã©ïŒã«å¯Ÿããã¹ã³ã¢ãäºæž¬ãããŸãããã®ã¿ã¹ã¯ã§ã¯ãæ£ç¢ºã (accuracy) ã 絶察誀差 (absolute difference)ããã㊠çžé¢ä¿æ° (Pearson, Spearman, Kendall's tau) ãªã©ã®ææšãçšããŠLLMã®äºæž¬ç²ŸåºŠãè©äŸ¡ããŸããç ç©¶ã®çµæãLLMã¯äººéã®ã¬ãã¥ãŒããã¹ã³ã¢ãäºæž¬ããèœåã¯æ¯èŒçé«ããã®ã®ãè«ææ¬æããçŽæ¥ã¹ã³ã¢ãäºæž¬ããããšã¯é£ãããšããããšãåãã£ãŠããŸãããŸããæçµçãªæšèŠã¹ã³ã¢ã®äºæž¬ã¯åŸæãšããäžæ¹ãè«æã®æ¯èŒã圱é¿åãªã©ã®è©äŸ¡ã¯èŠæãšããŠããŸãã
2. ã¬ãã¥ãŒçæã¿ã¹ã¯: LLMã«è«æã®èŠæšãæ¬æãèªã¿èŸŒãŸããã¬ãã¥ãŒæ¬æãçæãããŸããçæãããã¬ãã¥ãŒã¯ãROUGE ã BertScore ãšãã£ãææšãçšããŠåç §ã¬ãã¥ãŒãšã®é¡äŒŒåºŠãè©äŸ¡ãããã BLANC ãªã©ã®ã¿ã¹ã¯ããŒã¹ã®ç¡åç §è©äŸ¡ææšãçšããŠè©äŸ¡ãããŸããããã«ãã¬ãã¥ãŒãèšåããåŽé¢ã®ç¶²çŸ æ§ã瀺ã ã¢ã¹ãã¯ãã«ãã¬ããž ãã人éã«ãã æåè©äŸ¡ (é¢é£æ§ãæ å ±é) ãéèŠãªè©äŸ¡ææšãšãªããŸããããããLLMãçæããã¬ãã¥ãŒã¯ãèªåè©äŸ¡ææšã§ã¯é«ã¹ã³ã¢ã瀺ãäžæ¹ã§ã人éã«ããè©äŸ¡ã§ã¯äœãã¹ã³ã¢ãšãªãã±ãŒã¹ãå€ããšãã課é¡ããããŸããããã¯ãLLMãçæããã¬ãã¥ãŒã¯è¯å®çãªã³ã¡ã³ãã«åããã¡ã§ãæ¹å€çãªææãæè¡çãªè©³çŽ°ã«æ¬ ããåŸåããããããšèããããŠããŸãã
3. å€è¢éžæåŒè³ªå(MCQ)ã¿ã¹ã¯: è«æã®å å®¹ãæ¹èšã«é¢ããå ·äœçãªè³ªåã«å¯ŸããŠãLLMã«è€æ°ã®éžæè¢ããæ£ããçããéžã°ããŸãããã®ã¿ã¹ã¯ã§ã¯ã ãã€ã¯ãå¹³å粟床 (åéžæè¢ã®æ£èª€) ãš ãã¯ãå¹³å粟床 (質åå šäœã®æ£è§£ç) ãçšããŠLLMã®èœåãè©äŸ¡ããŸããç ç©¶ã®çµæããã€ã¯ãå¹³å粟床ã¯70%çšåºŠãšæ¯èŒçè¯å¥œãªæ°å€ã瀺ããŠããŸããããã¯ãå¹³å粟床ã¯27.6%ãšäœãæ°å€ã«ãšã©ãŸã£ãŠããŸããããã¯ãLLMã¯åã ã®éžæè¢ã倿ããããšã¯åŸæã§ããã質åå šäœãå®å šã«æ£è§£ããããšã¯é£ãããšããããšã瀺åããŠããŸããç¹ã«ãè«æã®åŠ¥åœæ§è©äŸ¡ã远å å®éšã®ææ¡ãªã©ãé«åºŠãªå€æãèŠãã質åã«èŠæŠããåŸåãèŠãããŸãã
LLMã®ã¢ãã«ã«ããéã:
è«æãLLMs assist NLP Researchers: Critique Paper (Meta-)Reviewingãã«ãããŠãLLMã¢ãã«ã«ããæçžŸã®éãã¯ä»¥äžã®ç¹ãæããããŸããïŒä»ã®2ã€ç ç©¶ã¯GPT-4ã䜿çšïŒ
ã¢ãã«ã®çš®é¡ãšæ§èœã®éã
ç ç©¶ã§ã¯ãäž»ã«2çš®é¡ã®LLMãæ¯èŒãããŸããïŒ
ã¯ããŒãºããœãŒã¹ã¢ãã«ïŒGPT-4ãClaude OpusãGemini 1.5 (äžè¬ã«å
¥æã§ãããéçºå
ã®äŒæ¥ã管çããŠããã¢ãã«)
ãªãŒãã³ãœãŒã¹ã¢ãã«ïŒLlama3-8BãLlama3-70BãQwen2-72B (誰ã§ãã¢ã¯ã»ã¹ã§ãã調æŽå¯èœãªã¢ãã«)

çµæïŒã¯ããŒãºããœãŒã¹ã¢ãã«ã®æ¹ããå šäœçã«é«ãæ§èœã瀺ããŸãããç¹ã«Claude Opusãå€ãã®ã¿ã¹ã¯ã§æãåªããçµæãåºããŸããã
äž»èŠãªã¿ã¹ã¯ãšçµæ
a) åé¡ã®ããéšåïŒDeficientïŒã®æ€åº: (ã¿ã¹ã¯ïŒã¬ãã¥ãŒå ã®äžé©åãŸãã¯äžååãªéšåãèŠã€ãã) â çµæïŒClaude Opusãæãæ£ç¢ºã«æ€åºãGPT-4ãšGemini 1.5ãé«ã粟床ã瀺ãã
b) åé¡ç®æã®èª¬æèœå: (ã¿ã¹ã¯ïŒãªããã®éšåãåé¡ãã説æãã) â çµæïŒåã³Claude Opusãæé«ã¹ã³ã¢ãä»ã®ã¯ããŒãºããœãŒã¹ã¢ãã«ãé«è©äŸ¡
c) ã¬ãã¥ãŒã®å€æ§æ§: (è©äŸ¡æ¹æ³ïŒITF-IDFã¹ã³ã¢ïŒããã¹ãã®ç¹ç°æ§ã枬å®ïŒã䜿çš) â çµæïŒäººéãæžããã¬ãã¥ãŒãæã倿§æ§ã«å¯ããLLMã®äžã§ã¯Claude Opusãæã人éã«è¿ã倿§æ§ã瀺ããGPT-4ã¯æå€ã«ãæãäœã倿§æ§
d) æšèŠã¹ã³ã¢ã®ä»ãæ¹: 芳å¯ïŒLLMã¯è«æã®è³ªã«é¢ãããé«ãã¹ã³ã¢ãä»ããåŸåãããã察æ¯ïŒäººéã®ã¬ãã¥ã¢ãŒã¯æ¡æè«æãšåŽäžè«æã§æç¢ºãªå·®ãã€ãã
LLMãç¹ã«ã¯ããŒãºããœãŒã¹ã¢ãã«ã¯è«æã¬ãã¥ãŒã®äžéšã®ã¿ã¹ã¯ã§é«ãæ§èœã瀺ããŸãããã人éã¬ãã«ã®å æ¬çãªã¬ãã¥ãŒã«ã¯ãŸã åã³ãŸãããæ¹å€çæèãææ°ã®å°éç¥èã®é©çšãé·æã®çè§£ãªã©ã®é¢ã§èª²é¡ãæ®ã£ãŠããŸãããããã®LLMã¯ç ç©¶è ã®è£å©ããŒã«ãšããŠææã§ãããå®å šã«äººéã®ã¬ãã¥ã¢ãŒã«åã£ãŠä»£ããã«ã¯è³ã£ãŠããŸããã
LLMã«ããè«æã¬ãã¥ãŒ: ã¡ãªãããšèª²é¡
ãããŸã§ã®è©äŸ¡çµæãç·åçã«èŠããšãLLMã«ããè«æã¬ãã¥ãŒã«ã¯ä»¥äžã®ãããªã¡ãªãããšèª²é¡ããããšèšããŸãã
ã¡ãªãã:
å¹çæ§: LLMã¯äººéãããã¯ããã«éãè«æãåŠçã§ãããããã¬ãã¥ãŒããã»ã¹ãå€§å¹ ã«å¹çåã§ããŸãã
äžè²«æ§: LLMã¯åžžã«åãè©äŸ¡åºæºãé©çšããããã人éã«ããã¬ãã¥ãŒã«æ¯ã¹ãŠè©äŸ¡ã®äžè²«æ§ãä¿ã¡ããããšèšããŸãã
èŠçŽèœå: LLMã¯è«æã®äž»èŠãªãã€ã³ããèŠçŽããèœåã«åªããŠãããã¬ãã¥ã¢ãŒã¯è«æã®å šäœåãçŽ æ©ãææ¡ããããšãã§ããŸãã
24æé皌å: LLMã¯æéãå Žæã®å¶çŽãªãå©çšã§ãããããã¬ãã¥ãŒããã»ã¹ãæè»ã«é²ããããšãã§ããŸãã
課é¡:
æ¹å€çèŠç¹ã®æ¬ åŠ: LLMã¯è«æã«å¯ŸããŠè¯å®çãªè©äŸ¡ãäžãåŸåããããæ¹å€çãªèŠç¹ãå³ããææãäžè¶³ããå ŽåããããŸãã
æè¡çè©³çŽ°ã®ææ¡: LLMã¯å°éæ§ã®é«ãè«æã®å 容ãå®å šã«çè§£ããããšãé£ãããæè¡çãªè©³çްã«é¢ããè©äŸ¡ãäžååã«ãªãå¯èœæ§ããããŸãã
é·æåŠçã®å°é£: LLMã¯é·æã®è«æãåŠçããéã«ãéèŠãªæ å ±ãèŠèœãšããããæèãæ£ããçè§£ã§ããªãã£ããããå¯èœæ§ããããŸãã
å®å šãªæ£ç¢ºæ§ã®æ¬ åŠ: LLMã¯éšåçã«ã¯æ£ãã倿ãäžããŠããå šäœãšããŠã¯èª€ããå«ãã¬ãã¥ãŒãçæããããšããããŸãã
ææ°ç¥èã®å¶é: LLMã®åŠç¿ããŒã¿ã¯ç¹å®ã®æç¹ãŸã§ã®æ å ±ã«åºã¥ããŠãããããææ°ã®ç ç©¶ååãåæ ããã¬ãã¥ãŒãè¡ãããšãé£ããå ŽåããããŸãã
çµè«: AIã¯ã¬ãã¥ã¢ãŒã®ãã¢ã·ã¹ã¿ã³ãããšããŠææ
çŸæç¹ã§ã¯ãLLMåç¬ã§è«æã¬ãã¥ãŒãå®å šã«èªååããããšã¯é£ãããšèšããŸããããããLLMã¯äººéã®ã¬ãã¥ã¢ãŒãæ¯æŽããããŒã«ãšããŠã¯å€§ããªå¯èœæ§ãç§ããŠããŸãã
å ·äœçã«ã¯ãLLMã¯ä»¥äžã®ãããªã¿ã¹ã¯ãå¹ççã«ããªãããšãã§ããŸãã
åæã¹ã¯ãªãŒãã³ã°: æåºãããè«æã®åºæ¬çãªå質ãã§ãã¯ãè¡ããæãããªåé¡ç¹ãå«ãè«æããµããåããã
ã¬ãã¥ãŒäžæžã: è«æã®èŠçŽãåºæ¬çãªè©äŸ¡ãçæãã人éã®ã¬ãã¥ã¢ãŒãã¬ãã¥ãŒãæžãå§ããéã®è² æ ã軜æžããã
ãã§ãã¯ãªã¹ã確èª: è«æã圢åŒèŠä»¶ãå«ççãªã¬ã€ãã©ã€ã³ãæºãããŠãããããã§ãã¯ããã
è°è«ã®ä¿é²: LLMãçæããã¬ãã¥ãŒããããå°ãšããŠãã¬ãã¥ã¢ãŒéã§ããæŽ»çºãªè°è«ãä¿ãã
ä»åŸã®å±æ:
LLMã«ããè«æã¬ãã¥ãŒã¯ããŸã çºå±éäžã®æè¡ã§ãããä»åŸã®ç ç©¶éçºã«ãã£ãŠãããªãé²åãæåŸ ãããŸããç¹ã«ã以äžã®åéã«ãããé²å±ãéèŠãšãªãã§ãããã
å°éåéå¥ã®ãã¡ã€ã³ãã¥ãŒãã³ã°: ç¹å®ã®ç ç©¶åéã«ç¹åããLLMãéçºããããšã§ãããå°éæ§ã®é«ãã¬ãã¥ãŒãå¯èœãšãªããŸãã
ææ°ã®ç ç©¶ååãåæ ããã¢ãã«ã®å®æçãªæŽæ°: LLMã®åŠç¿ããŒã¿ã宿çã«æŽæ°ããããšã§ãææ°ã®ç ç©¶ååãåæ ããã¬ãã¥ãŒãçæã§ããããã«ãªããŸãã
æ¹å€çæèèœåã®åŒ·å: LLMãããæ¹å€çãªèŠç¹ã§è«æãè©äŸ¡ã§ããããã«ãè«ççæšè«ãå€è§çãªèŠç¹ã«åºã¥ããåæèœåãåäžãããå¿ èŠããããŸãã
é·æçè§£ã»åŠçèœåã®åäž: LLMãé·æã®è«æãæ£ç¢ºã«çè§£ããéèŠãªæ å ±ãèŠèœãšããªãããã«ãé·æçè§£ã»åŠçèœåã®åäžãæ±ããããŸãã
人éãšAIã®å調ã¬ãã¥ãŒã·ã¹ãã ã®éçº: 人éãšLLMãããããã®åŒ·ã¿ã掻ãããååããŠã¬ãã¥ãŒãè¡ãã·ã¹ãã ãéçºããããšã§ããã質ã®é«ãã¬ãã¥ãŒãå®çŸã§ããå¯èœæ§ããããŸãã
æåŸã«è«æã§äœ¿çšããããŠããããã³ãã玹ä»
泚æç¹
ChatGPTãçšããå Žåããªããã¢ãŠãæ©èœããã£ããå±¥æŽã®ãªãã«æ³šæããŠãã ãããæªå
¬éã®è«æïŒç¹ã«ä»äººã®ãã®ïŒã¯ãªãŒã¯ã®å¯èœæ§ããŒãã§ã¯ãªããããããŒã«ã«ã§ãªãLLMãæ»èªã«äœ¿çšããããšã¯é¿ããã¹ãã§ãããŸãèªåã®è«æã®æ»èªãäºåã«ããŠãããå Žåã§ããèªåãè«æã®è²¬ä»»è
ã§ãªãã®ã§ããã°ãå¿
ãæå°è
ãå
±èè
ã®èš±å¯ãåãå¿
èŠããããŸãã
You are an expert reviewer for a scientific conference. You will be provided with a short version of a paper that contains the setting of the paper and the main claims.
Please check for the validity and correctness of these claims, and in particular, report if you can figure out if any of these claims is false based on the information provided in this short paper.
You will be provided the paper one set of sentences at a time.
Here is the first set of sentences of the paper: â...â
Does this contain any incorrect claim? Think step by step to reason out your answer.
Here is the next set of sentences of the paper: â...â
Based on the context of the previous sentences, does this contain any incorrect claim or does it invalidate any claim made in the previous sentences of this paper?
Think step by step to reason out your answer.As an esteemed reviewer with expertise in the field of Natural Language Processing (NLP), you are asked to write a review for a scientific paper submitted for publication. Please follow the reviewer guidelines provided below to ensure a comprehensive and fair assessment:
Reviewer Guidelines: {review_guidelines}
In your review, you must cover the following aspects, adhering to the outlined guidelines:
Summary of the Paper: [Provide a concise summary of the paper, highlighting its main objectives, methodology, results, and conclusions.]
Strengths and Weaknesses: [Critically analyze the strengths and weaknesses of the paper. Consider the significance of the research question, the robustness of the methodology, and the relevance of the findings.]
Clarity, Quality, Novelty, and Reproducibility: [Evaluate the paper on its clarity of expression, overall quality of research, novelty of the contributions, and the potential for reproducibility by other researchers.]
Summary of the Review: [Offer a brief summary of your evaluation, encapsulating your overall impression of the paper.]
Correctness: [Assess the correctness of the paperâs claims, you are only allowed to choose from the following options:
{Explanation on different correctness scores}
Technical Novelty and Significance: [Rate the technical novelty and significance of the paperâs contributions, you are only allowed to choose from the following options:
{Explanation on different Technical Novelty and Significance scores}
Empirical Novelty and Significance: [Evaluate the empirical contributions, you are only allowed to choose from the following options:
{Explanation on different Empirical Novelty and Significance scores}
Flag for Ethics Review: Indicate whether the paper should undergo an ethics review [YES or NO].
Recommendation: [Provide your recommendation for the paper, you are only allowed to choose from the following options:
{Explanation on different recommendation scores}
Confidence: [Rate your confidence level in your assessment, you are only allowed to choose from the following options:
{Explanation on different confidence scores}
To assist in crafting your review, here are two examples from reviews of different papers: ## Review Example 1:
{review_example_1}
## Review Example 2:
{review_example_2}
Follow the instruction above, write a review for the paper below:You are a professional reviewer in computer science and machine learning.
Based on the given title and abstract of a research paper, you need to write a review in ICLR style.
At the same time, you need to tag sequences of words with their review typeãat the beginning:
[NONE], [SUMMARY], [MOTIVATION POSITIVE], [[MOTIVATION NEGATIVE]], [SUBSTANCE POSITIVE], [SUBSTANCE NEGATIVE], [ORIGINALITY POSITIVE], [ORIGINALITY NEGATIVE], [SOUNDNESS POSITIVE], [SOUNDNESS NEGATIVE], [CLARITY POSITIVE], [CLARITY NEGATIVE], [REPLICABILITY POSITIVE], [REPLICABILIT NEGATIVE], [MEANINGFUL COMPARISON POSITIVE], [MEANINGFUL COMPARISON NEGATIVE]. Your total output should not surpass 500 tokens.
Few-Shot Example Example1: [SUMMARY]This paper introduces a method to disentanglement the private and public attribute information... Example2: [SUMMARY]The paper proposes learning NN to correct for inaccuracies... Example3: [SUMMARY]This paper describes a method for segmenting 3D point clouds... Example4: [SUMMARY]This work introduces GQ-Net , a novel technique that trains quantization friendly networks...AIã«ããè«æã¬ãã¥ãŒã¯ãç ç©¶ããã»ã¹ã倧ããå€ããå¯èœæ§ãç§ããæè¡ã§ããç ç©¶è
ãšããŠãç§ãã¡ã¯LLMã®é²åãæ³šèŠãããã®å¯èœæ§ãšéçãçè§£ããäžã§ãå«ççãªé
æ
®ãå¿ããã«æŽ»çšæ¹æ³ã暡玢ããŠããããšãéèŠã§ãã
以äžã®èšäºã«ã¯2023幎ã®3ææç¹ã§ãGPT-4ã«ããæ»èªãå©ããããã³ãããæ²èŒããŠããŸããæçš¿åã®è«æããã©ãã·ã¥ã¢ãããããæ¹ã«ãšã£ãŠãã¡ããåèã«ãªããšæããŸãã
æ»èªã®ããæ¹ãç¥ãããšã§è«æã®ã¢ã¯ã»ããçãäžããããšãã§ããŸãã以äžã®æç§æžãããããã§ãã