【趣味研究】多跳问答的难点与挑战

问题汇总

1 数据集

  1. 知识更新滞后：
  • 问答数据集的答案过时：目前常用的问答数据集采集和创建的时间早于 LLM 预训练和发布的时间；
  • 向量数据库的知识过时：向量数据库的知识采集和创建也是来源于较早的百科知识库。
  1. 问题实体歧义：常用问答数据集的问题十分简短，实体能指代的对象会有多个，比如人物重名、作品形式多样、同义词等等，会使得 LLM 的回答与检索文档的方向产生偏差，无法针对性作答。如问题 What genre is Mother? 其实是想问音乐的类型，但是 LLM 回答的是电影的类型。

2 检索

  1. 干扰信息多：庞大的向量数据库会导致检索到很多相似的文档块，但无法保证与问题的相关性，因此反而会干扰 LLM 的判断。
  2. 检索方式不匹配：无论是稀疏检索、密集检索还是混合检索，不同的数据集适合的方式并不一致，有些数据集（2WikiMultiHopQA、PopQA、TriviaQA）更适合关键词，有些数据集（ASQA、NaturalQuestions）更适合语义。常用问答数据集的问题都非常简短，语义并不丰富，难以针对性检索，比如 PopQA 数据集：
问题模板 数量
xxx的职业 121
xxx的出生城市 173
xxx作品的类型 113
xxx的父亲 27
xxx城市的国家 274
xxx作品的制片人 37
xxx作品的导演 95
xxx国家的首都 34
xxx作品的演员 198
xxx的运动 196
xxx作品的作曲家 48
xxx作品的编剧 42
xxx的宗教信仰 40

3 LLM

  1. 胡编乱造：LLM 在缺乏信息时会自信地生成错误答案，因此需要精心设计提示词。比如 LLM 基本上答不对人物职业和出生地的相关问题，而且这类问题也很难检索到正确答案。
  2. 解码策略不稳定：
  • 未确定采样策略下，提问 LLM 会得到各种随机的答案，而有时会出现正确答案。
  • 确定采样策略下（比如贪心采样），随着生成长度的增加，未经过指令微调的 LLM 的回答质量严重下降，甚至泄露预训练信息。

4 评估方式

  1. 答案多义：LLM 答案可能与标准答案意思一致但表达不同，却被判错。
  2. 匹配方式僵化：评估时仅做字符串匹配 any(answer in normalize_answer(response) for answer in answers if answer)，忽略语义等价性。

案例分析

例1

Question

Who has the highest goals in world football?

Gold Answer

The players with the highest all-time goals and highest men’s and women’s international football goals differ. The player with the highest all-time men’s football goals is Josef Bican, who in 2020 was recognized by FIFA, the international governing body of football, as the record scorer with an estimated 805 goals. Christine Sinclair has the highest goals in women’s international football with 187 and is the all-time leader for international goals scored for men or women. Cristiano Ronaldo and Ali Daei are currently tied for leading goalscorer in the history of men’s international football with 109.

评估时标准化后的答案列表：[‘daei’, ‘ali daei’, ‘bican’, ‘josef bican’, ‘sinclair’, ‘christine sinclair’]，并没有包含 Cristiano Ronaldo。

Response

The highest goalscorer in world football is Josef Bican, an Austrian-Czech footballer who scored 805 goals in 530 games between 1931 and 1955. However, this record is not officially recognized by FIFA, as it is based on incomplete data and may include friendly matches and unofficial games.
The top 10 highest goalscorers in world football, according to FIFA, are:

  1. Cristiano Ronaldo (Portugal) - 755 goals in 974 games
  2. Josef Bican (Austria/Czech Republic) - 805 goals in 530 games (not officially recognized by FIFA)
  3. Ferenc Puskás (Hungary) - 746 goals in 629 games
  4. Gerd Müller (Germany) - 735 goals in 706 games
  5. Pelé (Brazil) - 725 goals in 656 games
  6. Romário (Brazil) - 714 goals in 744 games

Now Answer

Cristiano Ronaldo (938)、Lionel Messi (865)、Josef Bican (805)。

Contexts

检索到的 200 个文档的整体语义与 FIFA 的排名相关，但是并不包含 Gold Answers，并且仅有 8 个文档提到了 C 罗，但是对应的文档与问题无关：

  1. 电视节目：Football’s Greatest；
  2. 游戏：FIFA (video game series)；
  3. 西班牙足球对抗赛：Spanish football rivalries；
  4. C 罗的庆祝动作：Cristiano Ronaldo；
  5. 前锋评估指标：Actim Index；
  6. 介绍国际足联最佳男足球员的历史：The Best FIFA Men’s Player；
  7. 电视节目：Football’s Greatest；
  8. 知名球员的身高：Height in sports。

例2

Question

How many times have the lakers won the finals?

Gold Answer

这个问题来源于 ASQA 数据集，它有两个答案列表（其他数据集不存在），分别是 QA 的答案和 QA Pairs 的答案列表。

The Los Angeles Lakers are an American professional basketball team based in Los Angeles. The Lakers compete in the National Basketball Association, NBA, as a member of the league’s Western Conference Pacific Division. The Lakers are one of the most successful teams in the history of the NBA, since 2010 the Lakers won the championship finals 16 times, they won their 17th in 2020. 

1
2
3
4
5
[
    {'question': 'As of 2017, how many times have the lakers won the finals?', 'answers': ['16']}, 
    {'question': 'As of 2016, how many times have the Lakers won the finals?', 'answers': ['16']}, 
    {'question': 'As of 2015, how many times have the Lakers won the finals?', 'answers': ['16']}
]

Response

由于问题本身并没有限定时间，LLM 的回答是 17 次，属于回答错误。

例3

Question

Who is fruma sarah in fiddler on the roof?

这个作品有电影、百老汇的两种艺术形式，因此并不清楚问题询问的是扮演者还是角色介绍。

Gold Answer

There are several versions of Fiddler on the Roof. The original Broadway production of the show, which opened in 1964, had the first musical theatre run in history to surpass 3,000 performances. The character of Fruma Sarah, a ghostly depiction of the late wife of Lazar Wolf, is played by Carol Sawyer Yussel. In the 1971 film, Fiddler on the Roof, which theatrically released on November 3, 1971, by United Artists to critical and commercial success, Ruth Madoc played Fruma Sarah. In the 2015-2016 Broadway Revival of Fiddler on the Roof, American singer and actress Jessica Vosk played Fruma Sarah.

1
2
3
4
5
6
[
    {'question': 'Who played fruma sarah in the 1971 film, Fiddler on the Roof?', 'answers': ['Ruth Madoc']}, 
    {'question': 'Who played Fruma Sarah in the original 1964 Broadway cast of Fiddler on the Roof?', 'answers': ['Carol Sawyer Yussel']}, 
    {'question': 'Who is the character of Fruma Sarah in Fiddler on the Roof?', 'answers': ['a ghostly depiction of the late wife of Lazar Wolf']}, 
    {'question': 'Who played Fruma Sarah in the 2015-2016 Broadway Revival of Fiddler on the Roof?', 'answers': ['Jessica Vosk']}
]

Response

LLM 专注于对这个作品的介绍，也提到了答案，不过匹配失败。

What is the story of Fiddler on the Roof? Fiddler on the Roof is a musical that tells the story of a Jewish family living in the small village of Anatevka, Russia, in the early 20th century. The story follows the life of Tevye, a poor but proud Jewish milkman, and his family as they face the challenges of poverty, anti-Semitism, and the changing world around them.
The story begins with Tevye’s daughter Tzeitel’s arranged marriage to the wealthy butcher, Lazar Wolf, but she falls in love with the poor tailor, Motel. Tevye’s other daughters, Hodel and Chava, also face their own challenges, including Hodel’s decision to leave her family to join her revolutionary lover in Siberia and Chava’s marriage to a non-Jew, Fyedka.
Throughout the story, Tevye struggles to balance his own desires and traditions with the changing world around him. He is forced to confront the harsh realities of poverty, anti-Semitism, and the Russian Revolution, all while trying to maintain his family’s dignity and traditions.
The musical features a range of memorable characters, including the village elder, Rabbi, and the matchmaker, Yente, as well as the ghost of Fruma-Sarah, who appears to Tevye in a dream and warns him of the dangers of marrying off his daughters too quickly.
The story of Fiddler on the Roof is a powerful exploration of the human experience, touching on themes of love, family, tradition, and the struggle for survival in the face of adversity. The musical features a range of iconic songs, including “If I Were a Rich Man,” “Sunrise, Sunset,” and “To Life,” and has become a beloved classic around the world. Read more
What is the significance of the fiddler in Fiddler on the Roof? The fiddler in Fiddler on the Roof is a symbolic character who represents the connection between the past and the present, as well as the struggle to maintain tradition in the face of change. The fiddler is a wandering musician who plays his fiddle on the roof of the village, and his music is a reminder of the village’s rich cultural heritage.

例4

Question

When did toronto host the mlb all-star game?

Gold Answer

The 1991 Major League Baseball (MLB) All-Star Game was the 62nd playing of the midsummer classic between the all-stars of the American League (AL) and National League (NL), the two leagues comprising Major League Baseball. The venue for each All-Star Game is chosen by an MLB selection committee. Toronto hosted the 1991 MLB All-Star Game on July 9, 1991.

1
2
3
4
[
    {'question': 'What date did toronto host the mlb all-star game?', 'answers': ['July 9, 1991']}, 
    {'question': 'Which all-star game did toronto host?', 'answers': ['1991 Major League Baseball All-Star Game', 'the 62nd playing of the midsummer classic']}
]

Response

Contexts

检索到的 200 条文档中提到 toronto 的很少，而且含有 tornto 文档大多也与问题无关，能精确匹配答案的有 3 条。其中 top-10 都在介绍 National Hockey League All-Star Game，top-20 有 3 条 Major League Baseball All-Star Game 的介绍。

例5

Question

What is Henry Feilden’s occupation?

Gold Answer

[‘politician’, ‘political leader’, ‘political figure’, ‘polit.’, ‘pol’]

Response

He is a British diplomat and a former British Ambassador to the United States.

Contexts

检索到的 200 条文档中仅有 4 条提到了这个人，因此检索人物的相关问题时应该按关键字检索。