ChatGPT, Pesak Types, and Semantic Shift
Here is my article from this past Shabbat in the Jewish Link.
In last week’s article, we saw that Rav Herschel Schachter maintains that, in theory, an AI can pasken, so long as it is accurate, which I labeled the cheftza approach. Meanwhile, Rav Natenel Wiederblank maintains that pesak halacha must issue from a human being, which I labeled the gavra approach. Further, in Horayot 2b, Abaye and Rava discussed capabilities associated with hora’ah, ruling, namely gemir and sevir, which we’ll roughly define as comprehensive knowledge and being adept at halachic reasoning. These might be necessary to ensure an accurate pesak.
Pesak halacha is an ambiguous term and describes at least three separate levels of difficulty. The more complex the situation, the greater the level of gemir and sevir that may be required. (A) The question may be straightforward, like whether there is a problem of speaking between tefillin shel yad and shel rosh; or, slightly more complicated, if he spoke in between, what should one do regarding the blessing(s) on the shel rosh? You can open up Shulchan Aruch, Orach Chaim 25:9-10 and read what Rav Yosef Karo and Rav Moshe Isserles respectively wrote. The level of knowledge required might be thorough knowledge of this halachic work; alternatively, basic knowledge plus the knowledge of how to navigate Shulchan Aruch to find where it is discussed. Instead of the questioner looking in the book himself, or using a computer program to pull it from the database, he’s asking the rabbi, who will look in the book. Gemir is required but not much sevir.
(B) The question may be more complex. The case posed by the questioner does not precisely match the cases in the books. There is a skill of מְדַמֵּה מִילְּתָא לְמִילְּתָא, being able to compare one case to another, to pattern match and see which case is the analogue. This could require a deeper understanding of the underlying principles that lead to specific halachic conclusions. Further, the situation might feature several concepts which interact with and impact one another. You need to identify those concepts and understand how they would lead to a conclusion.
(C) The case may be exceptionally novel, or the posek might be of sufficient caliber that the decision doesn’t involve simply reading, combining, and applying established halachic positions in halachic works of Acharonim. Instead, the posek might go back to the underlying gemaras and champion a particular old or novel interpretation, boldly stake out his own halachic principles, and creatively add to the body of halachic work. Each of these requires different levels of gemir and sevir.
To use an analogy of mathematics, this is (a) looking up 3 x 4 in the multiplication table; (b) solving a math word problem on a high school exam; (c) working as a theoretical mathematician to develop new theories and frameworks. When asking if ChatGPT can pasken, which version of pesak do we mean?
Analogy to Yo’atzot Halacha
While not a precise match, we might be מְדַמֵּה מִילְּתָא לְמִילְּתָא and compare the question of AI paskening to the modern phenomenon of yo’atzot halacha – that is, knowledgeable Jewish women who underwent extensive training in relevant metziut, together with specific sugyot in masechet Niddah, together with Rishonim, tracing it down to Tur and its commentaries, Shulchan Aruch and its commentaries, and specific teshuvot of Acharonim. A cheftza approach should allow them to pasken in hilchot niddah, since their halachic decisions are accurate, while a gavra approach could potentially disallow it, similar to disallowing an AI’s pesak. But, which type of pesak do we mean?
In a Headlines Podcast (shiur 402), Rebbetzin Chana Henkin, the Dean of Nishmat, the organization in Eretz Yisrael that trains yo’atzot, explained that yo’atzot act as an “address” for halachic questions, since women feel more comfortable asking such questions to another woman. For actual pesak, the yo’etzet consults with a posek! When pressed by Rabbi Dovid Lichtenstein as to whether they consult with a posek for every question, Rebbetzin Henkin explained (edited for brevity):
"Most of the questions don't require pesak. You know what pesak is? Pesak is not that you know something that someone else doesn't know. Pesak is extrapolating from one place to another place. It's called ledamot mileta lemileta. It's issuing an original ruling."
That is, type A questions are not true pesak. They are just look-up. Only type B and, I would add, type C, is true pesak. Rebbetzin Henken might adopt the gavra approach.
In the same podcast, Rav Hershel Schachter maintained that, in theory, a knowledgeable woman could pasken – presumably even types B and C. After all, the Tosefta paskens like Beruriah in one place. This is the same pesak as cheftza approach. However, he expressed concern about the American training and practice of yo’atzot for different reasons. First, Rav Aharon Lichtenstein who approved it was told that the plan was that the yo’atzot would merely be an address, but that the yo’etzet would then turn to a posek; and was surprised when told that they were not doing that.
Second, he expressed concern that some yo’atzot were willing to pasken outside their training, e.g. feeling competent in ruling about whether to break a fast on Yom Kippur. Third, even an intense study program of targeted study, learning every Shach and Taz in the relevant simanim may not suffice. This is because Rabbinic Hebrew is not Modern Hebrew or Biblical Hebrew. It is its own language, with a mix of Hebrew and Aramaic, and idioms that have particular meanings. That kind of linguistic familiarity is formed not in two years, but in many years of study of lots of gemara. The Shach and the Taz use melitzot / expressions drawn from the gemara in general, and without this Talmudic familiarity, you might not understand what the Shach and Taz are talking about. He was told by younger rabbanim involved in shuls and balebatim that some of the American yo’atzot made major mistakes and wrongly permitted Biblical violations.
Thus, within the cheftza approach, for yo’atzot and LLMs, we must be certain that the pesak is accurate. And, while there is a requirement of gemir, namely knowledge of the relevant material, we must be certain that this material is properly understood.
This calls to mind Sanhedrin 5b, about יוֹרֶה, יוֹרֶה, obtaining permission to rule whether something is permitted or prohibited. The gemara asks: if he is knowledgeable, why need he obtain authorization? They cite a brayta describing how Rabbi Yehuda HaNasi visited a town and observed them kneading dough while ritually impure, because a Torah scholar taught that mei betz’aim (swamp waters) don’t render food susceptible to ritual impurity. Meanwhile, the Torah scholar had referred to mei beitzim (egg whites). Here, the Torah scholar knew the halacha but perhaps conveyed it unclearly, so that the townsfolk misunderstood his intent. Still, linguistic misunderstandings present a problem even if one is gemir.
LLMs and Semantic Shift
Assuming that ChatGPT can be gemir by being trained on the relevant texts or by fetching the relevant texts (but stay tuned for a future column), does that mean that it won’t misconstrue the words in those texts?
This summer, I attended the 19th World Congress of Jewish Studies. My presentation, “Rabbinic Hebrew, Semantic Shift, and LLMs”, was about how ChatGPT often mistranslated Chazalic and Rishonic Hebrew words when a modern Hebrew word had supplemented or supplanted the original word sense. I developed a dataset of 1851 such words and used it as a benchmark to test ChatGPT’s accuracy – it had an error rate between 7% and 15.5%.
For example, אמרתי לשלומיאל לבנות את המקדש is sometimes translated as “I told the shlemiel to build the Temple.” Adding בן צורישדי makes it render Shlumiel as a name. Other examples: tishboret means plane geometry in Medieval Hebrew and thus in Moreh Nevuchim, but ChatGPT would render it in Modern Hebrew, where it means arithmetic of fractions. Consider a modern work from 2015, Badei HaAron. The author discusses a query posed to the Mabit, sixteenth century Rav Moshe ben Yoseph di Trani, and writes
לאחר שהוא דן בהרחבה בסוגיית ארנונה ובסוגיית חובת שביעית בזמן הזה, הוא מוסיף ודן בסוגיה בעבודה זרה ומביא את דברי הרי"ד.
ChatGPT translates mostly accurately but renders arnona as the Modern Hebrew “municipal tax” rather than the Chazalic “crop tax”, or the neighborhood in southern Yerushalayim. It also renders the Rid as the Ra’avad.
Thus, even if an AI “knows” the relevant ideas or fetches the relevant text, it can misconstrue it. Gemir might not be enough. On the other hand, I have not yet extensively tested Dicta’s LLM and translator (translate.dicta.org.il), which was trained specifically on massive amounts of Rabbinic Hebrew. It would presumably do a better job at utilizing the Chazalic word sense where appropriate. Indeed, it left arnona untranslated as arnona. Next week, I hope to discuss whether LLMs actually know anything, and how that might approximate the gemir requirement.