Estimate Emotion Probability Vectors: Interrogating the LLM with an Emotion Eliciting Tail Prompt

Table of Links

• Abstract and Introduction
• Interrogating the LLM with an Emotion Eliciting Tail Prompt
• PCA analysis of the Emotion of Amazon reviews
• Future Work
• Conclusions
• Acknowledgments and References

2. Interrogating the LLM with an Emotion Eliciting Tail Prompt

In this work we used Facebook’s open source LlaMa2 7 billion weight LLM as the core engine [2]. It was necessary to use a LLM that allowed access to raw token probabilities after a prompt. The model ran on a Mac Studio with 32 gigabytes of RAM. With this combination of hardware and model it took 2 minutes to compute the probabilities of the emotion descriptors in the emotion dictionary given below.

2.1. The Emotion Dictionary

The English language is blessed with many words and an extensive literature providing examples of the usage of these words in appropriate contexts. For the purposes of LLMs, it is the context of a word that conveys it’s meaning. A reader will infer the meaning of an unfamiliar word through the context they find the word used in, provided they understand the context. As an example, ‘The shotgun discombobulated the rabbit.’ shows how meaning can be moderated by context. The context created by a tail prompt will favour an associated class of words. For the experiment detailed in this paper the following tail prompt was used to elicit emotion descriptors, ’Reading this makes me feel’. It is likely that specific emotion eliciting tail prompts will favour specific sub classes of descriptor but studying this is beyond the scope of this paper

The following words were chosen to provide a broad sample of emotion descriptors.

acceptance, admiration, adoration, affection, afraid, agitation, agony, aggressive, alarm, alarmed, alienation, amazement, ambivalence, amusement, anger, anguish, annoyed anticipating, anxious, apathy, apprehension, arrogant, assertive, astonished, attentiveness, attraction, aversion, awe, baffled, bewildered, bitter, bitter sweetness, bliss, bored, brazen, brooding, calm, carefree, careless, caring, charity, cheeky, cheerfulness, claustrophobic, coercive, comfortable, confident, confusion, contempt, content, courage, cowardly, cruelty, curiosity, cynicism, dazed, dejection, delighted, demoralized, depressed, desire, despair, determined, disappointment, disbelief, discombobulated, discomfort, discontentment, disgruntled, disgust, disheartened, dislike, dismay, disoriented, dispirited, displeasure, distraction, distress, disturbed, dominant, doubt, dread, driven, dumbstruck, eagerness, ecstasy, elation, embarrassment, empathy, enchanted, enjoyment, enlightened, ennui, enthusiasm, envy, epiphany, euphoria, exasperated, excitement, expectancy, fascination, fear, flakey, focused, fondness, friendliness, fright, frustrated, fury, glee, gloomy, glumness, gratitude, greed, grief, grouchiness, grumpiness, guilt, happiness, hate, hatred, helpless, homesickness, hope, hopeless, horrified, hospitable, humiliation, humility, hurt, hysteria, idleness, impatient, indifference, indignant, infatuation, infuriated, insecurity, insightful, insulted, interest, intrigued, irritated, isolated, jealousy, joviality, joy, jubilation, kind, lazy, liking, loathing, lonely, longing, loopy, love, lust, mad, melancholy, miserable, miserliness, mixed up, modesty, moody, mortified, mystified, nasty, nauseated, negative, neglect, nervous, nostalgic, numb, obstinate, offended, optimistic, outrage, overwhelmed, panicked, paranoid, passion, patience, pensiveness, perplexed, persevering, pessimism, pity, pleased, pleasure, politeness, positive, possessive, powerless, pride, puzzled, rage, rash, rattled, regret, rejected, relaxed, relieved, reluctant, remorse, resentment, resignation, restlessness, revulsion, ruthless, sadness, satisfaction, scared, schadenfreude, scorn, self-caring, self-compassionate, self-confident, self-conscious, self-critical, self-loathing, self-motivated, self-pity, self-respecting, self-understanding, sentimentality, serenity, shame, shameless, shocked, smug, sorrow, spite, stressed, strong, stubborn, stuck, submissive, suffering, sullenness, surprise, suspense, suspicious, sympathy, tenderness, tension, terror, thankfulness, thrilled, tired, tolerance, torment, triumphant, troubled, trust, uncertainty, undermined, uneasiness, unhappy, unnerved, unsettled, unsure, upset, vengeful, vicious, vigilance, vulnerable, weak, woe, worried, worthy, wrath.

This set of words is not intended to be complete or definitive in any way. Using the tail prompt without restricting the return to emotion descriptors elicits general waffle responses that are not straightforward to extract sentiment form.

2.1.1. Estimating the emotion probability vector

LlaMa2 [2] has been released in such a way as to allow developers to access estimated token weights returned in response to a prompt. LlaMa2 has an internal vocabulary size of roughly 30,000 tokens. This means that when LlaMa 2 estimates the probability of the next token in a sequence the probability vector will have 30,000 elements. Some of the words in the emotion descriptor list are made up of more than one token in which case forward conditional probabilities are used.

Figure 2.1.1 shows the scaled probability distribution over words from the emotion dictionary elicited by the tail prompt for the Amazon review text, ‘I read a lot of negative reviews about the Fitbit inspire 2, I took a chance and hoped the one I ordered would be one of the great ones that worked. Unfortunately that was not the case. I unpacked it, charged it, downloaded the app. I took a walk with it on before the sun went down. I have the Google Fit app on my phone that tracks my steps also. The phone was in my jeans pocket. When I got home I compared the two, Google Fit said 4,458 steps, Fitbit said 1,168. Apparently Fitbit works with wrist motion which I don’t have while pushing a walker around the neighborhood. I downloaded the manual and noticed you can put it on a clip (that wasn’t included). That would work for me. So I started to scroll through the different features except I couldn’t scroll through all of them. While scrolling I must have turned on the stopwatch. I couldn’t turn it off. Then I couldn’t scroll through anything except water lock feature. I had to turn on the water lock to get back to the stopwatch. Then the side buttons stopped working. I had it a total of 5 hours. I packed it up and started the Amazon return. I did get a full refund. Very disappointing.’

The text from 50 Amazon reviews of a book was borrowed from https://www.amazon.com/dp/B000WM9UK2. The reviews were for the most part favourable. Example review texts include: ‘The Children of H´urin is a great tragedy mixed with grace. The genealogical records at the beginning may be difficult to get through, but the story very quickly gains speed. I only gave the story four stars because of the difficulty of the first few chapters, much like Matthew’s genealogy of Christ at the beginning of his gospel. Although such records are important in both they are nonetheless difficult to get through. I still definitely would recommend this book, though, as it depicts the horrible effects of evil powers on good men and women, and yet, we must continue to resist evil no matter the tragic end. It is very telling that in Tolkien’s world, at the end of days when Morgoth returns, that it is T´urin, a man, who ends him once and for all. Those whom Satan most destroys in this life are they who will ultimately deal his death blow, as the Revelation says, “They overcame him by the blood of the Lamb and by the word of their ‘martyria,’ [witness, testimony],” those like T´urin, or in Scripture, those like Job.’

Figure 2.1.2 shows the emotion vectors for all 50 processed reviews. The 10 most probable emotions experienced from purchasing the product were: depressed, kind, nostalgic, tired, hopeless, lonely, hope, calm, lazy, confident.