Technical Appendix
RecallM: An Adaptable Memory Mechanism with Temporal Understanding for LLMs
System Details
The details of the machine used to conduct all experiments are listed below:
Operating System | Windows 10 Pro (64-bit) |
---|---|
Processor | Intel Core i9-9900 CPU @ 3.10GHz |
Installed RAM | 32GB |
Graphics Card | NVIDIA GTX 980Ti |
Installed VRAM | 6GB |
All experiments were conducted using gpt-3.5-turbo (July 2023 version), Neo4J Desktop, ChromaDB (version 0.3.21) and LangChain (version 0.0.157). We provide detailed instructions on how to easily install RecallM and reproduce these experiments in the Code and Data Appendix.
TruthfulQA Results
It is worth noting that some of the main topics discussed in the subset of TruthfulQA used for the knowledge update, include common misconceptions, myths, conspiracy theories, superstitions and various topics in science. Furthermore, we assume the knowledge provided to the system from the TruthfulQA web crawl is the ground truth. Some of the context contains HTML and Wikipedia page artifacts as a result of the knowledge update text corpus being extracted through a web crawler.
Questions Directly from TruthfulQA
table
Our Own Questions After Performing a Knowledge Update Using TruthfulQA
table2
Temporal Experiment Dataset
Below are the statements and question-answer pairs used to conduct the temporal experiment. The statements are used to update the knowledge of the system in order from top to bottom. These statements are entirely fictional. The questions and reference answers used to human-grade the systems are presented in the tables below. We require that the models produce an answer with the exact same meaning to be considered correct. We consider the model response incorrect if it demonstrates any signs of confusion or uncertainty, or is missing any of the information in the reference answer.
Please note that for the purpose of this blind review we have replaced actual names and places with placeholder text. To reproduce these experiments, these placeholders should be replaced with real names, places or organizations.
Standard Questions
temp_questions
Long-Range Questions
long_temp_questions
Timestep () | Truth statement for knowledge update |
INITIAL | |
1 | personA is South African. |
2 | personC is American. |
3 | personD ’s nationality is Romanian. |
4 | personA lives in Townhome 2. |
5 | personB is employed at COMPANY . |
6 | personA loves coffee. |
7 | personA used to work at PENCIL Inc, although he no longer works there. |
8 | personA can speak English, French and Dutch. |
9 | personA broke his leg in a car crash when he was 10-years-old. |
10 | personA has only one brother called personF . |
END | |
LOOP REPETITION | |
11 | personA doesn’t like rock music anymore and now loves house music. |
12 | personA now works for COMPANY . |
13 | personA does not like tea. |
14 | personC likes tea. |
15 | personC also likes coffee. |
16 | personE does not like coffee. |
17 | personA just ate a steak. |
18 | personA is tired. |
19 | personA wants to go to Paris for his next vacation. |
20 | personC is planning on travelling to Japan for his next vacation. |
21 | personA does not like hiking, he never has. |
22 | personA ’s favorite color is orange. |
23 | personE ’s brother’s name is personG . |
24 | personC ’s brother’s name is personH . |
25 | personA ’s favorite color is blue. |
26 | personC hates hiking. |
27 | personA still works for COMPANY . |
28 | personA is planning on going to the gym at 7PM today. |
29 | personA is still working for COMPANY . |
30 | personC can speak Japanese. |
31 | personA just ate a salad. |
32 | personA is looking forward to his vacation in Paris. |
33 | personA lost his taste for house music and now likes country music. |
34 | personC ’s favorite music genre is classical. |
35 | personA got laid off from his current job and no longer works there. |
36 | personA is not tired anymore. |
37 | personA ’s favorite color is yellow. |
38 | personE broke his leg when he was 18-years-old. |
39 | personA does not work for COMPANY anymore. |
40 | personC just ate some chicken. |
41 | personC ’s first language is English. |
42 | personC can speak 2 languages. |
43 | personD convinced personA to go hiking soon. |
44 | personD does not have a brother or any siblings. |
45 | personA has started to enjoy hiking a lot. |
46 | personA ’s friends invited him to go bowling this weekend. |
47 | personA is going bowling with his Friends this weekend. |
48 | personA is still planning on going bowling with his Friends this weekend. |
49 | personA is now still unemployed. |
50 | personA just ate a banana. |
51 | personA likes hiking now. |
52 | personA is stuck in traffic and is now planning on going to gym tomorrow. |
53 | personA found a job at Lightbulb Ltd and is now employed. |
54 | personA is now tired. |
55 | personA ’s favorite color is red. |
56 | personA quit his current job after just 2 weeks of working there. |
57 | personC has never broken his leg. |
58 | personC has never broken any bones in an accident. |
59 | personA is unemployed. |
60 | personA is tired. |
61 | personC is employed at Potato Properties. |
62 | personC is currently working for Potato Properties. |
63 | personA cancelled his Paris trip, and is now planning on going to Brazil instead. |
64 | personC is planning on going to the gym today. |
65 | personA cancelled all of his previous gym plans and is now going to the gym in 3 days from now. |
66 | personA hates soft drinks. |
67 | personA is not tired anymore. |
68 | personA doesn’t listen to country anymore and loves Rock music now. |
69 | personA found a new job at COMPANY and is no longer unemployed. |
70 | personA is now going to a poker tournament this weekend instead of bowling with his friends. |
71 | personA works for COMPANY now. |
72 | personA ’s favorite color is green. |
END | |
… The statements now repeat from … | |
73 | personA doesn’t like rock music anymore and now loves house music. |
… | … |