GPT3-to-plan: Extracting plans from text using GPT-3

We use the three most common datasets for action sequence extraction tasks used in evaluating many of the previous task-specific approaches, including Feng, Zhuo, and Kambhampati (2018) or Miglani and Yorke-Smith (2020). Namely, the ”Microsoft Windows Help and Support” (WHS), the ”WikiHow Home and Garden” (WHG) and the ”CookingTutorial” (CT) datasets. The characteristics of these datasets are provided in Table 1.

The GPT-3 model is currently hosted online³³3More information at https://beta.openai.com/ and can be accessed via paid user queries with either their API or website in real time. Some example use cases of their service include keyword extraction from natural text, mood extraction from reviews, open-ended chat conversations and even text to SQL and JavaScript to Python converters amongst many others. In general, the service takes free natural language as input and the user is expected to encode the type of interaction/output desired in the input query. The system then generates output as a completion of the provided query. The API also allows the user to further tweak the output by manipulating the following parameters: Max Tokens sets the maximum number of words that the model will generate as a response, Temperature (between 0 and 1) allows the user to control the randomness (with 0 forcing the system to generate output with the highest probability consistently and rendering it effectively deterministic for a given input). Top P also controls diversity; closer to 1 ensures more determinism, Frequency Penalty and Presence Penalty penalize newly generated words based on their existing frequency so far, and Best of is the number of multiple completions to compute in parallel. It outputs only the best according to the model. In Table 2 we show the values that we used for all our experiments to ensure the most consistency in the model’s responses.

Each query consists of a few shot training in natural language text and the corresponding structured representation of the plan. For each example, we annotate the beginning of the natural language text portion with the tag TEXT followed by the plan (annotated with the tag ACTIONS). In the structure representation, each action is represented in a functional notation of the form $a_{0}^{j}(arg_{0}^{0},arg_{1}^{0}\dots arg_{k}^{0})\dots\ a_{n}^{j}(arg_{0}^{n},arg_{1}^{n}\dots arg_{k}^{n})$ where $a_{i}^{j}$ represents action i in sentence $j$ and $arg_{k}^{n}$ is the kth argument from action $a_{n}$ in the text. After the training pairs, we include the test sample in natural language text after another tag TEXT and then we add a final tag ACTIONS, with the expectation that GPT3 will generate the corresponding plan representation after that.

In order to directly compare the performance of GPT-3 to Miglani and Yorke-Smith (2020), the current state-of-art, we followed a translation scheme with three types of actions, namely, essential (essential action and its corresponding arguments should be included in the plan) exclusive (the plan must only contain one of the exclusive actions) and optional actions (the action may or may not be part of the plan). We use this scheme to generate both the example data points provided to the system and to calculate the final metrics.

In particular, we will use precision, recall and F1, similar to Feng, Zhuo, and Kambhampati (2018); Miglani and Yorke-Smith (2020) to measure the effectiveness of the method.

$$\begin{split}Precision&=\frac{\#TotalRight}{\#TotalTagged},\ Recall=\frac{\#TotalRight}{\#TotalTruth}\\ \vspace{2em}F_{1}&=\frac{2\times precision\times recall}{precision+recall}\end{split}$$

(1)

Note that the ground truth number and the number of true extracted actions depend on the type that each action in the text corresponds to. For example, a set of exclusive actions only contribute one action to #TotalTruth and we only count an extracted exclusive action in #TotalRight, if and only if, one of the exclusive actions is extracted. Both essential and optional actions only contribute once to #TotalTruth and #TotalRight.

In Table 3 we compare GPT-3 to several action sequence extractor models:

• •

  EAD: Mei, Bansal, and Walter (2016) design an Encoder-Aligner-Decoder method that uses a neural sequence-to-sequence model to translate natural language instructions into action sequences.

• •

  BLCC: The Bi-directional LSTM-CNN-CRF model from Ma and Hovy (2016) benefits from both word and character-level semantics and implement an end-to-end system that can be applied to action sequence extraction tasks with pre-trained word embeddings.

• •

  Stanford CoreNLP: in Lindsay et al. (2017) they reduce Natural Language texts to action templates and based on their functional similarity, cluster them and induce their PDDL domain using a model acquisition tool.

• •

  EASDRL and cEASDRL: Feng, Zhuo, and Kambhampati (2018) and Miglani and Yorke-Smith (2020) use similar reinforcement learning approaches; they define two Deep Q-Networks which perform the actions of selecting or rejecting a word. The first DQN handles the extraction of Essential, Exclusive and Optional actions while the second uses them to select and extract relevant arguments.

The corresponding precision, recall and F1 scores for each method were picked directly from their respective papers.

Given that GPT-3 is a few-shot learner we want to know how it performs given different amounts of training samples. To measure this, we query the language model with increasing numbers of examples (with a maximum of four examples) for all domains and report their F1 scores. We stop at the four-shot mark as the total amount of tokens or words that the request can contain is 2048. Additionally for the CookingTutorial and Wikihow Garden and Home datasets, 4-shot training examples already exceed this threshold, so we limit the length of input text to 10 sentences per training example. Specifically, we select the training examples as 1-shot (one datapoint is selected at random from the dataset), 2-shot (the two datapoints with the largest proportion of optional and exclusive actions from the dataset are selected), 3-shot (the three datapoints with the largest proportion of optional, exclusive and essential actions) and 4-shot (an additional random datapoint is added to 3-shot).

In Figure 1 we show how the $F_{1}$ score changes given 1, 2, 3 and 4-shot training samples when tested on the whole Windows Help and Support dataset. Unsurprisingly, Davinci, the model with the most amount of trainable parameters, performs best with over $80\%\ F_{1}$ score for each category. Both Davinci and Curie show the tendency to perform better the more examples they are given peaking at 3 and 4-shots respectively. Similarly, Babbage and Ada show their peaks given 2 and 4 examples while underperforming at one-shot training. This is unsurprising, given the fact that these models are simplified versions of GPT-3 which have also been trained on a smaller corpus of data for higher speed. Hence, they need more than one example to grasp the task.

In table 3 we compare the $F_{1}$ scores for action name and their argument extractions as reported by previous and current state of the art task-specific action sequence extractors against all GPT-3 engines: Davinci, Curie, Babbage and Ada, ordered from most to least powerful The scores are calculated based on 1 and account for essential, exclusive and optional actions and their respective arguments. All GPT-3 models are trained with two-shot examples. As expected, Davinci overall performs the best compared to the rest of engines. We can see that Davinci also outperforms the EAD, CMLP and STFC task-specific models for the Windows Help and Support domain on extracting actions. Even though it underperforms on the argument extraction task compared to the state of art, it’s worth nothing that still obtains better than random extraction scoring.

We want to assess whether GPT-3 is capable of inferring plan order from text. This is a feature which is mostly missing in previous task-specific state of the art like Feng, Zhuo, and Kambhampati (2018) or Miglani and Yorke-Smith (2020). As a preliminary evaluation, we create three examples (one for each dataset, shown in Figure 2), where order of the plans does not match how actions are listed in the text. In the Windows Help and Support example, we state on the second and third sentences that action click(advanced) must be performed eventually but only after click(internet, options), and, even though the corresponding sentences appear in the opposite order, GPT-3 places them as expected. Similarly, in the CookingTutorial example, we state that first we need to measure the quantity of oats and cook them only later and once again, it generates the actions in correct ordering. For the last example, GPT-3 shows to understand that action paint(walls) has to be done before remove(furniture) and, interestingly, even though decorate(floor) is stated on the first sentence, the model seems to understand that it can be performed anytime and places the action last. Note that these are just anecdotal evidences and we would need to perform studies over larger test sets to further evaluate GPT-3’s ability to identify the ordering of plans. Our current evaluation along this dimension is limited by the lack of annotation regarding the ordering in the currently available datasets and one of our future works would be to create/identify such text-to-plan dataset with additional annotations on action ordering.