An Exploration of Pattern Mining with ChatGPT

Iba & Isaku Iba and Isaku (2012) describe a holistic pattern mining approach that includes element mining, visual clustering, and the discovery of pattern “seeds”. Hanmer Hanmer (2012) documents patterns that facilitate the pattern writing and review process. Iba & Isaku (Iba and Isaku, 2016) present a detailed pattern language for mining Iba-style patterns. Weiss Weiss (2017) describes an empirical approach to discovering patterns from design documentation using word clouds to help identify pattern elements. Linden & Cybulski Linden and Cybulski (2009) show how the hermeneutic method can be used to induce patterns from interviews with designers. These existing approaches do not use AI.

De Paoli De Paoli (2023) describes an exploratory approach to conducting thematic analysis using LLMs. This paper is interesting for two reasons. It is an example of a hybrid approach that involves the collaboration between human researchers and an AI. Pattern mining and thematic analysis also involve similar inductive analysis steps. Schmidt Schmidt et al. (2023) share their experience using ChatGPT to discuss and refine a pattern description.

White White et al. (2023) document a catalog of patterns for prompt engineering, which the authors see as a way to program an LLM through prompts. Schukart Schuckart (2023) describes patterns for the effective use of generative AI in a work environment. However, patterns for integrating LLMs with data and tools have yet not been documented. I felt it would be a suitable test case for my approach.

The examples of applications from which the patterns were mined are shown in Table 2. For this experiment, the examples were applications implemented by the author as illustrative use cases of applying LLMs. The examples were chosen because they share certain elements, but also exemplify unique design challenges. Here we chose three examples, but more examples may be required.

To extract the common solutions, I provided the examples and then instructed ChatGPT to identify the recurring solutions common to those examples. The prompt begins with:

This is followed by the text of the application scenarios from Table 2. The prompt ends with:

The result is a list of solutions and their descriptions (for brevity, just the names of the solutions are shown below):

Here is the description of one of the solutions:

With the recurring solutions identified, I then asked ChatGPT to identify the common problems underlying the solutions:

The problem corresponding to the solution above was:

Next, I asked ChatGPT to compile the problem-solution pairs and describe them as patterns. The following prompt defines the expected output format by specifying the sections of a pattern:

Note that this template does not include a resulting context section. This section will be added during the refinement step. The resulting context should also include the related patterns that help address the consequences of the pattern. However, we cannot identify the relationships between the patterns until all the patterns have been described. These pattern descriptions created in this step can then be provided to ChatGPT as context.

Here is the output generated from the problem and solution for the Custom Application Logic pattern:

In this step, I used the following prompt to help me identify the affordances of the components. To guide ChatGPT and as a way of defining what we mean by affordances, the goal of this step is explicitly stated as part of the prompt.

The response to this prompt is a list of the affordances for each component. The result is better visualized as a table, see Table 3. These are affordances that can be actualized or realized by solutions, as well as, applications that use those solutions.

Now, we are ready to determine which patterns use or build on which affordances by asking:

This output lists the affordances actualized by each pattern. To make it easier to view the output, these affordance can then be collated into a table (Table 4). Each entry of the table represents an affordance and which patterns it is used in. For example, for the Custom Application Logic pattern the output is:

Table 4 shows the affordances cross-referenced with the patterns that use them. Based on the information in this table, we will add affordances to the solution section of the pattern descriptions. Documenting the affordances actualized by each pattern allows us to describe a solution in terms of the capabilities of the underlying components it uses or builds on.

Dependencies between the patterns can be explored with the help of the prompt below. This prompt includes instructions for what we mean by pattern dependencies and how to identify them.

Dependencies will be documented in the resulting context section of a pattern. For example, the generated resulting context for the Custom Application Logic pattern is:

Data sources provide data in a variety of formats.

$\ast\ast\ast$

Raw data is often unstructured, noisy, and not in a directly usable format.

The diversity of data sources and formats, the presence of irrelevant or redundant information, and the need for data to be in a specific structure for analysis or processing make this problem challenging.

Therefore,

Implement preprocessing steps to clean, normalize, and structure raw data into a usable format.

Data preprocessing and enhancement, including scraping, cleaning, and formatting raw data, is crucial to making the data usable for analysis by LLMs or storage in databases. The content generation capabilities of an LLM can also be used to preprocess text and normalize or summarize information before it is indexed in a database or further analyzed in an LLM.

$\ast\ast\ast$

In the customer support example, FAQs are parsed and structured for indexing. In the research assistant scenario, research abstracts are retrieved and preprocessed for topic modeling.

After data is retrieved and preprocessed, consider applying Data Structuring and Enhancement to organize and enhance the data to make it more suitable for analysis or querying. This is critical because structured, enhanced data can be more effectively indexed, searched, and analyzed, especially when dealing with complex queries or the need for semantic understanding.

Raw or preprocessed data lacks organization, categorization, or semantic richness.

$\ast\ast\ast$

Data is unstructured and lacks semantic organization, making it difficult to analyze or query effectively.

The inherent complexity of natural language, the variety of possible interpretations, and the need for semantic understanding to perform specific tasks contribute to the challenge.

Therefore,

Structure and enhance data using techniques like semantic indexing, embedding, or categorization to add meaningful organization and context.

To organize preprocessed data in an accessible manner, it is essential to structure the data (structured data storage), for example, as a list or in JSON format. Semantic indexing and retrieval can be used to enhance the retrieval process, making it more relevant and efficient. LLMs can help structure data by identifying semantic relationships or categorizations, which can, in turn, inform database indexing (semantic search and matching). The specialized analytical capabilities of tools such as topic modeling algorithms can also be used to structure and categorize text based on its content.

$\ast\ast\ast$

The vector database indexing of FAQs in the customer support example and the categorization of research abstracts into topics in the research assistant example demonstrate this solution.

Once data is structured and enhanced, Semantic Understanding and Synthesis often follow, where LLMs can use the organized data to generate insights, summaries, or responses.

LLM capabilities need to be augmented with specialized functions for analysis or processing.

$\ast\ast\ast$

Despite their versatility, LLMs cannot handle all specialized tasks required by specific applications.

The specialized nature of certain tasks, the limitations of LLMs in specific analytical domains, and the need for processing efficiency and scalability in make this a challenging problem.

Therefore,

Augment LLMs by integrating external tools that provide specialized functions or capabilities not inherent to LLMs.

LLMs can adjust their responses based on output from external tools (adaptive learning). The tools provide specialized analytical capabilities that LLMs do not inherently have.

$\ast\ast\ast$

The use of a topic modeling tool in the research assistant scenario and a clustering algorithm in the startup failure analysis example illustrate this integration.

The output of the tool may not be directly suitable for analysis, requiring Data Processing to transform the output into a suitable format that can be consumed by further processing steps.

An application requires deep understanding of context and generation of coherent, contextually appropriate content.

$\ast\ast\ast$

Simple models fail to capture the nuances of language and context, leading to superficial or irrelevant responses.

The complexity of natural language, the subtlety of context, and the need for nuanced understanding and synthesis in responses add to the challenge.

Therefore,

Harness LLMs for their deep semantic understanding and synthesis capabilities to generate coherent and contextually relevant content.

Natural language understanding and content generation are at the heart of this pattern, enabling LLMs to understand context and generate coherent, contextually relevant content.

$\ast\ast\ast$

The generation of topic labels and summaries in the research assistant example and the synthesis of cluster descriptions in the analysis of startup failures demonstrate this capability.

Following semantic understanding and synthesis, consider Adaptive Response to ensure that the system can handle edge cases or queries outside its training range. This is important to maintain user engagement and trust by providing relevant responses even when definitive answers are not available.

LLMs may encounter user queries that fall outside their training or the scope of available data.

Inevitable encounters with edge cases or limitations in data or LLM capabilities can lead to inadequate responses.

The unpredictability of user queries, the inherent limitations of LLMs, and the finite scope of available data all contribute to this challenge.

Therefore,

Design LLMs to generate adaptive responses, either by acknowledging limitations or by providing the best possible alternative response.

Natural language understanding enables an LLM to recognize when a user query falls outside its training or the scope of the data provided to it as part of the context. LLMs can craft responses that acknowledge limitations or provide the best possible alternative response (content generation).

$\ast\ast\ast$

The customer support example, where the LLM generates a response when no suitable FAQ answer is found, demonstrates adaptive response generation.

After adaptive response generation, revisit Semantic Understanding and Synthesis to further refine the system’s understanding and output based on user interactions and feedback. This could be achieved, for example, by fine-tuning the LLM or adding in-context examples. This cycle improves the system’s ability to generate even more contextually appropriate and helpful responses over time.

Applications of LLMs cover a wide range of domains, user needs, and data sources.

$\ast\ast\ast$

A one-size-fits-all approach does not adequately address the specific requirements of different applications.

The diversity of application domains, the specificity of user needs, and the variability of data sources contribute to the application’s complexity.

Therefore,

Develop custom logic that defines the specific integration and interaction of LLMs, data, and tools tailored to the goals of the application.

LLMs can be prompted to generate application-specific outputs required by the application (content generation). Data sources can be structured according to the custom logic of the application (data organization and categorization). Finally, interoperability and integration (for example, through APIs) are essential to enable the integration of multiple external tools and systems.

$\ast\ast\ast$

The specific sequences of data retrieval, processing, and LLM interaction in each example illustrate how custom application logic is tailored to meet different objectives.

After the custom application logic is created, revisit Tool Integration to ensure that all the necessary tools are effectively integrated into the workflow. Specifically, focus on the following aspects: data transformation (ensuring that each tool receives information in the required format), task alignment (matching the most appropriate tool to each task), and task granularity (the idea that each tool should perform a specific task or a small set of related tasks to improve maintainability).

The quality of the examples and the variety of design choices they represent will have a significant impact on the patterns that can be extracted. Also, to a large extent, patterns will only be generated for design choices that are reflected in the examples. Domain experts will be aware of other patterns and design choices, but these may not be directly supported by evidence from the examples.

ChatGPT is also known to occasionally produce generic answers, especially when asked specific questions that go beyond the data on which it was trained. This can also be observed in the patterns it generates. Out of seven patterns in this experiment, ChatGPT suggested one pattern that was quite general (Iterative Refinement and Feedback). While, iteration is obviously a widely used principle, including such a pattern in the pattern language does not strengthen the language. This pattern was also not supported by the pattern stories generated for the examples.

The output produced by ChatGPT should be taken as a starting point for the pattern descriptions. Pattern mining with ChatGPT should not be seen a one-way street where we simply accept the output it produces, but as a co-creation process where the AI augments our human ability.³³3In a recent book, Mollick Mollick (2024) refers to this human-AI collaboration process as co-intelligence. Thus, instead of asking ChatGPT to combine the various pattern elements it produced (initial description, resulting context, and affordances) into a complete pattern description, I chose to combine them manually.⁴⁴4I initially experimented with asking ChatGPT to expand the pattern descriptions given all the additional information. However, the result was wordy and formulaic.

This gave me a chance to digest what ChatGPT had produced, and ”humanize” the pattern descriptions, which meant reflecting them through the lens of a domain expert, correcting errors and inconsistencies, and adding explanations and my own interpretations. Besides general editorial changes to improve readability, there were also recurring specific changes, as discussed below.

Pattern names generated by ChatGPT were often too long to be useful as handles for referring to the patterns. For example, Custom Application Logic was renamed to Custom Logic.

The context element required frequent tweaking. ChatGPT’s output was structured in the format “when some need arises”, citing specific examples instead of a general description of the setting. For instance, the suggested context for Data Preprocessing read: “When raw data is needed for analysis or processing in applications like customer support systems, research assistants, or startup failure analysis.” This was generalized by making it explicit that the data could come in a variety of formats.

Occasionally, I had to add a missing resulting context. For instance, there was no resulting context for the Tool Integration pattern. However, since tools produce data in formats that may not be suitable for processing by the LLM, this data must first be processed by applying the Data Processing pattern. The explanation given in the resulting context was also sometimes lacking in detail. For example, in Adaptive Response, I added this explanation for why Semantic Understanding and Synthesis should be considered next: “This could be achieved, for example, by fine-tuning the LLM or adding in-context examples.” This is a necessary detail that was not highlighted as part of the known uses.

Sometimes, the terminology suggested by ChatGPT was also imprecise. For example, ChatGPT wrote that LLMs can be ”programmed”, when the term ”prompted” would be more accurate. This is to be expected, as ChatGPT does not always have the necessary domain knowledge. As a statistical model, it might also suggest a word that is commonly used, but not correct in a specific context.

Furthermore, some suggestions were too specific. For example, ChatGPT provided this explanation of an affordance: ”Tools like topic modeling algorithms can structure and categorize data based on content, which can then be indexed in databases.” However, the power of topic modeling is not limited to its use for indexing. While a topic model can be used to assign documents to topics, the output (e.g., the keywords associated with a topic) can also be analyzed by an LLM or another tool.

I did not use ChatGPT to add to the pattern descriptions, such as expanding the discussion of the forces or adding implementation details to the solution. Keeping the generated pattern descriptions concise gives human domain experts more opportunity to flesh out the patterns with their own insights into the domain. However, it might be interesting to explore how to design prompts to enhance the information added by domain experts, for example, by identifying gaps in their thinking.

There are three potential limitations of the current process: in how broadly the process is applicable, in how known uses are made available to the LLM for processing, and in the how the outcomes depend on the quality of the prompts. I will discuss each in turn.

As an exploratory paper, this paper does not pretend to have all the answers. There is much room for improvement and expansion of the work described. Some of these opportunities are identified next in the form of future experiments to be conducted.

One interesting experiment would be to compare the granularity of the patterns extracted with the patterns extracted by domain experts when given the same known uses. For example, in the case study (integrating LLMs with data and tools), one might have expected two patterns (Tool Integration and Data Integration), instead of one pattern (Tool Integration) that covers both cases, treating data sources as a type of tool. However, one could also argue that this choice has some validity and that the two patterns would necessarily have significant overlap.

A variant of this experiment would be to apply the process to an existing pattern language, starting with the known uses and examples documented in the pattern language. In this way, one could compare the output produced by the process with the actual patterns in the pattern language. Such an experiment would focus on the completeness of the pattern language rather than the granularity of the patterns.

A related experiment would be to repeat the prompts several times, starting from the same examples, and observe what differences, if any, are produced, and whether the differences are significant. A difference would be considered significant if one experiment produces a pattern that a domain expert would consider essential, and another experiment does not produce a similar pattern. A difference would not be considered significant, if the patterns merely ”partition” the space of design choices in different ways. Such differences are, at least to some extent, a matter of preference.

We could also build on existing work using thematic analysis for pattern mining such as the hermeneutic approach by (Linden and Cybulski, 2009), and combine it with the recent work on conducting thematic analysis using ChatGPT (De Paoli, 2023). As noted above, this would make the proposed process more scalable by allowing it to include a larger and richer set of known uses.

Another aspect to be discussed in future work is the effectiveness of using affordances to describe the details of a solution. Can the concept be generally applied or is limited to domains where we can model pattern solutions in terms of components.

A low-hanging fruit is also to experiment with modifications to the prompts themselves. For instance, we might integrate explicit definitions of key concepts (e.g., patterns or affordances), or more specific instructions for conducting the steps of the process.