To think inside the box, or to think out of the box?
Scientific discovery via the reciprocation of insights and concepts

Relying on conceptual knowledge is not an obstacle to investigating scientific discovery, but a better chance for understanding insight problem-solving. Current insight problem-solving tasks have been long troubled by the subjects’ unawareness of the meta-cognitive strategies they have used (Metcalfe and Wiebe, , 1987). Though this inability naturally unveils the sudden come of insights, it is avoidable by improving experimental tasks. Some current insight problems focus on stimulating the eureka moment, such as the nine-dot problem (Kershaw and Ohlsson, , 2004), the matchstick arithmetic (Knoblich et al., , 1999), and the eight-coin problem (Ormerod et al., , 2002)—these tasks provide highly confined problem space such that subjects can solve the problems without applying any semantics or commonsense knowledge other than the specific background knowledge given by the problem settings. Although such designs are motivated to disentangle representation reconstruction from solution cleanly, it makes interpreting the solutions from the trajectories hard, since the trajectories can only be mapped to the given problem settings. Hence, if we expect mapping behavioral trajectories to meta-strategic knowledge, a generally understandable semantics of problem context is necessary, such as conceptual knowledge in human language. Other insight problems introduce general semantics, including insight physical problem solving (i.e., intuitive physics as semantics) (Allen et al., , 2020) and remote association test (i.e., word association as semantics) (Mednick, , 1963). However, these tasks come in a one-to-one input-output fashion, where the measured behavior is directly generated from the single stimuli one-step, thus hard to track the representation change. This is crucial in scientific discovery because there are usually multiple steps of insights that lead to the target (Moszkowski, , 1972). In contrast to one-shot problem solving, scientific discovery is more like a path-finding process where the navigation map changes every time reaching a critical point. Putting the two reasons together, Mindle should equip with general-understandable semantics in the context of the target problem.

The domain knowledge of sciences is believed to be organized as conceptual knowledge (Hiebert and Lefevre, , 1986; Rittle-Johnson et al., , 2001). Conceptual knowledge systematically combines declarative and procedural knowledge, consisting of both facts about the concepts and active processes about how concepts interact with each other (Abend, , 2008). Though there are many perspectives on concept representation, we take the theory theory as a prerequisite because it is the most accepted theory in terms of scientific knowledge representation (Gopnik, , 1994), thus we can mimic the domain knowledge under the form of theory theory. One implementation of theory theory is that concepts are maintained in a fully-connected network, where each concept is related to all other concepts in the set—many calculi on fully-connected graphs, such as the general pattern theory (Grenander, , 2012), can be applied to formalize the operations over concepts. Such tools help describe scientific knowledge and meta-strategies in a computable way. Thanks to that, the theory theory can also model how a child acquires concepts in cognitive development (Carey, , 1985; Gopnik and Meltzoff, , 1997; Carey, , 2009); people similarly organize normal concepts to organize scientific knowledge. Hence, we may use such fully-connected networks statistically extracted from natural language corpus to simulate the domain knowledge of sciences, so we can carry out large-scale behavioral studies. Most interestingly, there is a major feature shared by both normal and scientific knowledge—though the semantics of concepts and relations are static and invariable viewing knowledge in the world holistically, they may be highly overloaded according to diverse context, task utility, and inner preference, from the view of individuals (Wang and Bi, , 2021). In this way, the compounded concepts are projected to simplified semantic attribute spaces (Grand et al., , 2022), which are much more tractable to be processed due to the rational use of limited cognitive resources (Gershman et al., , 2015; Lieder and Griffiths, , 2020; Ho et al., , 2022). On this basis, conceptual knowledge in scientific discovery should be studied in a dynamic way rather than a static way. However, most current behavior-level experimental methods on semantic understanding of concepts are confined to general and fixed contexts (Huth et al., , 2016; Wang et al., , 2020), in a straightforward way that is given the stimuli (input words) to obtain the descriptions or similarity judgments (output measurements) directly; other studies on the use of concepts, such as memory replay and human reinforcement learning (Momennejad et al., , 2017), mostly focus on the short-term memory for skill learning without retrieving from long-term memory that has been already formed. To capture the two features, Mindle should test the representation of conceptual knowledge in sequential decision-making.

In summary, we profile Mindle with two unique features to simulate the process of scientific discovery (see Tab. 1 for details), and importantly, the two work reciprocatively: (1) Mindle should equip with a general-understandable conceptual knowledge as domain knowledge to help interpret the process of insight problem solving; (2) Mindle should equip with a sequential decision-making task to stimulate the flexible dynamic use of conceptual knowledge.

Mindle requires the participants to dig out a hidden secret word. In every single challenge, the participant is given only a starting word. In each guess, the participant inputs a guessed word, and Mindle outputs a score (the given starting word can be viewed as the initial guess). The score indicates how far is the current guess to the target secret word, on a 0 to 100 scale—when the score is more close to 100, the guess is more close to the target. Once the secret word is hit, one challenge ends. The participant can choose to quit at any time during the challenge. The guesses can be entered by the participant through an input text bar or be selected by the participant through some given options; the policy for proposing options will be described later. A variant of challenge mode comes with a hint implying what topic the secret word is related to. The topic can either be abstract or concrete, such as kitchen supplies, classical music, or freedom, which confines the problem space.

The score is provided by the cosine similarity between the embedding vectors of the current guess and the target word. The embedding vectors can be generated by arbitrary embedding methods, such as Word2Vec (Mikolov et al., 2013b, ), Skip-gram (Mikolov et al., 2013a, ), or Glove (Pennington et al., , 2014). The score transforms the 0 to 1 scale of cosine similarity to a 0 to 100 scale. The vocabulary we used here is a subset consisting of about 40K most frequently used concepts in the natural corpus. Here we denote the set as $C\in\mathcal{C}$ where $\mathcal{C}$ is the space of all concepts. In bar-entering mode, participants would be informed if the input guess is out of $C$.

We have mentioned that conceptual knowledge can be represented as a fully-connected network. It can be implemented as a directed graph encoding an adjacent matrix. The graph is obtained from natural corpus through a Transformer model (Vaswani et al., , 2017). The connection weights in the graph capture the co-occurrence frequency for each pair of concepts. Let graph $G=\langle C,C\times C\rangle$ be the system of concepts, each node $c_{i}\in C$ denotes a concept, $c_{k},k\neq i$ is a possible related concepts that shapes $c_{i}$, and $w(c_{i},c_{k}),k\neq i$ indicates the weight $c_{i}$ is shaped by $c_{k}$ among all related concepts. The higher $w(c_{i},c_{k})$ is, the semantics of $c_{i}$ is more influenced by that of $c_{k}$. Note that usually $w(c_{i},c_{k})\neq w(c_{k},c_{i})$ because the weights indicates the conditional probability $p(c_{i}|c_{k})=w(c_{i},c_{k})/\sum_{j\neq i}w(c_{i},c_{j})$. Intuitively, this can be viewed as the probability of describing the concept $c_{i}$ by concept $c_{k}$ in a rational fashion (Frank and Goodman, , 2012). $p(c_{i})$ is a vector that has $p(c_{i}|c_{k})$ as its $k$-th dimension. And since the statistical feature is obtained from the general natural corpus, the real conceptual knowledge distribution in individuals’ minds is more or less different because that is highly conditioned on diverse individual prior experiences. In this way we define all general and individual operations over concepts as formal operations on a graph.

Solving Mindle can be ideally modeled as a mdp (mdp). An mdp is a tuple $T=(S,A,P,R,\rho)$, where $S=C$ is the set of states, $A=C$ is the set of actions, $P:S\times A\times S\mapsto[0,1]$ is the transition probability function $s_{t+1}\sim P(\cdot|s_{t},a_{t})$, $R:S\times A\times S\mapsto\mathbb{R}$ is the reward function (the game score here plays the role of reward) indicating in what condition the task is solved, and $\rho$ is the initial guess $\rho$ where $s_{0}\sim\rho$. Under this formulation, solving Mindle is finding the policy $\pi=(s_{0},a_{0},s_{1},a_{1},\dots)$ that generates trajectory $\tau\sim\pi$ optimizing the objective $\max_{\pi}\mathbb{E}_{\tau\sim\pi}[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},s_{t+1})]$. But this modeling is extremely ideal because the spaces of state and action are much smaller than the whole set of concepts, due to highly limited memory and attention slots. Hence, a mask $M$ can be applied to both $S$ and $A$ that $S\cdot M\subset C$ and $A\cdot M\subset C$. $M$ can either be predefined by heuristics (will be introduced next) or be extracted from trajectories generated by subjects in pre-experiments.

Given the current guess, where can we go? Take worker as the exemplar current guess, there are three types of candidate actions (see Fig. 2): (1) Concepts that are highly similar to current guess, e.g., labor, navvy, and staff. (2) Concepts that are highly related to current guess by specific rules, e.g., machine (by rule usage), salary (by rule reward), and supervisor (by rule organization). (3) Concepts that are hardly related to current guess by any mean, e.g., arts, word, and hippocampus. Suppose we have K proposals in each type and denote $c_{t}$ and $c_{t+1}$ as current and next guess, respectively. The similar concepts are generated by top-K $\max_{k}cos(\text{Vec}(c_{t}),\text{Vec}(c_{k})),k\neq t$, the related concepts are generated by top-K $\max_{k}w(c_{t},c_{k}),k\neq t$, and the unrelated concepts are generated by top-K $\min_{k}w(c_{t},c_{k}),k\neq t$. Most similar concepts are synonyms to $c_{t}$, which are not highly related to $c_{t}$; exceptions exist, thus making a filter afterward. When selecting related concepts, we try to maximize the diversity of the candidate concepts by filtering out duplicate concepts generated by the same rule, e.g., boss and supervisor (by rule organization). We implement this by applying Agglomerative clustering to the candidate concepts and selecting from the root (Murtagh and Legendre, , 2014). The three types are actions in the higher level of $A$, which can be observed directly and easily identified to decision patterns, such as persistent searching in a local minima or flexibly jumping between locals. Besides analyzing behavior data, the graph pruned by the three modes is also used to control the hardness of challenges by tuning the length of the shortest path and the number of possible paths traversing from the initial guess to the target word.

One significant indicator to be evaluated is the emergence of the eureka moment. We use both absolute and relevant measurements. First, we identify the Aha! moments in a single trajectory $\tau=(s_{0},a_{0},a_{1},s_{1},\dots,a_{-1},s_{-1})$. First, we calculate the first-order reward difference between adjacent trials $\Delta r(t)=r_{t+1}-r_{t}$. For action $a_{t}$, if there exists $i<t$ that $\Delta r(i)\geq\Delta r(t)$, let $r(i:t)=\sum_{k=i+1}^{t}r_{k}/(t-i)$, if not then let $i=0$; if there exists $j>t$ that $\Delta r(j)\geq\Delta r(t)$, let $r(t:j)=\sum_{k=t+1}^{j}r_{k}/(j-t)$, if not then let $j=-1$. We have $\Delta a(t)=r(t:j)-r(i:t)$ as the cumulative difference between the critical points. Those critical points with high $a(t)$ tend to be eureka moments from the view of a single trajectory. Also, we observe the trajectories in a counterfactual fashion, and we ask what would happen if other actions have been taken. The action space can be either the concept space $C$ or masked $C\cdot M$ in the low level, or the space of the three high-level action types (similarity, rule, and unrelated). Then, we define the updating rate as $\min\{0,1-\min_{a\in A}R(s_{t},a,s_{t+1})/R(s_{t},a_{t},s_{t+1})\}$. Thus, the current guess can be viewed as an intervention to the status quo following the semantics gradient, supporting investigations on semantics overload. The evaluation metrics can be flexibly modified to meet the need of specific experiments.

In the post-interview of our pilot study (25 subjects, 11 are female), after 3 challenges per subject, 24 in 25 subjects think that playing Mindle is interesting and 18 in 24 subjects think that I want to play Mindle everyday; 15 in 25 have succeeded at least one challenge and 8 in 15 have experienced at least one eureka moment in at least one challenge. All solved challenges are finished in about 80 steps of guessing on average, and the number of insightful solutions is about 50. These results imply that Mindle is attracting enough with an appropriate success rate, having the potential to propagate with ease. Also, Mindle is unique for its sense of infinite answer space and harmless trial (Hamari, , 2007). Thus, Mindle can be played through a long term, even though a challenge is disrupted in the middle—instead, a long period of discontinuous thinking may even stimulate the Aha! moment.

Some challenges succeed due to action-level insights. The trajectories have a common point—participants switch smoothly between searching in a local minima or making traversals between locals globally. Once they have been optimizing a local minima for a time without gaining a significant score increase, they changed to jump randomly between concepts that seem unrelated to each other, until they hit a local with a much higher score. Then they settle down again to optimize the guess locally with synonyms or similar concepts. The eureka moments usually come when hitting a hot solution in the global random jump. Interestingly, the concepts in the trajectories are highly different from each other, indicating that such behavior pattern is not constrained by semantics, but be driven by the inner preference of participants. That is, no matter what concepts I am guessing, I just switch between local search and global jump flexibly. This lead to a hypothesis: Do people apply meta-strategies ignoring the problem context?

Some challenges succeed due to semantic-level insights. This happens when participants hit a local with a relatively high score, which is easy to believe that target lies in this local. Affected by prior experience, participants tend to search in a subspace of the semantics space, where the selected concepts are projected onto a plate of reduced semantics space. For example, a participant has guessed school, class, and grade, where she projects the concept to the semantics subspace of school-related concepts. However, these words cannot help go further. The participant decides to project the anchor concept, say class, to another semantics space, say computer-related terminologies. Then, she guessed type and unexpectedly takes a large step toward the target. Hence, she understands that she has been trapped in a semantics subspace. This case shows that representation reconstruction can also change the semantics subspace. Hence, we come up with another hypothesis: Do people apply meta-strategies according to the problem context? This hypothesis seems to be in contrast to the one in the last paragraph, but combining these two together, we have a comparative hypothesis, which is more related to our big picture on scientific discovery: Do people use meta-strategy as policy regardless of context, or subject to the subjective understanding of context semantics?

Some challenges succeed through analytical solutions, especially when the participant, fortunately, reaches the right track at the start. The participant optimizes a gradient of the semantics landscape in mind. The gradient can be extremely flexible—for example, hierarchy, arts to painting to gallery; extent, large to larger to largest; or the distance with human, human to chimpanzee to monkey. The hypothesis space for such a gradient is almost infinite because the semantics space is in very high dimensionality (Grand et al., , 2022). Since the choice of the gradient is subjected to personal cognitive bias, the trajectories for the same challenge can be highly diverse. This echoes the diverse mindsets of different genres of science. And compared with previous work on testing personal diversity in knowledge representation (Wang and Bi, , 2021), Mindle (1) stimulates the spontaneous use of the commonsense knowledge, in contrast to other experimental paradigms that probe human knowledge representation explicitly; (2) empowers the scaling-up of pilot studies, both in the broadness of semantics and the diversity of subjects, to obtain more elaborated results on the landscape that where people converge or diverge on concept representation. By recovering all trajectories generated by the same group of participants, we can build a computational model that captures their semantics landscape, i.e., a function that outputs the sense of which concepts are more similar or more related to each other than to others. The function can be approximated through inverse reinforcement learning (Abbeel and Ng, , 2004), thus we can analyze group diversities of concept representation quantitatively. In this way, we may reverse-engineer the organization of conceptual knowledge in peoples’ minds.

Testing the three hypotheses helps us understand the interplay between insight-seeking and domain-knowledge-relying. First, on a confined problem space, we test the existence of these thinking patterns, by stimulating the spontaneous use of action-level metastrategies, semantic-level metastrategies, and semantic-level landscape optimization. After this, we study given different constrained topics, are people tend to emerge and converge to a set of similar parameters that controls the interplay. Assuming that insight-seeking and domain-knowledge-relying are on the two ends of a continuum, one possible hypothesis is that people rationally control the interplay according to their uncertainty on the use of domain knowledge—high uncertainty on knowing what may lead to reconstruction at action level; high uncertainty on deciding which may lead to reconstruction at semantic level; and low uncertainty may lead to maintaining the current representation. Such intuition defines the balance point between the two ends, thus irrational behaviors, e.g., more close to insight-seeking, can be identified comparing with the rational case. On this basis, we scale up the behavioral studies to generalize the results to larger groups of individuals, and also collect a large dataset of trajectories and reverse-engineer the semantic landscapes in the open domain.

The authors thank Mr. David Turner for inspiring and helpful discussions on designing Mindle.