Expressivity of Neural Networks with Random Weights and Learned Biases

The universal approximation theorems Hornik et al. (1989); Funahashi (1989); Hornik (1991) of the late 1900s highlighted the expressivity of neural network models, i.e. their ability to approximate or express a broad class of functions through tuning of weights and biases, heralding the central role neural networks play in Machine Learning (ML) and neuroscience today. Since these foundational studies, a rich literature has explored the limits of the expressivity of neural networks by finding smaller parameter subsets whose tuning still results in the approximation of wide classes of functions or dynamics. Prior work has explored approximation capabilities of feed-forward (FFNs) and Recurrent Neural Networks (RNNs) where only the output weights are trained Rosenblatt et al. (1962); Rahimi and Recht (2008); Ding et al. (2014); Neufeld and Schmocker (2023); Jaeger and Haas (2004); Sussillo and Abbott (2009); Gonon et al. (2023); Hart et al. (2021), and deep FFNs where only subsets of parameters Rosenfeld and Tsotsos (2019), normalization parameters Burkholz (2023); Giannou et al. (2023), or binary masks–either over units or parameters–are trained Malach et al. (2020). Recently, a study has also explored the approximation abilities of transformers where only context is tuned Petrov et al. (2024). Here, motivated by recent insights from neuroscience, we initiate the investigation of expressivity in both FFNs and RNNs where only biases are trained, an avenue that remains largely unexplored.

Learned biases are of fundamental relevance to neuroscience and ML. In the latter domain, recent work has highlighted the optimization or the careful selection of bias vectors: fine-tuning of biases for multi-task learning Zaken et al. (2021) and the context tokens used for in-context learning Von Oswald et al. (2023) can all be viewed as methods of carefully setting the bias in a model where other connectivity parameters are fixed in order to perform a task. In neuroscience, single neurons employ diverse mechanisms to adjust their response to inputs, beyond synaptic plasticity. Importantly, some of these mechanisms can be construed as manipulating the firing onset of neurons which, in a firing rate model, is represented by the bias. Among these, we have shunting inhibition Holt and Koch (1997), threshold adaptation Azouz and Gray (2000), and a host of other mechanisms that participate in shaping the input-output transfer function of neurons (reviewed in Ferguson and Cardin (2020), see also Mazzucato et al. (2019)). Experimental evidence also suggests that bias-related signals play a role in learning Zhang and Linden (2003); Sehgal et al. (2013) but, despite this, most work modelling learning in neuroscience has focused on synapses.

If tuning the biases of a neural network will only span a reduced set of functions, or output dynamics, then this would solidify the role of synaptic plasticity as the critical component in biological and artificial learning. Conversely, if one can express arbitrary dynamics solely by changing biases, this would motivate deeper investigation of when and how non-synaptic plasticity mechanisms might shoulder some of the effort of learning. In this paper we address the question of the expressivity of bias learning. In a regime where all weight parameters are randomly initialized and frozen, and only hidden layer biases are optimized, we provide theoretical guarantees demonstrating that:

1. 1.

   feed-forward neural networks with wide hidden layers are universal function approximators with high probability;

2. 2.

   RNNs with wide hidden layers can arbitrarily approximate finite-time trajectories from smooth dynamical systems with high probability.

We provide empirical support for, and a deeper interrogation of, these theoretical findings with an array of numerical experiments.

Neuroscience. Evidence from experimental and theoretical neuroscience shows that cortical circuits flexibly regulate their dynamical phases and toggle between different regimes solely by changing the distribution of input biases to a circuit. Bias-induced modulations can explain the context-dependent effects of expectation Mazzucato et al. (2019), movements Wyrick and Mazzucato (2021) and arousal Papadopoulos et al. (2024) on sensory processing across modalities and brain areas. Changes in the bias distribution modulate the gain of the input-output neuronal transfer function Mazzucato et al. (2019). Moreover, two bias-related biological mechanisms, neural firing threshold and network inputs, have also been noted for their ability to shape network dynamics: theoretical work shows that threshold heterogeneity can improve network capacity Gast et al. (2024, 2023), while a mix of theory and experimental evidence suggests a role for inputs from upstream brain areas in reconfiguring circuit dynamics on fast timescales Perich et al. (2018); Remington et al. (2018). Within the RNN reservoir computing approach to modelling in neuroscience, where recurrent weights are random and fixed, bias modulations can toggle between multiple phases (including fixed point, chaos, and multistable regimes) and, strikingly, enable RNN multi-tasking in the absence of any parameter optimization Ogawa et al. (2023). A repertoire of dynamical motifs can also be generated in RNN reservoirs with dynamic feedback loops Logiaco et al. (2021). However, in all these studies input biases or other parameters were either i.i.d. generated or set to specific values, but never learned. The role of plastic biases has not been investigated yet and will be the main focus of our efforts below.

Machine Learning. We mention primarily two bodies of literature in ML/AI that are related to this study. Many efforts have explored neural networks that are trained to quickly meta-learn new tasks via dynamics in activation space alone, without any adaptation of weights (see e.g. Feldkamp et al. (1997); Klos et al. (2020); Cotter and Conwell (1990, 1991)). Like our work, this thread of research proposes a mechanism by which a network might adapt to any new task without changing weights. However, prior work differs form the current study in that it requires an initial meta-training of all parameters in a network, weights included, before operating in the "fast learning" regime where network variables maintain context information that allow the networks to rapidly adapt to new tasks. In some cases these context variables can be thought of as biases Cotter and Conwell (1991). The closest ML contributions to our work are on the topic of masking–particularly the Strong Lottery Ticket Hypothesis (SLTH). This hypothesis conjectured that a given desired parameterization of a neural network can be found as a sub-network in a larger, appropriately initialized, network Ramanujan et al. (2020). SLTH is typically formulated with respect to weights, i.e. a subnetwork is defined by deleting weights from the full network. However, a few studies have investigated SLTH when subnetworks are constructed by deleting units Malach et al. (2020). While our study is different at face value for its focus on function approximation via bias optimization, rather than finding “lottery ticket" subnetworks, a key step in our analytic derivations relies on masking in a fashion analogous to proofs of the unit masking version of SLTH. Thus, our work also provides two results that may be of interest to the SLTH theory. First an alternative proof (complementing the first proof in Malach et al. (2020)) of the SLTH for units in single-hidden layer neural networks; second, a first proof of the SLTH over units for RNNs (see Section §2 for more details). Flavours of the lottery ticket hypothesis for RNNs has been explored empirically Yu et al. (2019); García-Arias et al. (2021); Schlake et al. (2022) but we have not encountered its proof, over weights or units, in the literature.

This section studies the single-layer FFN, whose output is given by

with $A\in\mathbb{R}^{l\times n}$, $B\in\mathbb{R}^{n\times d}$, $b\in\mathbb{R}^{n}$, and $\theta=\{B,A,b\}$. We shall investigate the approximation properties of this neural network when all the weights in $B$ and $A$ are fixed and sampled uniformly from the $n(l+d)$ dimensional centered hypercube, with half-edges of length $\gamma$, and only $b$ is tuned. We begin by outlining the activation function assumptions necessary for our theoretical results.

From the universal approximation theorems of 1993 Leshno et al. (1993); Hornik (1993), a sufficient condition for $\phi$ to be a suitable activation is that it is non-polynomial. In this paper we conceptualize universal approximation as the approximation of continuous functions on compact sets with respect to an $L^{1}$ functional norm, but we remark that the literature has also studied other conditions on $h$ (for example, continuity and measurability) and other forms of convergence (for example, with respect to the $L^{\infty}$ norm). For a review of the literature, see Pinkus (1999).

The proof is in Appendix §2.1 for completeness. A key subtlety of parameter-bounding is that it is a bound on individual, scalar, parameters. Thus, as a network grows in width the bias vector and weight matrix norms will still grow accordingly. This may be important, as research suggests band-limited parameters cannot universally approximate, at least for certain activation types Li et al. (2023). We leave it to future work to determine which other activations are parameter bounding.

We make one final definition:

Obviously the ReLU is a bias-learning activation. We leave the study of discontinuous functions like the Heaviside to future work. We conclude with the main result of this section. Define $p_{R}$ to be a uniform distribution on $[-R,+R]$, where $\gamma<R<\infty$.

Proof Intuition: We provide intuition about the proof of Theorem 1 and its Corollary, whose details can be found in the Appendix. According to the universal approximation theorem, given a continuous function, we can find a one-hidden-layer network, $\mathcal{N}_{1}$, that is close to that function in the $L^{1}$ norm on the (compact) space of its inputs. If $\mathcal{N}_{1}$ has been constructed using $\gamma$-parameter bounding activation functions, then we know that each parameter will be on the interval $[-\gamma,\gamma]$. Next, we construct a second network, $\mathcal{N}_{2}$, to approximate $\mathcal{N}_{1}$ by randomly sampling each of its parameters, weight or bias, from $p_{R}$. For $\mathcal{N}_{2}$ to approximate $\mathcal{N}_{1}$, each parameter of $\mathcal{N}_{2}$ should fall within a tiny window of an analogous parameter in $\mathcal{N}_{1}$. This window must have half-length less than $\epsilon$ to yield the desired error bound. Without loss of generality we can assume $\epsilon<R-\gamma$. Then, if we sample parameters uniformly on $[-R,R]$ there will be a non-zero probability that a given parameter of $\mathcal{N}_{2}$ will end up within the tiny $\epsilon$-window centered at a corresponding parameter value in $\mathcal{N}_{1}$; because $\epsilon<R-\gamma$ we know that the $\epsilon$-window won’t stretch outside the distribution support. If we randomly sample a very large number of units to construct the hidden layer of $\mathcal{N}_{2}$ the probability of finding a subnetwork of $\mathcal{N}_{2}$ corresponding to $\mathcal{N}_{1}$ can be made arbitrarily close to $1$. If the activation function is bias-learning we can use biases to pick out this subnetwork by setting them appropriately smaller than the threshold given in Definition 3. We stress that this proof requires exceedingly massive hidden layer widths.

This section studies a discrete-time RNN given by

where $r_{t}\in\mathbb{R}^{m}$ for all $0\leq t\leq T$ for some $T\in\mathbb{N}$, $\alpha$ and $\beta$ control the time scale of the dynamics, $W\in\mathbb{R}^{m\times m}$, $C\in\mathbb{R}^{l\times m}$, and $B$ and $b$ are as in the previous section. The parameters are now $\theta=\{W,B,C,b\}$. The time-dependent input $x_{t}$ belongs to a compact subset $U_{x}\subset\mathbb{R}^{d}$ for all $t$. Note that when $\alpha=0$, $\beta=1$ one gets the standard vanilla RNN formulation; alternatively, $\alpha$ and $\beta$ can be set to approximate continuous-time dynamics using Euler’s method.

We will approximate the following class of dynamical systems by learning only biases:

where $t$ and $x_{t}$ are as defined for the RNN, $F:U_{z}\times U_{x}\to\mathbb{R}^{s}$ is continuous, and $Q\in\mathbb{R}^{l\times s}$. Because we build from the classic universal approximation results, we must be working with functions operating on compact sets. To guarantee that this will be the case we must make several more assumptions about the dynamical system. First, $U_{z}\subset\mathbb{R}^{s}$ is assumed to be a compact invariant set of the dynamical system: if the system is in $U_{z}$ it remains there for all $t$ for all inputs in $U_{x}$. Second, we assume that the dynamical system is well-defined on a slightly larger compact set $\tilde{U}_{z}\times U_{x}$, where $\tilde{U}_{z}=\{z_{0}+c_{0}:z_{0}\in U_{z},\>||c_{0}||<c\}$ for some $c>0$, with $\tilde{U}_{z}\supset U_{z}$. Letting $p_{R}$ be defined as in Section 2.1, the main result of this section is:

This proof is weaker than the proof given in the previous section in that it shows point-wise convergence, rather than $L^{1}$, convergence. We conjecture that this can be extended straightforwardly but we leave this result to future work.

Proof Intuition: The complete proof is in the Appendix; here we provide intuition. As in Theorem 1 the proof proceeds in two steps. First, the dynamical system is approximated by an RNN, $\mathcal{R}_{1}$, using universal approximation theory for RNNs (see e.g., Schäfer and Zimmermann (2006)). $\mathcal{R}_{1}$ is then approximated by a much wider, random RNN, $\mathcal{R}_{2}$, with parameters sampled from $p_{R}$. Analogous to Theorem 1, we show that one can find a sub-network of hidden units in $\mathcal{R}_{2}$ that approximates $\mathcal{R}_{1}$ for very large hidden widths of $\mathcal{R}_{2}$.

We first validated the theory by checking whether a single-hidden-layer, bias-learning FFN could perform digit recognition on the MNIST dataset Deng (2012) increasingly well as its hidden layer was widened. Initial weights were sampled from a uniform distribution on $[-0.1,0.1]$ and then frozen. As expected, the network learned the task and validation accuracy increased with layer width. The largest gains in performance occurred before 5000 units (Fig. 1A).

Intuitively, bias learning should allow a single random set of weights to be used to learn multiple tasks by simply optimizing task-specific bias vectors. We confirmed this intuition by training a single-hidden-layer FFN with 32000 hidden units on 7 different tasks: MNIST Deng (2012), KMNIST Clanuwat et al. (2018), Fashion MNIST Xiao et al. (2017), Ethiopic-MNIST, Vai-MNIST, and Osmanya-MNIST from Afro-MNIST Wu et al. (2020), and Kannada-MNIST Prabhu (2019). All tasks involved classifying 28$\times$28 grayscale images into 10 classes. The random weights were fixed across tasks while different biases were learned. We compared bias learning against a fully-trained neural network with the same size and architecture (Fig. 1B). We found that the bias-only network achieved similar performance to the fully-trained network on most tasks (only significantly worse on KMNIST). A crucial difference here is that the networks had matched size and architecture, so that the number of trainable parameters in the bias-only network (32 000 parameters) was several orders of magnitude smaller than in the fully-trained case (25 440 010 parameters). Notably, a different set of biases was learned for each task. We conclude that bias-only learning in FFNs is a viable avenue to perform multi-tasking with randomly initialized and fixed weights, but that it requires a much wider hidden layer than fully trained networks.

Next, we investigated the neural mechanism underlying bias learning of multiple tasks. We investigated the task-specific functional organization of the hidden units by estimating single-unit Task Variance (TV) Yang et al. (2019), defined as the variance of a hidden unit activation across the test set for each task. The TV provides a measure of the extent that a given hidden unit contributes to the given task. A unit with high TV in a particular task indicates that its responses vary across stimuli, suggesting that the unit is recruited for solving that task. A unit with high TV in one task and low TV for all other tasks is specialized to one particular task. We clustered the hidden units TV using K-means clustering ($K$ chosen by cross-validation) on the vectors of TVs for each unit and found that distinct functional clusters of hidden units emerged (Fig. 1C). Most units reflected strong task specialization, i.e., they were only used for specific tasks (ex: cluster 3 for KMNIST and cluster 10 for Osmanya). Others were used for many or all tasks (ex: clusters 1 and 8), although a smaller fraction of clusters exhibited such non-selective activation patterns. Overall, we conclude that multi-task bias-learning leads to the emergence of task-specific functional organization.

We finally explored the relationship between the bias of a hidden unit and its task variance (TV). If the neural networks are using biases to shut-off units, analogous to the intuition in our theory work (Section 2.1), then the network units that do not actively participate in a task should be more quiet due to a low bias value learned during training on that particular task. In other words, this intuition would suggest that units should exhibit a correlation between bias and TV, especially in task-specific clusters. In our experiments, all clusters did exhibit the statistical trend of a positive correlation between bias and TV, although to a varying degree across clusters (see numbers at the bottom of Fig. 1C).

As our theory shows that bias-learning networks can universally approximate simply by turning units off, we wished to test whether bias learning performs similarly to learning masks, and to what extent solutions learned by these approaches are different from each other. We compared training mask to bias learning on networks with the same random input/output weight matrices. For mask-training, we approximated binary masks with ‘soft’ sigmoid masks by optimizing the sigmoid parameters. The sigmoid was steepened over the course of training to approximately learn a mask, and at test time the slope of the sigmoid was made steep enough to effectively binarize the learned soft mask. We compared masks learned in this fashion with learned biases on single-hidden-layer ReLU networks with $10,000$ hidden units. We observed a trend of bias-training slightly improving upon mask-training (Fig.2A), an expected increase given the added flexibility of biases over masks. Further research is needed to determine if this trend is reliable across datasets and different network parametrizations, and whether there might be scenarios where one style of learning works better or worse.

Next, we compared the solutions found via bias and mask learning. We calculated the variance of each hidden unit across $10,000$ MNIST images, in both the bias and mask-trained paradigms, as a measure of the hidden-layer representation of MNIST. The histogram over hidden units of these variances is plotted for trained masks/biases (black/orange) for the same network weights (Fig.2B). The mask-trained networks found slightly sparser solutions (more $0$s, far left histogram bin) and performed the task with lower unit variance values (see middle/right of histogram) compared to the bias-trained networks. We also plotted the hidden unit variances for units in mask and bias-trained networks and computed their correlation (Fig.2C). The scatter plot and correlation were calculated after removing unit variances that were zero in the mask-trained net and sufficiently close to zero (with variance values less than the mean variance divided by 100) in the bias-trained network. Despite the differences observed in the histogram, the correlation coefficient was $0.461\pm 0.022$ (mean $\pm$ SD across $n=4$ networks), suggesting that there is some overlap in the way that these two methods solve MNIST.