Multilayer Perceptron Network Discriminates Larval Zebrafish Genotype using Behaviour

Random initialisation of a neural network has been shown to lead to slower convergence and toward poorer local minima [4]. Xavier initialisation is a popular arrow in the quiver of deep neural network (NN) researchers as it decreases the probability of encountering the vanishing/exploding gradients problem during training [2]. However, this method is based on the assumption that are linear [9]. This assumption is invalid for the ReLU activation function used in our proposed architecture. He et al. proposed a new initialisation strategy for ReLU-activated networks [6]. He initialisation sets the network weights randomly from a normal distribution, as in Xavier initialisation, but with a variance of:

$$v^{2}=2/N$$

(1)

in order to reduce the probability of the dying neuron problem.

Batch size is an important contributor to the generalisation ability of a network. When employing stochastic gradient descent as a learning strategy, it is necessary to ensure that the batch size is not too large so as to not reduce the gradient noise [8, 5]. Fluctuation in gradient is sometimes required to escape a poor local minima, thus reduction of this ability can be detrimental. A batch size of $10$ proved helpful for our network to reach good minima in an adequate amount of epochs.

A significant contribution to the field in terms of improving the generalisation of a network is Dropout [13]. The proposed idea is to randomly drop perceptrons and their connections from the network during training. Although simple it is extremely effective and has seen massive adoption to network architectures [10]. By randomly inhibiting the network it prevents units from co-adapting too much. Metaphorically speaking, it prevents the network from memorising the answers to the test since it is easier than studying the concepts [17]. We found that inciting a dropout ($p=0.5$) after the first and second layers of our network was ideal for achieving greater generalisation.

Understanding the flow of information through NNs is a challenging problem that has recently gained increased attention. Being able to analyse the impact of input features on outputs allow further analysis and selection of features for obtaining better results [1]. A simple manner of evaluating the importance of features given by a network can be by analysing the input gradients, that is, the gradients between the input layer and the first hidden layer. But this has a clear limitation of only representing a small portion of the network, akin to looking at it through a peep hole.

We make use of a method that satisfies two fundamental axoims of attribution - Sensitivity and Implementation Invariance. In simpler words, a lack of sensitivity causes gradients to focus on irrelevant features and networks that produce the same outputs, given the same input data, should always have the same attributions. Sundararajan et al. show that most methods do not satisfy these axioms and proposed a new method [14]. Integrated gradients postulates an additional axiom called completeness, stating that, for an input $x$, the attributions should add up to the difference between the output of a network and a baseline. It is suggested that the baseline be chosen such that its prediction is near zero. The proposed technique can be applied to a range of network architectures, including ours.

We designed a lightweight neural network for classifying wildtype and mutant genotypes. An important design choice was for the network to be able to be trained using a portable laptop, in our case, an M1 Macbook Pro. This was accomplished by only having a hidden depth of three layers that are fully connected. A sufficiently wide neural network with just a single hidden layer can approximate any (reasonable) function given sufficient training data, although increasing the hypothesis space requires an exponential increase in width [3]. Our hypothesis space is sufficiently small where three layers is adequate to approximate our ideal function.

The output of the final layer in the NN is passes through a sigmoid function to produce a probability distribution. A logistic loss (binary cross entropy) with one-hot encoded labels is used to calculate the error between the predicted distribution and the true distribution. The loss is backpropagated via stochastic gradient descent (SGD) [12] where, paired with the non-linear activation function ReLU, we are able to reliably find the global minima of the training loss [18]. Adding momentum to SGD has also been shown to be helpful in traversing sub-optimal local minima [15]. A learning rate of $5e^{-4}$ was chosen in combination with a momentum of $0.99$ and proved to be near optimal for our network training.

Zebrafish larvae with wildtype and mutant dnajc6 genotypes were generated by heterozygous incross of two parent pairs. Two families were generated on subsequent days. Larvae were placed in 24-well plates in 1 mL of E3 embryo medium [16] at 4 days old, before being subjected to behavioural testing at 5 days old. Video footage was collected of larvae swimming in the 24-well plates for 15 minutes using the DanioVision (Noldus). A sudden light to dark transition was used to provoke a stress response in the larvae once 5 minutes had elapsed, as we believed that this stimulation would emphasise any genotypic effects on behaviour (specifically locomotor deficits). Each tray of larvae was subjected to 4 separate behaviour trials throughout the course of the day to augment the data available for the network. Fish were genotyped after behaviour testing.

Tracking of the zebrafish and calculation of behavioural variables was done using the EthoVisionXT software (RRID: SCR_000441). The “centrepoint” (trunk) of each larva was tracked by the software from live 2D video footage at 15 frames/second. Table 1 shows the statistics calculated for each 1 minute time bin. Note that there is no correspondence across one minute windows.

Pre-processing of the raw data was performed in R. Behavioural data from both families (spawned and tested on different days) was joined to the fish metadata (genotype, tray number, position in tray, trial number). Two fish were omitted, one due to a deformity that prevented movement and the other due to issues with detection of the fish by the behaviour tracking software.

The absolute z-score (distance from the mean in standard deviations) was calculated for each data point based on the mean value of that statistic within a genotype group. This was done to prevent the removal of attributes that were due to differences between genotypes. Any values with a z-score $>$4 (4 standard deviations from the mean) were removed. This was done separately for each family before merging all data.

The values of each behavioural statistic were averaged for each fish across three time bins from the first light period (bins 3, 4, and 5), and the dark period (bins 6, 7, and 8). These summarised values were used as behavioural features (eg. distance travelled in the light, distance travelled in the dark). A total of 41 features were used here. For the dark period, all statistics in Table 1 were used and for the light period, all statistics other than the distance to middle (variance), meander (all), and heading (all).

Prior to training, the data was standardised to have a mean of zero and a standard deviation of one to encourage faster convergence. A random 85/15 train/validation split was done to set aside data for testing the generalisation ability of the models. Each model was trained on the validation data for 500 epochs and the model was evaluated after each epoch. The model with the highest validation accuracy during this time was chosen as the most fit model and was used for calculating integrated gradients.

Ten experiments were run with differing numbers of perceptrons in the hidden layers to allow comparison in terms of overfitting and selection of the best model. We performed each experiment on the same $85/15$ train/validation split so that results were easier to compare. Each experiment trained ten models of the same architecture and were evaluated on the validation set. Early stopping was used to select the best version of each model.

The average train accuracy and validation accuracy of each experiment for analysis, shown in Table 2. The architecture with 70 perceptrons in the hidden layers produced the best train accuracy. The 80-perceptron architecture had the best accuracy on the validation set and came a close second on the train set. We found that architectures with more than 30 perceptrons in their hidden layers were more prone to loss divergence but still achieved high train and validation accuracies. In comparison, the architectures with less than 30 perceptrons had a lower frequency of divergence but could not achieve the same accuracy. We theorise that this is due to the networks not having a sufficient number of parameters to fit the data well. In comparison, the losses of architectures with more perceptrons were more likely to diverge, but were also more likely to achieve better generalisation.

The average integrated gradients from the ten models were calculated for each experiment. Each behavioural feature was given an attribution value representative of the impact it had on the classification outcome across the entire dataset. Positive attributions indicate that the feature increases the probability of the genotype being chosen, and vice versa. The average integrated gradients for all models with hidden layers of 80 perceptrons are visualised in Figure 3.

The mean distance travelled, mean velocity, velocity variance, time in middle variance, and total time moving have very small attribution scores indicating that the model is generally unsure how to make use of these features. Whereas we can see that the other features have larger attribution scores, indicating that the model depends on them for the label decision.

Overall, the behavioural features from the dark period (after a stressful stimulus) show higher attribution scores than those from the light period. This may be due to the overall increased movement seen in larvae following the light to dark transition (Figure 2) providing more information with regards to their movement.

The velocity and time moving in the light have very small attribution scores indicating that the model is generally unsure how to make use of these features. This is also seen in the variance for the time spend in the middle in the dark. Many other features have larger attribution scores, indicating that the model depends on them for the label decision.