This paper was converted on www.awesomepapers.org from LaTeX by an anonymous user.
Want to know more? Visit the Converter page.

Tuna Nutriment Tracking using Trajectory Mapping in Application to Aquaculture Fish Tank

Hilmil Pradana Graduate School of Life Science and Systems Engineering
Kyushu Institute of Technology
Fukuoka, Japan
Email: [email protected]    Keiichi Horio Graduate School of Life Science and Systems Engineering
Kyushu Institute of Technology
Fukuoka, Japan
Email: [email protected]
Abstract

The cost of fish feeding is usually around 40 percent of total production cost. Estimating a state of fishes in a tank and adjusting an amount of nutriments play an important role to manage cost of fish feeding system. Our approach is based on tracking nutriments on videos collected from an active aquaculture fish farm. Tracking approach is applied to acknowledge movement of nutriment to understand more about the fish behavior. Recently, there has been increasing number of researchers focused on developing tracking algorithms to generate more accurate and faster determination of object. Unfortunately, recent studies have shown that efficient and robust tracking of multiple objects with complex relations remain unsolved. Hence, focusing to develop tracking algorithm in aquaculture is more challenging because tracked object has a lot of aquatic variant creatures. By following aforementioned problem, we develop tuna nutriment tracking based on the classical minimum cost problem which consistently performs well in real environment datasets. In evaluation, the proposed method achieved 21.32 pixels and 3.08 pixels for average error distance and standard deviation, respectively. Quantitative evaluation based on the data generated by human annotators shows that the proposed method is valuable for aquaculture fish farm and can be widely applied to real environment datasets.

Index Terms:
Productivity, Fish Feeding, Nutriment, Tracking Algorithm, Real Environment Datasets

I Introduction

Aquaculture is the one of farming type in which aquatic creatures require acceptable environment for living habitat and availability nutriment to increase productivity and sustain healthy growth [1, 2, 3, 4]. Within current requiring acceptable habitat, water quality is also a vital component to enlarge fish fertility rate [3, 4, 5, 6]. Water quality can be obtained by cleaned often and give optimal amount of nutriment. Increasing number of nutriment can affect a lot of foods wasted in the water and quality of water occurs highly polluted. On the other hand, reducing feeding will lead starvation and drop fish quality. So that, management of nutriment delivered is vital component to balance productivity rate [7, 8].

The cost of fish feeding is usually around 40 percent of total production cost [9, 10, 11]. Estimating a state of fishes in a tank and adjusting an amount of nutriments play an important role to manage cost of fish feeding system. It is applied to control the amount of nutriment and realizes the fish behavior in tank. Lately, application to monitor fish behavior has been adopted by a telemetry-based approach [12, 13] and a computer vision(CV)-based approach [14, 15, 16, 17, 18, 19, 20].

A telemetry-based approach is a technique attaching an external transmitter by mounting, or surgical implantation in the peritoneal cavity [12]. Attaching a transmitter in each fish will spend higher cost and its transmitter can only set in large fish. When their fishes had been farmed, attachment will always be given to new fishes. On the other hand, CV-based approach studies are not required complexity analysis such as ripple activity and tracking analysis in which, small number of fishes and small tanks with special environment assist creating result. Tracking approach is applied to acknowledge movement of nutriment to understand more about the fish behavior. Fish behaviors can be obtained by combination between tracking analysis and ripple activity. Then, these fish behaviors can be a decision to start and stop fish feeding machine by understanding of ripple activity after giving several nutriments. By explaining of fish behavior, tracking nutriment is important and it is required to analyze the complexity data in real environment.

Recently, there has been increasing number of researchers focused on developing tracking algorithms to generate more accurate and faster determination of object. Tracking can be represented as a graph problem which can solved by a frame-by-frame [21, 22, 23, 24] or track-by-track [25, 26]. Interpretation of tracking problems with data association mostly uses a graph, where each detection is called as vertex, and each edge is pointing any possible link among them out as object tracked. Data association can be declared as minimum cost problem [27, 28, 29, 30] with learning cost problem [31] or motion pattern maps [32]. Alternative formulations to solve optimization problems is minimum clique problem [33] and lifted multicut problem [34] where its formulations follow body pose layout to obtain estimated model. Recently, efficient and robust tracking of multiple objects with complex relations remain unsolved. Hence, focusing to develop tracking algorithm in aquaculture is more challenging because tracked object has a lot of aquatic variant creatures. By summarizing aforementioned problems, we proposed tuna nutriment tracking based on the classical minimum cost problem [28, 29, 30] where each detection calculates minimum distance among them and creates a trajectory to be tracked line. By collaborating with an active aquaculture fish farm, we develop tuna nutriment tracking using trajectory mapping. A video camera is placed above the boat with a highly disturbance of ocean wave and many dense nutriments. The camera captures between ocean surface and fish feeding machine. After that, videos transfer to a computer for further analysis the behavior of fish.

The aim of this research is tracking approach to acknowledge the behavior of tuna. For next, it can be useful to improve the production profit in fish farms by controlling the amount of nutriment in optimal rate.

To summarize, we make the following contributions:

  • •

    We propose tuna nutriment tracking based on trajectory mapping which can perform well as well as human annotator results.

  • •

    We propose a new novel small nutriment tracking method with collecting information of leading line into ripple.

  • •

    We show significantly improvement result of trajectory mapping in real environment datasets.

II The Proposed Method - Trajectory Mapping

Refer to caption
Figure 1: Flowchart of the proposed trajectory mapping. The input video is received and applied image stabilization as data normalization. (a) Creating model for tuna nutriment detection using YOLOv3 [36] and obtaining bounding box for each tuna nutriment prediction in all frames of a video. (b) Tuna nutriment prediction as input tracking approach to be initialization for nf=1n_{f}=1 detection. After that, we obtain upper and lower limit trajectory 𝑻lu\mbox{\boldmath$T$}^{u}_{l}, 𝑻fl\mbox{\boldmath$T$}^{l}_{f} and the maximum height of all tuna nutriment predictions δ1\delta_{1}. Next, we find the value of Cf2C_{f_{2}} using the shortest distance between Cf1C_{f_{1}} and all tuna nutriment predictions in nfn_{f} appearing in inside area between 𝑻fu​(1)\mbox{\boldmath$T$}^{u}_{f}(1) and 𝑻fu​(1)\mbox{\boldmath$T$}^{u}_{f}(1). Its process is repeatedly until nfn_{f}. Next, we obtain final trajectory as tracking result.
Refer to caption
Figure 2: Algorithm of the proposed trajectory mapping. Iterative method are applied to improve trajectory 𝑻f\mbox{\boldmath$T$}_{f} by collecting each centroid of nutriment in every frame.

Our formulation is based on the classical minimum cost problem where each detection calculates minimum distance among them and creates a trajectory to be tracked line. In order to provide some background and formally introduce our approach, we start by providing flowchart and algorithm of tuna nutriment tracking. We then explain how the proposed method works to real environment. The proposed trajectory mapping contains a data normalization process, tuna nutriment detection and tuna nutriment tracking. The system flowchart of the proposed method is shown in Fig. 1, and the algorithm of the proposed trajectory mapping is represented in Fig. 2 where DD and Tf\textbf{T}_{f} are input video and trajectory of time-ordered tuna nutriment, respectively.

II-A Data Normalization

For data normalization, image stabilization is applied to reduce a hand-held camera and ocean waves. Image stabilization is created by transformation from previous to current frame using optical flow for all frames. [35] accumulates rigid transformation χ\chi to obtain linked between frame LL. New rigid transformation χϕ\chi_{\phi} in frame ϕ\phi can be written as:

χϕ=χϕ−1+(1γ​∑τ=ϕ−γϕ+γLτ)​–​Lϕ−1,D^=χϕ​{D},\begin{split}\chi_{\phi}=&\chi_{\phi-1}+(\dfrac{1}{\gamma}\sum_{\tau=\phi-\gamma}^{\phi+\gamma}L_{\tau})–L_{\phi-1},\\ \hat{D}=&\chi_{\phi}\{D\},\end{split} (1)

where D^\hat{D} is output video after applied image stabilization and γ\gamma is smoothing radius where the radius is number of frames used for smoothing and defined by 30.

II-B Tuna Nutriment Detection

The idea of tuna nutriment detection is to produce boundary box in each nutriment associated in tracking method. In implementation of tuna nutriment detection, YOLOv3 [36] accumulates bounding box of tuna nutriment prediction B=(x^,y^,w^,h^)B=(\hat{x},\hat{y},\hat{w},\hat{h}) by training model with bounding box P=(pxP=(p_{x}, pyp_{y}, pwp_{w}, ph)p_{h}) of ground truth data where pxp_{x}, pyp_{y}, pwp_{w}, and php_{h} are centroid xx, centroid yy, width, and height of bounding box in ground truth data, respectively. ςx\varsigma_{x} and ςy\varsigma_{y} represent the absolute location of the top-left corner of the current grid cell. ww and hh are the absolute width and height to the whole image. Bounding box of tuna nutriment prediction BB can defined as:

x^=δ​(px)+ςxy^=δ​(py)+ςyw^=epw∗wh^=eph∗h\begin{split}\hat{x}&=\delta(p_{x})+\varsigma_{x}\\ \hat{y}&=\delta(p_{y})+\varsigma_{y}\\ \hat{w}&=e^{p_{w}}*w\\ \hat{h}&=e^{p_{h}}*h\end{split} (2)

where δ\delta is model followed by [36].

II-C Tuna Nutriment Tracking

In order to represent tracking of tuna nutriment, introducing how to collect set of tuna nutriment prediction corresponding to time-ordered path in the graph is important. We are given 𝝈𝒄​(n)={C1,C2,C3,…,Cn}\mbox{\boldmath$\sigma^{c}$}(n)=\{C_{1},C_{2},C_{3},\ldots,C_{n}\} as input centroid of tuna nutriment predictions where nn is the total number of nutriment for all frames of video D^\hat{D}. Each tuna nutriment prediction is represented by Cn={x^nc,y^nc}C_{n}=\{\hat{x}^{c}_{n},\hat{y}^{c}_{n}\}. Definition of a trajectory is denoted as centroid of time-ordered tuna nutriment predictions 𝑻f​(nf)={Cf1,Cf2,Cf3,…,Cfnf}\mbox{\boldmath$T$}_{f}(n_{f})=\{C_{f_{1}},C_{f_{2}},C_{f_{3}},\ldots,C_{f_{n_{f}}}\} where nfn_{f} is the number of detections formed by trajectory ff. So that, ϱ={n1,n2,n3,…,nnf}\varrho=\{n_{1},n_{2},n_{3},\ldots,n_{n_{f}}\} can be denoted as the total of number of nutriments appearing in every time-ordered trajectory 𝑻f​(nf)\mbox{\boldmath$T$}_{f}(n_{f}).

II-C1 Problem Statement

Refer to caption
Figure 3: Visualization of trajectory of tuna nutriment predictions 𝑻f={Cf1,Cf2,Cf3,…,Cfnf}\mbox{\boldmath$T$}_{f}=\{C_{f_{1}},C_{f_{2}},C_{f_{3}},\ldots,C_{f_{n_{f}}}\} in which every tuna nutriment predictions is connected by node {(f1,f2),(f2,f3),…,(fn−1f,fnf)}\{(f_{1},f_{2}),(f_{2},f_{3}),\ldots,(f_{n-1_{f}},f_{n_{f}})\}.

The problem can be represented with an undirected graph G=(V,E)G=(V,E), where V:={1,…,n},E⊂V2V:=\{1,...,n\},E\subset V^{2}, and each node f∈Vf\in V denotes a unique detection Cf∈σcC_{f}\in\sigma^{c}. The task of dividing the set of tuna nutriment predictions into trajectories can be observed as grouping nodes in graph. Fig. 3 shows that each trajectory 𝑻f​(nf)={Cf1,Cf2,Cf3,…,Cfnf}\mbox{\boldmath$T$}_{f}(n_{f})=\{C_{f_{1}},C_{f_{2}},C_{f_{3}},\ldots,C_{f_{n_{f}}}\} in the scene can be mapped into a group of nodes {(f1,f2),(f2,f3),…,(fn−1f,fnf)}\{(f_{1},f_{2}),(f_{2},f_{3}),\ldots,(f_{n-1_{f}},f_{n_{f}})\}. To produce each Cf[1,nf]C_{f_{[1,n_{f}]}}, trajectory mapping is applied in next section.

In two-dimensional trajectory, the component of trajectory is divided by horizontal and vertical direction. In vertical direction, acceleration is constant and has quadratic function. Trajectory mapping applies the idea of acceleration and chooses quadratic function as basis.

To produce quadratic function yc=a3c​x2+a2c​x+a1cy^{c}=a^{c}_{3}x^{2}+a^{c}_{2}x+a^{c}_{1} as a result of trajectory 𝑻f\mbox{\boldmath$T$}_{f}, we apply polynomial fitting [37] defined by calculation of x^nfc\hat{x}^{c}_{n_{f}} to form Vandermonde matrix 𝑽V with 33 columns as results of 𝒂c\mbox{\boldmath$a$}^{c}.

[1x^1c(x^1c)21x^2c(x^2c)21⋮⋮1x^nfc(x^nfc)2]​[a1ca2ca3c]=[y^1cy^2c⋮y^nfc]\begin{split}\begin{bmatrix}1&\hat{x}^{c}_{1}&(\hat{x}^{c}_{1})^{2}\\ 1&\hat{x}^{c}_{2}&(\hat{x}^{c}_{2})^{2}\\ 1&\vdots&\vdots\\ 1&\hat{x}^{c}_{n_{f}}&(\hat{x}^{c}_{n_{f}})^{2}\end{bmatrix}\begin{bmatrix}a^{c}_{1}\\ a^{c}_{2}\\ a^{c}_{3}\end{bmatrix}=\begin{bmatrix}\hat{y}^{c}_{1}\\ \hat{y}^{c}_{2}\\ \vdots\\ \hat{y}^{c}_{n_{f}}\end{bmatrix}\end{split} (3)

(3) can be inverted directly. To yield the solution vector 𝒂c\mbox{\boldmath$a$}^{c}, it can be defined as:

𝒂c=(𝑿T​𝑿)−1​𝑿T​𝒀,\begin{split}\mbox{\boldmath$a$}^{c}=(\mbox{\boldmath$X$}^{T}\mbox{\boldmath$X$})^{-1}\mbox{\boldmath$X$}^{T}\mbox{\boldmath$Y$},\end{split} (4)

II-C2 Tuna Nutriment Predictions CfnfC_{f_{n_{f}}} where nf=1n_{f}=1 as Initialization Point Detection

Tuna nutriment predictions Cf1C_{f_{1}} are obtained from every tuna nutriment prediction CnC_{n} in around cutting area of ww. To define cutting area, we use centroid x^f1\hat{x}_{f_{1}} as component of Cf1C_{f_{1}} by thresholding in ww which is defined as:

w∗γ≤x^f1≤w,\begin{split}w*\gamma\leq\hat{x}_{f_{1}}\leq w,\end{split} (5)

where γ\gamma is an input parameter and empirically defined as 0.90.9.

Direction of nutriment is calculated by leading nutriment to ripple area around sea levels. We are given a pair set of ripple area detection 𝝈𝒓​(nf)={(R11,R12),(R21,R22),(R31,R32),…,(Rnf​1,Rnf​2)}\mbox{\boldmath$\sigma^{r}$}(n_{f})=\{(R_{11},R_{12}),(R_{21},R_{22}),(R_{31},R_{32}),\ldots,(R_{n_{f}1},R_{n_{f}2})\} as time-ordered ripple predictions in number of detections nfn_{f}. Each ripple prediction is represented by Rnf​1,2={x^nfr,y^nfr,w^nfr,h^nfr}R_{n_{f}1,2}=\{\hat{x}^{r}_{n_{f}},\hat{y}^{r}_{n_{f}},\hat{w}^{r}_{n_{f}},\hat{h}^{r}_{n_{f}}\}. We divide component of ripple prediction to be an area of top-left αnf=(x^nfα,y^nfα)\alpha_{n_{f}}=(\hat{x}^{\alpha}_{n_{f}},\hat{y}^{\alpha}_{n_{f}}) and bottom right θnf=(x^nfθ,y^nfθ)\theta_{n_{f}}=(\hat{x}^{\theta}_{n_{f}},\hat{y}^{\theta}_{n_{f}}) of ripple detection by following:

x^nfα=x^nfr−w^nfr2,y^nfα=y^nfr−h^nfr2,x^nfθ=x^nfr+w^nfr2,y^nfθ=y^nfr+h^nfr2,\begin{split}\hat{x}^{\alpha}_{n_{f}}=&\hat{x}^{r}_{n_{f}}-\dfrac{\hat{w}^{r}_{n_{f}}}{2},\\ \hat{y}^{\alpha}_{n_{f}}=&\hat{y}^{r}_{n_{f}}-\dfrac{\hat{h}^{r}_{n_{f}}}{2},\\ \hat{x}^{\theta}_{n_{f}}=&\hat{x}^{r}_{n_{f}}+\dfrac{\hat{w}^{r}_{n_{f}}}{2},\\ \hat{y}^{\theta}_{n_{f}}=&\hat{y}^{r}_{n_{f}}+\dfrac{\hat{h}^{r}_{n_{f}}}{2},\end{split} (6)

To obtain more feature, we need to know possibly coverage area for possibly nutriment appearing in next frame by creating upper and lower limit trajectory 𝑻fu\mbox{\boldmath$T$}^{u}_{f} and 𝑻fl\mbox{\boldmath$T$}^{l}_{f}, respectively. Upper and lower limit trajectory 𝑻fu​(nf)\mbox{\boldmath$T$}^{u}_{f}(n_{f}) and 𝑻fl​(nf)\mbox{\boldmath$T$}^{l}_{f}(n_{f}) formed by trajectory ff are initialized by following:

𝑻fu​(nf)={Cf1,δnf,αnf},𝑻fl​(nf)={Cf1,θnf},\begin{split}\mbox{\boldmath$T$}^{u}_{f}(n_{f})=&\{C_{f_{1}},\delta_{n_{f}},\alpha_{n_{f}}\},\\ \mbox{\boldmath$T$}^{l}_{f}(n_{f})=&\{C_{f_{1}},\theta_{n_{f}}\},\\ \end{split} (7)

where δ\delta is the maximum height of all nutriment detections in nfn_{f}. (7) can be simplify by substituting nf=1n_{f}=1 to be:

𝑻fu​(1)={Cf1,δ1,α1},𝑻fl​(1)={Cf1,θ1},\begin{split}\mbox{\boldmath$T$}^{u}_{f}(1)=&\{C_{f_{1}},\delta_{1},\alpha_{1}\},\\ \mbox{\boldmath$T$}^{l}_{f}(1)=&\{C_{f_{1}},\theta_{1}\},\\ \end{split} (8)

where δ1=(x^1α+w2,min1≤z≤n1⁡y^zc)\delta_{1}=(\dfrac{\hat{x}^{\alpha}_{1}+w}{2},\min\limits_{1\leq z\leq n_{1}}\hat{y}^{c}_{z}).

II-C3 Tuna Nutriment Predictions CfnfC_{f_{n_{f}}} where nf=2n_{f}=2

To be a candidate of Cf2C_{f_{2}}, we use all tuna nutriment predictions nn appearing in the inside of area between yu=a3u​x2+a2u​x+a1uy^{u}=a^{u}_{3}x^{2}+a^{u}_{2}x+a^{u}_{1} and yl=a3l​x2+a2l​x+a1ly^{l}=a^{l}_{3}x^{2}+a^{l}_{2}x+a^{l}_{1}. Vector 𝒂u\mbox{\boldmath$a$}^{u} and 𝒂l\mbox{\boldmath$a$}^{l} are produced by calculating 𝑻fu​(nf−1)\mbox{\boldmath$T$}^{u}_{f}({n_{f}}-1) and 𝑻fl​(nf−1)\mbox{\boldmath$T$}^{l}_{f}({n_{f}}-1) with Vandermonde matrix shown in (3) and (4), respectively. Given 𝝈𝜿​(μ)={κ1nf,κ2nf,κ3nf,…,κμnf}\mbox{\boldmath$\sigma^{\kappa}$}(\mu)=\{\kappa_{1_{n_{f}}},\kappa_{2_{n_{f}}},\kappa_{3_{n_{f}}},\ldots,\kappa_{\mu_{n_{f}}}\} is a set of candidate CfnfC_{f_{n_{f}}}. CfnfC_{f_{n_{f}}} is defined by the nutriment predictions which have shortest distance denoted by:

Cfnf\displaystyle C_{f_{n_{f}}} =\displaystyle{}={} arg​minμ(𝒁(fnf−1)−𝝈𝜿(μ))T\displaystyle\operatorname*{arg\,min}_{\mu}(\mbox{\boldmath$Z$}(f_{n_{f}-1})-\mbox{\boldmath$\sigma^{\kappa}$}(\mu))^{T} (9)
(𝒁​(fnf−1)−𝝈𝜿​(μ))\displaystyle(\mbox{\boldmath$Z$}(f_{n_{f}-1})-\mbox{\boldmath$\sigma^{\kappa}$}(\mu))

where 𝒁​(fnf−1)={Cfnf−11,Cfnf−12,Cfnf−13,…,Cfnf−1μ}\mbox{\boldmath$Z$}(f_{n_{f}-1})=\{C^{1}_{f_{n_{f}-1}},C^{2}_{f_{n_{f}-1}},C^{3}_{f_{n_{f}-1}},\ldots,C^{\mu}_{f_{n_{f}-1}}\}. (9) can be simplify to be;

Cf2=arg​minμ(𝒁(1)−𝝈𝜿(μ))T(𝒁(1)−𝝈𝜿(μ)),\begin{split}C_{f_{2}}=\operatorname*{arg\,min}_{\mu}(\mbox{\boldmath$Z$}(1)-\mbox{\boldmath$\sigma^{\kappa}$}(\mu))^{T}(\mbox{\boldmath$Z$}(1)-\mbox{\boldmath$\sigma^{\kappa}$}(\mu)),\end{split} (10)

Updating upper trajectory 𝑻fu​(nf)\mbox{\boldmath$T$}^{u}_{f}(n_{f}) can be defined as:

𝑻fu​(nf)={Cf1,Cf2,∑z=1nfδznf,αnf},\begin{split}\mbox{\boldmath$T$}^{u}_{f}(n_{f})=&\{C_{f_{1}},C_{f_{2}},\dfrac{\sum_{z=1}^{n_{f}}\delta_{z}}{n_{f}},\alpha_{n_{f}}\},\end{split} (11)

II-C4 Tuna Nutriment Predictions CfnfC_{f_{n_{f}}} where nf=3n_{f}=3

Minimum requirement for trajectory of quadratic functions must have at least 3 tuna nutriment predictions collected. To produce Cf3C_{f_{3}}, (9) is applied using nf=3n_{f}=3 as parameter. Then, updating upper limit trajectory 𝑻fu​(nf)\mbox{\boldmath$T$}^{u}_{f}(n_{f}) is denoted as follows:

𝑻f​(nf)=𝑻fu​(nf),\begin{split}\mbox{\boldmath$T$}_{f}(n_{f})=&\mbox{\boldmath$T$}^{u}_{f}(n_{f}),\\ \end{split} (12)

II-C5 Tuna Nutriment Predictions CfnfC_{f_{n_{f}}} where nf≥4n_{f}\geq 4

To precise accuracy of trajectory 𝑻f\mbox{\boldmath$T$}_{f}, we refine its trajectory by collecting more tuna nutriment prediction CfnfC_{f_{n_{f}}}. Tuna nutriment prediction CfnfC_{f_{n_{f}}} is calculated using the nearest nutriment detection in area of yu=a3u​x2+a2u​x+a1uy^{u}=a^{u}_{3}x^{2}+a^{u}_{2}x+a^{u}_{1} with tolerance degree from quadratic function between ±30\pm 30 degree.

To handle losing tuna nutriment prediction, we used previously tuna nutriment prediction by calculating the speed of nutriment in next frame.

x^fnfc=3​x^fnf−1c−3​x^fnf−2c+x^fnf−3c,y^fnfc=a3c​(x^fnfc)2+a2c​x^fnfc+a1c,\begin{split}\hat{x}^{c}_{f_{n_{f}}}=&3\hat{x}^{c}_{f_{n_{f}-1}}-3\hat{x}^{c}_{f_{n_{f}-2}}+\hat{x}^{c}_{f_{n_{f}-3}},\\ \hat{y}^{c}_{f_{n_{f}}}=&a^{c}_{3}(\hat{x}^{c}_{f_{n_{f}}})^{2}+a^{c}_{2}\hat{x}^{c}_{f_{n_{f}}}+a^{c}_{1},\end{split} (13)

where a1ca^{c}_{1}, a2ca^{c}_{2}, and a3ca^{c}_{3} are coefficients of quadratic function formed by trajectory 𝑻f\mbox{\boldmath$T$}_{f}

III Experiment

In this section, we first explain the details of our datasets. We then describe evaluation approach to calculate error rate distance and show quantitative evaluation with various of nfn_{f} to discover an optimal value.

III-A Datasets

We report our datasets containing 1 video which has interference of hand-held camera and ocean waves with 419 frames. Each dimension of frame has 1920×10801920\times 1080 pixels. Its video sequences are in MOV format with frame rate 30 frames/second. Range size of nutriment is starting from 9×69\times 6 to 13×3613\times 36 pixels.

III-B Evaluation Approach

Evaluation approach is defined by measuring minimum euclidean distance based on number of nutriment collected with ground truth 𝑻g\mbox{\boldmath$T$}_{g}. Best trajectory T∗T^{*} with minimum error rate distance is defined as:

T∗=arg​minnf(𝑻g−𝑻(nf))T(𝑻g−𝑻(nf)),\begin{split}T^{*}=\operatorname*{arg\,min}_{n_{f}}(\mbox{\boldmath$T$}_{g}-\mbox{\boldmath$T$}(n_{f}))^{T}(\mbox{\boldmath$T$}_{g}-\mbox{\boldmath$T$}(n_{f})),\end{split} (14)

where nf∈[3,9]n_{f}\in[3,9].

III-C Quantitative Evaluation with various of nf=[3,9]n_{f}=[3,9]

Refer to caption
Figure 4: Accuracy detected nutriment and precision trajectory in various nf∈[3,9]n_{f}\in[3,9]. The optimum value of detected nutriment and precision trajectory is nf=6n_{f}=6.
Refer to caption
Figure 5: 95% Confidence interval of error rate distance in various of nf∈[3,9]n_{f}\in[3,9]. The lowest error rate distance of proposed method is nf=6n_{f}=6.
TABLE I: Statistical analysis of various nf∈[3,9]n_{f}\in[3,9] for quantitative evaluation using one-samples T-Test.
nf n Mean (pixels) Std. Dev. (pixels) Std. Error (pixels)
95% Confidence
Interval of the Diff.
Lower
(pixels)
Upper
(pixel)
3 30 185.47 93.81 17.13 150.44 220.5
4 30 78.58 33.64 6.14 66.02 91.14
5 30 25.17 13.97 2.55 19.96 30.39
6 30 21.32 3.08 0.56 20.18 22.48
7 30 35.12 19.34 3.53 27.9 42.35
8 30 32.67 4.19 0.76 31.11 34.24
9 30 44.1 13.32 2.43 39.12 49.07

Quantitative evaluation is computed by performance of detected nutriment and precision trajectory showed in Fig. 4. Number of detected nutriment is defined as percentage of detected nutriment divided by ground truth of nutriments appearing in frame. Meanwhile, precision trajectory is computed by total number of nutriments having trajectory leading to ripple area divided by detected nutriment. Fig. 5 and Table I show the confidence interval and statistical analysis of error rate distance in various of nfn_{f}. The results show that the optimal value of nfn_{f} is 66 in which this parameter produces smallest error rate distance.

IV Result

In this section, we compare proposed method and state-of-the-art benchmark methods on our datasets. After that, we show the figures to explain the advantage of the proposed method and computational time between proposed method and state-of-the-art benchmark methods.

IV-A Evaluation Result

Refer to caption
Figure 6: mAP result during training model using YOLOv3 [36]. After 7k iterations, curve of mAP of model has more stable.

Refer to caption
(a) Trajectory Mapping
Refer to caption
(b) JDE

Refer to caption
(c) YOLOv3+SORT
Refer to caption
(d) Our Detection Model + SORT
Figure 7: Observation of proposed method namely trajectory mapping shown in (a) and benchmark method results shown in (b), (c), and (d). Left and right images for each method represent first node of nutriment and nutriment tracked after several frames, respectively. In trajectory mapping method, red curve is defined as trajectory result of proposed method. Red box in each image represents ground truth of nutriment. We can see that both red curve and red box in trajectory mapping show precisely tracked result and it proves that trajectory mapping creates trajectory very well while benchmark methods perform poor without tracking results of nutriments even SORT is able to detect some nutriments.

Precision of mAP in object detection is computed by performance YOLOv3 [36] to train our datasets with 10k iterations with 416×416416\times 416 pixels for image resizing from 1920×10801920\times 1080 pixels. Fig. 6 displays training result of our datasets using YOLOv3 [36] and reaches 67% of maximum mAP with 10k iterations. We also tested proposed methods and state-of-the-art benchmark results. There are many state-of-the-art methods using multiple object tracking (MOT) [38, 39, 40, 41, 42, 43, 44]. These methods perform well using six publicly available datasets on pedestrian detection, MOT and person search provided by [45, 46, 47]. In evaluations, we choose JDE [38] to represent MOT as benchmark method because JDE is very fast and accurate based on re-implementation of faster object detection compared with [39, 40, 41, 42, 43, 44]. We also use SORT [48] as benchmark methods and add our detection model to completely understand performance of tracking method.

In Fig. 7a, the proposed method is demonstrated to be able to track small nutriment while JDE and SORT with original YOLOv3 and our detection model perform poor (Fig. 7b, 7c, 7d) without tracking results of nutriments even SORT is able to detect some nutriments. By our experiment, the benchmark methods fail to run our datasets because the size of nutriment is too small (maximum size is 13×3613\times 36 pixels) and the speed of nutriment is fast (average nutriment movement from start to end node is 23.823.8 frames).

IV-B Implementation Details

TABLE II: Hardware and software environment for running proposed method and benchmark methods for comparison.
Spesification
Hardware CPU Intel Core i7-9700 CPU @3.00GHz (8 CPUs)
RAM 16 GB
GPU NVIDIA GeForce GTX 745
Software OS Windows 10 Pro 64-bit
IDE
Microsoft Visual Studio Professional 2017
v.15.9.25
Language Python 3.6 64bit
TABLE III: Comparison computational time proposed method and benchmark methods.
Methods N Mean (fps) Std. Dev. (fps) Std. Err. (fps)
95% Confidence
Interval of the Diff.
Lower
(fps)
Upper
(fps)
Ours 419 1.93 0.61 0.03 1.87 1.99
JDE 419 1.87 0.07 0.00 1.86 1.87
YOLOv3 + Sort 419 0.45 - - - -
Our Detection +
Sort
419 0.47 - - - -

For analysis of the computational complexity and execution time of the proposed methodology, a computational time analysis is conducted using a video with 419 frames. Table II shows the specification of hardware and software for comparison. Table III compares the computation time (in fps) for proposed method, namely trajectory mapping and benchmark approaches: JDE and SORT with original YOLOv3 and our detection model. For average and standard deviation of computational time, we reach 1.931.93 and 0.610.61 fps, while JDE spends 1.871.87 and 0.070.07 fps, respectively. SORT only provides average computational time without information of computational time for individual frame. Computational time for both detection model of YOLOv3 and our detection model with SORT performs worst and these benchmark approaches reach 0.450.45 fps and 0.470.47, respectively. By analyzing computational complexity, proposed method runs faster than JDE with the different speed is 0.60.6 fps.

V Conclusion and Discussion

Tracking approach is the one of features to analyze fish behavior to create a decision to optimize the amount of nutriment. Recent studies have shown that it is possible to track movement objects in entire of frames on video. However, there is no agreement to track multiple small nutriments in the video which has interference of hand-held camera and ocean waves. In this paper, tuna nutriment tracking using trajectory mapping in application to aquaculture fish tank has been presented and demonstrated to be promising for interference video containing multiple small nutriment datasets. We have demonstrated tuna nutriment tracking using trajectory mapping and the method consistently performs well on the interference video with good precision trajectory result. We expect our approach to open the door for future work and to go beyond for feature extraction of ripple activity and focus on integrating tracking approach and ripple activity to be a decision to control fish feeding machine.

References

  • [1] F. Fazio, Fish hematology analysis as an important tool of aquaculture: A review. Aquaculture, vol. 500, 237-242, 2019. doi:org/10.1016/j.aquaculture.2018.10.030
  • [2] J. Freitas et al., From aquaculture production to consumption: Freshness, safety, traceability and authentication, the four pillars of quality. Aquaculture, vol. 518, 734857, 2019. doi:org/10.1016/j.aquaculture.2019.734857.
  • [3] B. Carmen et al., Seagrass meadows improve inflowing water quality in aquaculture ponds. Aquaculture, vol. 528, 735502, 2020. doi:org/10.1016/j.aquaculture.2020.735502.
  • [4] W. Liu et al., Characterizing the water quality and microbial communities in different zones of a recirculating aquaculture system using biofloc biofilters. Aquaculture, vol. 529, 735624, 2020. doi:org/10.1016/j.aquaculture.2020.735624.
  • [5] U. Farheen et al., Automatic Controlling of Fish Feeding System. International Journal for Research in Applied Science & Engineering Technology (IJRASET), vol. 6, Issue. 7, 362-367, 2018.
  • [6] H. Liu et al., Biofloc formation improves water quality and fish yield in a freshwater pond aquaculture system. Aquaculture, vol. 506, 735624, 2019. doi:org/10.1016/j.aquaculture.2019.03.031.
  • [7] K. Higuchi et al., Effect of long-term food restriction on reproductive performances in female yellowtail, Seriola quinqueradiata. Aquaculture, vol. 486, 224 - 231, 2018. doi:org/10.1016/j.aquaculture.2017.12.032.
  • [8] J.M. Barron et al., Evaluation of effluent waste water from salmonid culture as a potential food and water supply for culturing larval Pacific lamprey Entosphenus tridentatus. Aquaculture, vol. 517, 734791, 2020. doi:org/10.1016/j.aquaculture.2019.734791.
  • [9] Y. Atoum et al., Automatic Feeding Control for Dense Aquaculture Fish Tanks. IEEE Signal Process. Lett., vol. 22, 1089-1093, 2015. doi:10.1109/LSP.2014.2385794, 2015.
  • [10] A.K. Sabari et al., Smart Fish Feeder. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, vol. 2, Issue. 2, 111-115, 2017.
  • [11] P.C. Oostlander et al., Microalgae production cost in aquaculture hatcheries. Aquaculture, vol. 525, 735310, 2020. doi:org/10.1016/j.aquaculture.2020.735310.
  • [12] C. J. Bridger and R. K. Booth, The effects of biotelemetry transmitter presence and attachment procedures on fish physiology and behavior. Rev. Fisheries Sci., vol. 11, No. 1, 13–34, 2003.
  • [13] S. G. Conti et al., Acoustical monitoring of fish density, behavior, and growth rate in a tank. Aquacult. Eng., vol. 251, No. 2, 314–323, 2006.
  • [14] C. Costa et al., Extracting fish size using dual underwater cameras. Aquacult. Eng., vol. 35, No. 3, 218–227, 2006.
  • [15] J. Xu et al., Behavioral responses of tilapia (oreochromis niloticus) to acute fluctuations in dissolved oxygen levels as monitored by computer vision. Aquacult. Eng., vol. 35, No. 3, 207–217, 2006.
  • [16] L.H. Stien et al., A video analysis procedure for assessing vertical fish distribution in aquaculture tanks. Aquacult. Eng., vol. 37, No. 2, 115–124, 2007.
  • [17] B. Zion et al., Real-time underwater sorting of edible fish species. Comput. Electron. Agricult., vol. 56, No. 1, 34–45, 2007.
  • [18] S. Duarte et al., Measurement of sole activity by digital image analysis. Aquacult. Eng., vol. 41, No. 1, 22–27, 2009.
  • [19] Y. Atoum et al., Automatic Feeding Control for Dense Aquaculture Fish Tanks. IEEE Signal Processing Letters, vol. 22, No. 8, 1089-1093, 2015, doi: 10.1109/LSP.2014.2385794.
  • [20] M.A. Adegboye et al., Incorporating Intelligence in Fish Feeding System for Dispensing Feed Based on Fish Feeding Intensity. IEEE Access, vol. 8, 91948-91960, 2020, doi: 10.1109/ACCESS.2020.2994442.
  • [21] A. Ess et al., A mobile vision system for robust multi-person tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–8, 2008.
  • [22] M. Breitenste et al., Robust tracking-by-detection using a detector confidence particle filter. IEEE International Conference on Computer Vision (ICCV), 1515–1522, 2009.
  • [23] S. Pellegrini et al., You’ll never walk alone: modeling social behavior for multi-target tracking. IEEE International Conference on Computer Vision (ICCV), 261–268, 2009.
  • [24] T. Ueno et al., Motion-blur-free microscopic video shooting based on frame-by-frame intermittent tracking. IEEE International Conference on Automation Science and Engineering (CASE), 837-842, 2015, doi: 10.1109/CoASE.2015.7294185.
  • [25] J. Berclaz et al., Robust people tracking with global trajectory optimization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 35, No. 3, 744–750, 2006.
  • [26] X. Zhang and E. Izquierdo, Real-Time Multi-Target Multi-Camera Tracking with Spatial-Temporal Information. IEEE Visual Communications and Image Processing (VCIP), 1-4, 2019, doi: 10.1109/VCIP47243.2019.8965845.
  • [27] J. Berclaz et al., Multiple object tracking using k-shortest paths optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 33, No. 9, 1806–1819, 2011.
  • [28] H. Jiang et al., A linear programming approach for multiple object tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–8, 2007.
  • [29] L. Zhang et al. Global data association for multi-object tracking using network flows. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–8, 2008.
  • [30] H. Pirsiavash et al., Globally optimal greedy algorithms for tracking a variable number of objects. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1201–1208, 2011.
  • [31] L. Leal-Taixe et al., Learning an image-based motion context for multiple people tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  • [32] L. Leal-Taixe et al., Everybody needs somebody: Modeling social and grouping behavior on a linear programming multiple people tracker. IEEE International Conference on Computer Vision (ICCV) Workshops. 1st Workshop on Modeling, Simulation and Visual Analysis of Large Crowds, 2011.
  • [33] A. Zamir et al., Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs. ECCV, 2012.
  • [34] S. Tang et al., Multi people tracking with lifted multicut and person re-identification. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  • [35] Nghia, Simple video stabilization using OpenCV. http://nghiaho.com/?p=2093, 2010.
  • [36] J. Redmon and A. Farhadi, YOLOv3: An Incremental Improvement. CoRR, vol. abs/1804.02767, 2018.
  • [37] Weisstein and W. Eric, Least Squares Fitting–Polynomial. From MathWorld–A Wolfram Web Resource, https://mathworld.wolfram.com/LeastSquaresFittingPolynomial.html.
  • [38] Z. Wang et al., Towards Real-Time Multi-Object Tracking. CoRR, vol. abs/1909.12605, 2019, http://arxiv.org/abs/1909.12605.
  • [39] W. Lin et al., Real-time multi-object tracking with hyper-plane matching. Technical report, , Shanghai Jiao Tong University and ZTE Corp, 2017.
  • [40] S. Tang et al., Multiple People Tracking by Lifted Multicut and Person Re-identification. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, 3701-3710, 2017, doi: 10.1109/CVPR.2017.394.
  • [41] Y. Zhang et al., A Simple Baseline for Multi-Object Tracking. CoRR, vol. abs/2004.01888, 2020, https://arxiv.org/abs/2004.01888.
  • [42] F. Yu et al., POI: Multiple Object Tracking with High Performance Detection and Appearance Feature. Computer Vision - ECCV 2016 Workshops - Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II, vol. 9914, 36-42, 2016, 10.1007/978-3-319-48881-3_\_3.
  • [43] M. Babaee et al., A dual CNN-RNN for multiple people tracking. Neurocomputing, vol. 368, 69-83, 2019, 10.1016/j.neucom.2019.08.008.
  • [44] B. Pang et al., TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model. CoRR, vol. abs/2006.05683, 2020, https://arxiv.org/abs/2006.05683.
  • [45] A. Milan et al., MOT16: A Benchmark for Multi-Object Tracking. CoRR, vol. abs/1603.00831, 2016, http://arxiv.org/abs/1603.00831.
  • [46] L. Leal-Taixe et al., MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking. CoRR, vol. abs/1504.01942, 2015, http://arxiv.org/abs/1504.01942.
  • [47] A. Ess et al., Depth and Appearance for Mobile Scene Analysis. IEEE 11th International Conference on Computer Vision, ICCV 2007, Rio de Janeiro, Brazil, October 14-20, 2007, 1-8, 2007, doi:org/10.1109/ICCV.2007.4409092.
  • [48] Bewley et al., Simple online and realtime tracking. 2016 IEEE International Conference on Image Processing (ICIP), 3464-3468, 2016, 10.1109/ICIP.2016.7533003.