Absolute 3D Pose Estimation and Length Measurement of Severely Deformed Fish from Monocular Videos in Longline Fishing

Inspired by [7, 8], we introduce a relative 3D fish template, whose unit is a pixel. It is a 3D point set on a surface generated from a Pacific Halibut Template, as illustrated in Fig.1. The origin point is the center point of the fish body. We denote the initial template point set as $S_{0}=\{(x_{0i},y_{0i},z_{0i})\in R^{3}\}_{i=1}^{N}$, where $N$ is the total number, and $0$ represents the initial point set before any deformable transformation. To simulate the deformable pose of the fish, the template iteratively executes the following four transformation sequentially:

Scale Transformation We denote the scale parameter as $s\in R$. The scaling transformation is to multiply $s$ with each point, $(x_{0i},y_{0i},z_{0i})$, in the template point set $S_{0}$:

$$\left(x_{1i},y_{1i},z_{1i}\right)=s\cdot\left(x_{0i},y_{0i},z_{0i}\right),$$

(1)

resulting in a new point set $S_{1}=\{(x_{1i},y_{1i},z_{1i})\in R^{3}\}_{i=1}^{N}$.

Bending Transformation We denote the bending parameter as $r\in R$ being the radius of a cylinder, which is tangent with the template at the initial position, as illustrated in Fig.1. The bending transformation maps each point, $(x_{1i},y_{1i},z_{1i})$, in $S_{1}$ to a new 3D point, $(x_{2i},y_{2i},z_{2i})\in R^{3}$, on the cylinder:

$\displaystyle\begin{cases}x_{2i}&=x_{1i},\\ z_{2i}&=r\left(1-\cos\frac{y_{1i}}{r}\right),\\ y_{2i}&=\frac{z_{2i}}{\tan\frac{y_{1i}}{2r}},\\ \end{cases}$

(2)

which is fish length preserving, resulting in a new point set, $S_{2}=\{(x_{2i},y_{2i},z_{2i})\in R^{3}\}_{i=1}^{N}$ (the purple surface in Fig.1).

Translation Transformation We denote the translation parameters as $(T_{x},T_{y},0)\in R^{3}$:

$$\left(x_{3i},y_{3i},z_{3i}\right)=\left(x_{2i},y_{2i},z_{2i}\right)+\left(T_{x},T_{y},0\right),$$

(3)

resulting in a new point set $S_{3}=\{(x_{3i},y_{3i},z_{3i})\in R^{3}\}_{i=1}^{N}$.

Rotation Transformation We denote three rotation parameters as $\alpha,\beta,\gamma\in[0,2\pi]$, which respectively represent the rotation degree around $x$, $y$, $z$ axis [9]. We construct three basic (elemental) rotation matrices, $R(\gamma)$, $R(\beta)$, and $R(\alpha)\in R^{3\times 3}$ and rotation transform each point, $(x_{3i},y_{3i},z_{3i})$ to a new point $(x_{4i},y_{4i},z_{4i})\in R^{3}$:

$\displaystyle\left(x_{4i},y_{4i},z_{4i}\right)=\left(x_{3i},y_{3i},z_{3i}\right)\cdot R(\gamma)\cdot R(\beta)\cdot R(\alpha).$

(4)

Finally, after performing these four transformations one-by-one iteratively, we denote the final relative 3D template as $S_{4}=\{(x_{4i},y_{4i},z_{4i})\}_{i=1}^{N}\in R^{3}$. These four transformations are differentiable with respect to their parameters, which enables us to do relative 3D pose estimation in the next section.

Based on a target’s 2D mask, we can directly estimate the relative 3D pose of the target fish and 2D keypoints in image without using expensive 3D ground truth [11, 12] or multi-view images. We iteratively minimize the chamfer distance between the orthographic projection contour of 3D template and one target’s 2D mask contour, as illustrated in Fig.3.

Orthographic Projection Contour We get the orthographic projection points of the 3D template by directly setting each point, $(x_{4i},y_{4i},z_{4i})$ to $(x_{4i},y_{4i},0)$. Then, we convert these points into a binary mask and execute Canny edge detector [13] and denote the orthographic projection contour points as, $S_{template}=\{(x_{4i},y_{4i},z_{4i})\in R^{3}\}_{i}^{n}\in S_{4}$, where $n$ varies given different transformations’ parameters.

Target Contour The target contour points are acquired by applying the Canny edge detector on the target’s 2D binary mask from Stage-1. Because the template’s center point is at the origin point before four transformations, we also transform the mask image center to the origin point and then get the final target contour point set, denoted as $S_{target}=\{(x_{j},y_{j},z_{j})\in R^{3}\}_{j=1}^{m}$.

Chamfer Distance Loss Given a fixed $S_{target}$, and an adjustable $S_{template}$, we iteratively and sequentially adjust each point in $S_{template}$ with seven parameters in four deformation transformations so that the chamfer distance, Eq. 5, is minimized by gradient descent [14]:

	$\displaystyle d_{CD}\left(S_{template},S_{target}\right)=\sum_{p_{i}\in S_{template}}\min_{q_{j}\in S_{target}}\|p_{i}-q_{j}\|_{2}^{2}$		(5)
	$\displaystyle+\sum_{q_{j}\in S_{target}}\min_{p_{i}\in S_{template}}\|p_{i}-q_{j}\|_{2}^{2},$

where the first term implies, for each point $p_{i}=(x_{4i},y_{4i},z_{4i})\in S_{template}$, calculate the Euclidean distance of this point and its nearest point $q_{j}=(x_{j},y_{j},z_{j})\in S_{target}$. The second term implies, for each point in $q_{j}=(x_{j},y_{j},z_{j})\in S_{target}$, find the nearest point $p_{i}=(x_{4i},y_{4i},z_{4i})\in S_{template}$ to calculate their Euclidean distance. After optimization, the relative 3D fish template’s pose is the estimated target’s relative 3D pose in the camera coordinate system but with $pixel$ as a unit. Plus, since we can predefine three keypoints (head, center and tail) on the template, their orthographic projections are the 2D keypoints in the target mask, as illustrated in Fig.3, denoted as $h_{2d}$ $(U_{h},V_{h})$, $c_{2d}$ $(U_{c},V_{c})$, $t_{2d}$ $(U_{t},V_{t})$ respectively in the whole image coordinate system. We denote center, head, and tail points in target’s relative 3D pose as $h$ $(x_{h},y_{h},z_{h})$, $c$ $(x_{c},y_{c},z_{c})$, and $t$ $(x_{t},y_{t},z_{t})$ respectively.

To have a 3D length measurement of the fish body, we introduce an absolute 3D localization method to locate three keypoints in absolute 3D, denoted respectively as $H^{\prime}$ $(X_{hc},Y_{hc},Z_{hc})$, $C^{\prime}$ $(X_{cc},Y_{cc},Z_{cc})$, and $T^{\prime}$ $(X_{tc},Y_{tc},Z_{tc})$, where the second subscript $c$ means the camera coordinate system and the unit is $millimeter$ instead of $pixel$. This task is formulated as the following closed-form solution.

Back Projection We first use camera intrinsic parameters to back project $h_{2d}$ $(U_{h},V_{h})$, $c_{2d}$ $(U_{c},V_{c})$, $t_{2d}$ $(U_{t},V_{t})$ to 3D space in the camera coordinate system without depth:

$$\left[\begin{array}[]{c}X\\ Y\\ 1\end{array}\right]=K^{-1}\left[\begin{array}[]{c}U\\ V\\ 1\end{array}\right],$$

(6)

where $K$ denotes camera intrinsic parameters, that is a $3\times 3$ matrix, obtained by method from [15]. Now for $h_{2d}$, $c_{2d}$, $t_{2d}$, respectively we get $h^{\prime\prime}$ $(X_{h},Y_{h},1)$, $c^{\prime\prime}$ $(X_{c},Y_{c},1)$, and $t^{\prime\prime}$ $(X_{t},Y_{t},1)$ in the camera coordinate system.

Reference Plane In the longline fishing [16, 17], all fish are hooked and pulled up by a line. So our assumption is that all fish’s center points are always on a reference plane, which coincides with the plane of checkerboard of known grid size used in the camera calibration, defined as $Z=0$ plane in the world coordinate system, as illustrated in the right bottom corner of Fig.2. Then we use the solvePnP method [18] to calculate the rotation matrix, $R_{3\times 3}$, and translation vector, $T_{3\times 1}$, between the world coordinate system and the camera coordinate system. Under our assumption, center point $C^{\prime}$ is a point $(X_{cw},Y_{cw},0)$ in the world coordinate system. With $R_{3\times 3},\ T_{3\times 1}$ and $K_{3\times 3}$, we can get $Z_{cc}$ (depth) with Eq.7. We also need to calculate homography matrix $H$ between the image plane and $Z=0$ plane in the world coordinate system:

$\displaystyle\begin{cases}H\quad=\quad K_{3\times 3}&\cdot\left[\begin{array}[]{lll}R_{1}&R_{2}&T\end{array}\right]_{3\times 3},\\ \left[\begin{array}[]{c}X_{cw}/Z_{cc}\\ Y_{cw}/Z_{cc}\\ 1/Z_{cc}\end{array}\right]&=H^{-1}\cdot\left[\begin{array}[]{c}U_{c}\\ V_{c}\\ 1\end{array}\right],\end{cases}$

(7)

where $R_{1},\ R_{2}$ are the first two columns in $R_{3\times 3}$.

Finally, with $Z_{cc}$ (depth), we can get the center point $C^{\prime}$:

$$(X_{cc},Y_{cc},Z_{cc})=Z_{cc}\cdot(X_{c},Y_{c},1).$$

(8)

3D Localization A 3D localization method with a closed-form solution is introduced to calculate head point $H^{\prime}$ and tail point $T^{\prime}$, which are on the following two lines respectively:

	$\displaystyle Line_{1}:(X,Y,Z)$	$\displaystyle=m\cdot(X_{h},Y_{h},1),m\in R,$		(9)
	$\displaystyle Line_{2}:(X,Y,Z)$	$\displaystyle=n\cdot(X_{t},Y_{t},1),n\in R.$

As illustrated in Fig.4, we set up a new coordinate system with unit being $pixel$, called $tmp$, whose origin point is at $C^{\prime}$ and has no relative rotation with respect to the camera coordinate system but the translation vector between them is $(X_{cc},Y_{cc},Z_{cc})$. In Section 2.2, we got the relative 3D pose, i.e., $h$, $c$, and $t$ in the camera coordinate system with unit being $pixel$. Now we want to estimate the absolute fish pose centered at $C^{\prime}$ so we translate $h$, $c$ and $t$ to the $tmp$ coordinate system by translating $c$ to $tmp$’s origin point:

	$\displaystyle c^{\prime}$	$\displaystyle=(X_{cc},Y_{cc},Z_{cc}),$		(10)
	$\displaystyle h^{\prime}$	$\displaystyle=(x_{h}-x_{c},y_{h}-y_{c},z_{h}-z_{c})+(X_{cc},Y_{cc},Z_{cc}),$
	$\displaystyle t^{\prime}$	$\displaystyle=(x_{t}-x_{c},y_{t}-y_{c},z_{t}-z_{c})+(X_{cc},Y_{cc},Z_{cc}),$

where $t^{\prime}$ $(x_{tc},y_{tc},z_{tc})$ and $h^{\prime}$ $(x_{hc},y_{hc},z_{hc})$ are relative head point and tail point in the camera coordinate system. Note that the absolute head point, $H^{\prime}$, and tail point, $T^{\prime}$, must also lie on the following lines $C^{\prime}h^{\prime}$ and $C^{\prime}t^{\prime}$ respectively:

	$\displaystyle(X,Y,Z)$	$\displaystyle=a\cdot(x_{hc},y_{hc},z_{hc})+(1-a)\cdot(X_{cc},Y_{cc},Z_{cc}),a\in R,$		(11)
	$\displaystyle(X,Y,Z)$	$\displaystyle=b\cdot(x_{tc},y_{tc},z_{tc})+(1-b)\cdot(X_{cc},Y_{cc},Z_{cc}),b\in R.$

So, $H^{\prime}$ must be the intersection point of line $C^{\prime}h^{\prime}$ and $Line_{1}$, and $T^{\prime}$ must be the intersection point of line $C^{\prime}t^{\prime}$ and $Line_{2}$. To find $H^{\prime}$ and $T^{\prime}$, we minimize the distance between two lines, which results in a closed-form solution with the simple least-squares method [19], omitted due to the pages limit. Finally, and the absolute fish length can thus be calculated:

$\displaystyle{Length}$

$\displaystyle={\mid H^{\prime}T^{\prime}\mid}\cdot{\frac{\overset{\frown}{ht}}{\mid ht\mid}},$

(12)

where $\frac{\overset{\frown}{ht}}{\mid ht\mid}$ is the bending ratio. The left bottom corner of Fig.2 is a 3D localization example.