A horizon line annotation tool for streamlining autonomous sea navigation experiments

In maritime images captured from terrestrial platforms, such as buoys, ships, and USVs, the horizon feature (sometimes called the sea-sky line) is defined as the line separating the sea region and the region right above it zardoua2021survey (1) (see Figure 1 and 1). The literature includes several ways to represent the location of the horizon line. The position and tilt representation, which we denote by the pair of parameters $\{Y,\phi\}$ (see Figure 1), is by far the most common and useful representation zardoua2021survey (1, 2) because it provide a direct and precise measure of the horizon’s location. Additionally, measuring the detection error based on this representation provides a clear and intuitive idea of how well the horizon detection algorithm is performing.

The published literature shows that the sea horizon line exhibits a wide range of applications, especially in terms of autonomous navigation. In the case of autonomous sea navigation, the horizon feature is involved on almost every stage of the processing pipeline. We’ll cite four major application categories. First, numerous papers and patents used the horizon line to auto-calibrate the camera rs13142795 (3, 4, 5), which include non-maritime scenes as well calibration (6, 7). Second, the position and tilt of the horizon allows the computation of rotational and translational matrices that would digitally stabilize captured video frames overview_stabilization (8, 9, 10, 11, 12, 13, 14, 15, 16). A stabilized video is not only easier to watch but facilitates subsequent analyses as well Reichert2021 (16, 2, 10). Third, many works used the horizon to segment the image (into sea and non-sea region). In the same context, the horizon feature provides computational benefits to the object detection task by reducing the search region down to that around the horizon or by eliminating the sky pixels that will obviously not depict any obstacle asv (17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 4, 34). Fourth, even after the detection of maritime objects on the image coordinates, some researchers used the horizon line to geo-localize surrounding objects and obstacles 6070512 (35, 36, 37, 38), i.e., find the distance in meters from the camera to the obstacle.

Our decision to develop and publicly share our annotation tool was driven by the need for a fast and reliable sea line annotation tool, which we clarify in the following points:
•  the most extensive dataset, the SMD, lacks various sea conditions that must be collected and properly annotated
•  the sea line annotation of the SMD include significant mistakes and must be corrected before the experiment
•  many papers such as liang (39, 40, 10, 12) include their own private image dataset but do neither mention how did they annotate their image nor share their annotation tool.
•  currently available tools, such as LabelMe and SuperAnnotate, lack public free access and customization. The proposed software freely provide unique and tailored features and can be directly used without further modifications.

Section 2 outlines the primary components and features of the proposed software, which are summarized below: •  An intuitive graphical user interface (GUI) and easy browsing through video frames
•  Visualization tools, such as the ability to adjust the thickness of the annotated sea line
•  The option to duplicate current annotations onto previous non-annotated frames, which is particularly useful for on-shore fixed cameras
•  Shortcut keys and the ability to annotate all video frames with just the mouse
•  Misuse warnings, such as alerting the users if they forget to annotate all the frames
•  The ability to save incomplete annotation files and return to them later for completion, which is especially helpful for video files with a large number of frames.

Overall, these features enhance the functionality and user-friendliness of the software and make it a valuable tool for video annotation.

Load directories is the first block to use. As shown in Figure 3, the block contains three self-explanatory buttons: load video file, save annotated file, and load existing gt file. The software supports two video file formats: .avi and .mp4. The loaded video will appear on the Image display block, which we detail in Section 2.2. The third button (load existing gt file) allows the user to start the annotation process from an existing annotated file (also known as the Ground Truth file). As the user browses through the video frames, the annotated line of the viewed frame will be drawn in red, as shown in Figure 4. Such a feature allows the user to review the annotated frames and correct the eventual mistakes. Whether the user annotates the video from scratch or starts the annotation from an existing GT (Ground Truth) file, clicking on the save annotated file will always save in the directory specified by the user an annotation file by suffixing the current video file name with LineGT.npy.

The image display block will show the first frame of the loaded video file. The frame size will not change as long as the entire GUI fits into the computer screen. Otherwise, the frame will be down-scaled to the size that would allow the entire GUI to fit onto the computer screen. If the current frame corresponds to an annotated line, that line will be automatically drawn and shown on the displayed frame. The user can draw the annotation line by maintaining the left mouse button. one that button is released, the application will automatically infer and draw the full line on the image display, as shown in Figure 5. If the user is not satisfied with the line he is drawing, i.e., before releasing the left button of the mouse, he can abort the drawing process by right-clicking the mouse before releasing the left mouse button. While the mouse is on the image display, its Cartesian Coordinates $(x,y)$ are constantly updated and displayed on the bottom left of the Image display block, as shown in Figure 6. We note that the image frame displayed in Figure 5 is downsized so that the GUI fits in the screen.

Even in such a case, the Cartesian coordinates displayed are computed to correspond to the original size of the frame. This choice is not arbitrary for two reasons: (1) the downsizing factor changes according to the frame size and the computer screen; (2) the position of the annotated line must correspond to the original frame size.

We show the annotation block in Figure 7. The features of this block involve the manipulation of annotated lines. The Validate button validates the non-aborted line drawn, such as the line we mentioned in Figure 5. The Delete button deletes the annotated line of the current frame. The Hide button hides the annotation of the current frame. It can be shown again using the Show button. Instead of using these buttons, the user may prefer keyboard keys. The key corresponding to each button is written between parenthesis (e.g., (v) for Validate button). Section 4 provides a keymap containing the full list of the GUI shortcuts.

The Annotated line thickness shown under the four buttons in Figure 7 is a text entry that allows the user to specify the thickness of the annotated line. We note that such a thickness is not an annotation information and serves only a visualization purpose. This entry has built-in features to handle non-conforming thickness values, such as texts or negative numbers. The last element of the annotation block is the Current annotation, which displays the position and tilt of the annotated line. When no annotation line exists for the current frame, the values displayed will be replaced by three interrogation marks ???.

After loading a video file, drawing a line, and validating it using the Validate Button or its corresponding shortcuts, the user will browse to the next frames. The Browse block and its features ensure easy browsing through the video frames. As we show in Figure 8, the two buttons $<<$ and $>>$ browse to the previous and next frames, respectively. Easier browsing alternatives are listed in Section 4.

Right under the two browsing buttons, the interface shows the index of the current frame and the total number of frames. The browsing offset defaults to 1. This means that the index of the next (or previous) frame is equal to the index of the current frame + 1 (or - 1). Browsing with an offset of 1 slows down the browsing process when the video file has a large number of frames. Thus, the user can quickly go to any video frame by changing the browsing offset in the Enter a browsing offset entry.