Modern geospatial intelligence combines multimodal sensor data (e.g., satellite EO/IR, SAR, AIS, and RF) as seen in Fig. 1  for persistent risk and threat monitoring. Maritime compliance (e.g., sanctions monitoring), supply chain and critical infrastructure risk (e.g., attacks on undersea Internet cables) monitoring, and Know Your Vessel (KYV) capability for ship financing are example applications.

Figure 1: MaritimeIQ multimodal maritime domain awareness platform.

Figure 2: Ship volume comparison from computer vision applied to Planet imagery and commercial AIS-reported data.

The world's oceans are vast, and the ships traversing them often carry more than just cargo; some carry secrets. These are the "dark vessels": ships that intentionally switch off their Automatic Identification System (AIS) transponders to avoid detection for security or other reasons (e.g., Fig. 2) and frequently engage in ship-to-ship transfer (STS), illegal, unreported, and unregulated (IUU) fishing, smuggling, or other illicit activities.

Persistently monitoring dark fleets is a critical challenge for maritime security. To achieve this, we rely on marrying two key data sources: (1) the last known AIS position of a vessel before it went dark, and (2) later computer vision detections of ships from satellite imagery, with a time delay. The challenge is: how do you confidently match that ghostly last vessel location to a potential current satellite detection when there are multiple ships in the satellite image? This is where combinatorics meets computer vision.

The Dark Ship Tracking Problem and the Shortcomings of Heuristics

Imagine a scenario: a vessel V transmits its last AIS signal at location X before disappearing. A few minutes or hours later, a satellite image covers the area, and a computer vision model flags five different ship detections, D_1, D_2, D_3, D_4, D_5. Which one is the V?

The straightforward and popular solution is a simple heuristic:

1. Calculate the geographic distance between the last known AIS location of the dark vessel V and every new location from satellite imagery detection (i.e., bounding boxes around ships).

2. Assign the AIS track to the closest satellite detection within a plausible range.

The Critical Flaw of Heuristics: This greedy approach is local in its thinking. It only considers the best match for one dark ship at a time. In a congested maritime area of interest (e.g., the Strait of Hormuz in Fig. 2), or when multiple dark ships are present, this leads to significant problems, such as false matches and inaccurate pairings. The heuristic fails to find the best overall set of unique, non-conflicting assignments.

The Combinatorial Optimization Advantage

Our mathematical approach reframes the problem as a combinatorial optimization problem. It finds the unique, one-to-one mapping between a set of known AIS target locations and a set of newly detected ship positions from satellite imagery that results in the minimum total cost (determined by the business use case) for all assignments simultaneously. The cost function may represent the mismatch between the AIS location and the imagery detection location mapping. By moving beyond simple geographic distance, we can also create a cost function that incorporates the physics and typical behavior of ships, making the final assignments highly plausible.

Dynamic Plausibility via Speed and Trajectory

Simple distance is static. A dynamic approach uses the vessel's last known state to predict its movement.

• Use the vessel's last recorded Speed Over Ground (SOG) and Course Over Ground (COG) from the final AIS transmission.

• Employ a prediction model (like a Kalman Filter) trained on historical AIS data to predict the most likely position and uncertainty of the dark vessel at the exact time the satellite image was taken.

The cost includes the difference between the actual detection and the predicted location. The system also introduces a heavy penalty for matches that imply an unrealistic speed or an impossible course change since the AIS cut-off, ensuring the result adheres to the laws of physics (e.g., a heavy tanker cannot stop and reverse course instantly).

Comparative Analysis

The integration of advanced data modeling into combinatorial optimization algorithms will transform maritime surveillance, turning noisy satellite detections into high-confidence intelligence for catching dark ships.

Figure 3. Dark vessel mapping between the last known AIS locations (blue points) and locations from satellite imagery analytics (green points). Red points are ships from satellite imagery that do not have a corresponding AIS location. 

Figure 4. Dark vessels identified in the Kerch Strait from Planet imagery.

Example Analysis

Figure 3 shows a result from our combinatorial optimization approach. 27 vessels went dark, i.e., switched off their AIS. A total of 59 vessels were detected from the satellite image. The figure shows that the optimization approach correctly identifies all 27 vessels in the image, while the remaining 32 vessels do not have a corresponding AIS location. Extensive experimental analysis of this approach indicates that this method has an accuracy between 93% and 100% on a large commercial ground truth dataset, even when the gap between the last known AIS location and satellite image acquisition is several days apart. 

Fig. 4 shows an example of dark vessels identified in the Kerch Strait applying the combinatorial optimization solution. Observe that this method identifies the MMSI numbers, vessel names, vessel characteristics (e.g., size, shape, type), etc., for a complete picture. Cross-referencing these vessels against sanctions lists, ownership information, flags, etc., provides a risk score for every vessel – enabling complete maritime domain awareness. 

Finally, this tool can be applied for other geospatial applications including multimodal space domain awareness.

Interested in learning more about applying this capability for your business use case and multimodal data? Contact us at AI for Maritime Intelligence