News

SHI and NEC to develop AI system to identify and report near miss incidents at construction sites

Construction industry operations are heavily influenced by unpredictable factors, including weather, geological conditions, and constantly changing work environments, leading to frequent hazardous incidents.

To enhance safety, there is growing demand for new support systems that leverage digital technologies. One such demand is for a system capable of automatically extracting, visualising, and summarising potentially hazardous scenes (“risk scenes”) from site-specific and time-dependent video footage and work logs.

However, within the construction industry, no technology is currently available for comprehensively supporting the entire process, from the accumulation of video data on construction machinery operations, to the extraction of risk scenes, the analysis and visualisation of near miss incidents, and ultimately, the generation of reports in an end-to-end manner.

Against this backdrop, SHI, leveraging its expertise in construction machinery and data analytics, and NEC, with its long cultivated capabilities in video recognition and generative AI technologies, will jointly initiate the development of a first of its kind system for the construction industry that automatically extracts near miss incidents and generates corresponding reports based on camera footage and sensor data captured from hydraulic excavators.

The new system will utilise an extraction AI model, trained on real-world hydraulic excavator data accumulated on the SHI Group’s ICT and IoT common platform “SHICuTe”, to first identify and extract risk scenes from recorded video footage.

These risk scenes, together with operational data from the hydraulic excavators, will then be analysed using NEC’s proprietary technology, which combines video recognition with generative AI, and stored as multimodal data that incorporates temporal and spatial information.

Based on this data, along with SHI’s expertise in construction site machinery operations and human workflows, the system will cross-reference hazardous and prohibited behaviour data, which are defined by accidents, construction equipment failures, and operations requiring particular attention, as well as company-specific data.

Based on these matching results, the system will automatically identify the risk scenes that should be reported and automatically generate high-quality near-miss reports that provide a concise summary of the circumstances surrounding each incident.

Prior to this initiative, a technical proof of concept was conducted in September 2025 to verify a system that automatically extracts near miss incidents and generates reports from video footage captured by cameras mounted on hydraulic excavators. The results confirmed that, based on the risk scenes extracted from the footage, the system was able to report near miss cases, including potential accident scenarios and their associated circumstances.

Building on these outcomes, the joint development starting in April 2026 will aim to further expand the types of near miss incidents that can be identified and enhance the report generation capabilities in line with customers’ safety management needs, thereby contributing to the realisation of safer construction sites.

In fiscal year 2026, technical development and validation of the system will be advanced by utilising on site data and safety management expertise from SHI, together with AI technologies and advanced technology consulting services provided by NEC, with the aim of achieving practical implementation in fiscal year 2027.

Looking ahead, the companies plan to broaden the system’s applicability beyond scenes where physical contact between workers and machinery may lead to occupational accidents to include unsafe conditions that may not be readily recognised by workers, as well as considerations for site specific operational rules, thereby further expanding its scope of use.

SHI and NEC will continue to combine their respective strengths to develop and validate new technologies, contributing to the realisation of safer construction sites.

Tell us your news story - news@earthmoversmagazine.co.uk