Field Safety Inspection PoC with a Multimodal LLM

TL;DR A multimodal LLM can be useful as an assistant that quickly summarizes possible risk factors from site photos. Final safety judgment still needs verifiable criteria and human confirmation.

Testing safety inspection support with site photos and safety-rule prompts

Field safety inspections still depend heavily on human experience and manual work. Inspectors need to review site photos, check protective equipment and risk factors, and then write the findings into a report.

Recent multimodal LLMs can understand images and text together. This PoC explored whether site photos and safety-rule prompts can be used to support safety checks and draft inspection reports.

PoC Goals

The goal was not to make final judgments about a specific site or device. The goal was to check whether a multimodal LLM can act as an assistant for industrial safety inspection.

• Validate whether safety rules can be checked from site photos
• Check whether rules can be applied through prompts without separate model training
• Explore risk summarization and inspection report drafting
• Consider future expansion into video-based inspection flows

Approach

The input flow was intentionally simple. A site photo was provided with a prompt containing safety rules and the desired output format. The model was asked to identify visible workers, protective equipment, and risk factors, then return a structured inspection result.

The advantage of this approach is that teams can experiment without first preparing a dedicated training dataset or a specialized vision model. By changing the prompt, the inspection criteria and output format can be adjusted quickly.

What We Found

The PoC confirmed several useful possibilities.

• Helmet usage could be identified.
• Safety vest usage could be identified.
• The approximate number of workers in the photo could be estimated.
• Visible risk factors could be summarized.
• Inspection results could be organized into a report format.

There were also clear limits. Accuracy dropped with low-resolution images, distant workers could be missed, and small or partially hidden details such as safety harness fastening were difficult to judge reliably.

Insight

In this test, the multimodal LLM was more suitable as an inspection assistant than as a final decision maker. It was useful for summarizing visible risk candidates and organizing the items that a human inspector should review.

Report drafting was especially promising. Instead of writing every finding from scratch after reviewing photos, inspectors can start from a generated draft and focus on final confirmation and correction.

Future Expansion

The next step is to explore a flow that extracts snapshots from CCTV, drone, or mobile video and uses them for periodic safety inspection support.

The point is not to let AI replace all safety judgment. Field safety requires responsibility and context. A realistic starting point is to use AI to support repetitive inspection, summarize risk candidates, and help people focus on more important decisions.

Closing

Multimodal LLMs show potential for interpreting site photos together with safety rules. They are not reliable enough to judge every situation on their own, but they are worth testing for safety inspection assistance and report automation.

In industrial safety, AI can start by helping people see faster and record more consistently, rather than replacing human responsibility.

References

• NIST AI Risk Management Framework
• OWASP Top 10 for Large Language Model Applications