AR & LiDAR Wound Assessment
A React Native clinical tool that measures wound area, perimeter and volume from the phone's camera and depth sensor, tracks healing across visits, and feeds a structured documentation and treatment workflow.
Anonymized under NDA — the client and clinical imagery are omitted, and algorithms are described without proprietary detail. The engineering is real.
A US wound-care company runs a point-of-care platform that nurses in long-term-care facilities use to evaluate wounds, document them to a mandatory minimum dataset, and receive treatment recommendations matched to the wound and the facility's protocols. I led the mobile measurement layer: ARKit depth and LiDAR for distance, OpenCV for contours, Skia for the clinician's editing canvas, and a ghost overlay that shows how a wound changed since the last visit — turning the phone into the instrument the rest of the workflow depends on.
Sub-cm
Distance measurement stability
Kalman filter + adaptive EMA smoothing
3
Wound metrics computed
Area, perimeter, volume
RYB
Tissue classification
Red / yellow / black with live controls
0
Patient data stored on the device
Encrypted in transit, nothing at rest locally
- Role
- Lead Mobile Engineer (contract)
- Timeline
- Jul 2025 – Dec 2025
- Type
- Contract · Healthcare (US wound care)
- Scope
- Mobile architecture, computer vision, AR/LiDAR, canvas tooling
Stack
Chapter 01
Where it started
Wound care runs on measurement, and measurement in most facilities is a paper ruler and a guess. Length × width, eyeballed at an uncontrolled distance, varies by clinician and by day — which makes healing impossible to trend, treatment hard to justify, and documentation vulnerable in exactly the setting where audits and reimbursement reviews are routine.
The documentation itself was no better. Free-text wound notes meant every nurse described the same wound differently, mandatory fields went missing, and the downstream workflow — treatment selection, supply orders, progress reporting — inherited incomplete data it couldn't trust.
Phones now ship with depth sensors, but raw depth data is noisy, hand-held distance drifts by centimetres, and a wound boundary is irregular. Turning that into a repeatable clinical number is a sensor-fusion and computer-vision problem — inside a cross-platform app, operated one-handed by a nurse at the bedside.
And the codebase I inherited worked against all of it: images flowed through untyped global state, aspect-ratio and canvas transformations were duplicated across screens, and rendering artifacts differed between iOS and Android — unacceptable in an app whose output is a medical measurement.
The brief, in effect: make the phone a trustworthy instrument, make the documentation structured and complete by construction, and never let patient data rest on the device.
Chapter 02
What I built
The system first, then the people it serves, then the build log of how it came together.
The system
Client
- React Native app (iOS first)
- Skia canvas: freehand & polygon ROI
- Ghost contour overlay
- Dropdown-guided evaluation forms
- RYB slider controls
Core
- ARKit depth + LiDAR capture
- Kalman filter + adaptive EMA
- OpenCV contour & ROI normalization
- Pixel-to-mm calibration, area/perimeter/volume
- Minimum-dataset validation
Services & data
- ImageService (capture, transforms)
- AssetManager (storage, history)
- Redux Toolkit state
- Patient visit history & progress reports
- Encrypted transport, zero local PHI
Sensor fusion and vision run on-device; the clinician edits the result before it is saved to the visit.
Sensor fusion and computer vision run entirely on-device: ARKit depth feeds the Kalman/EMA distance pipeline, OpenCV turns the capture into a calibrated contour, and the Skia canvas lets the clinician correct it before anything is committed. The structured evaluation wraps the measurement in the mandatory minimum dataset, and the visit record — measurements, tissue classification, images — syncs to the clinical backend over encrypted transport, feeding patient history, healing trends and the treatment-recommendation workflow. Nothing clinical ever rests on the phone.
Who uses it
Clinician / nurse
- Capture a wound with guided distance feedback
- Accept or edit the detected contour
- Complete the structured evaluation — minimum dataset enforced
- Compare with previous visits via ghost overlay
Care team
- Review visit history, measurements and tissue trends
- Track healing trajectories over time
- Pull structured metrics into progress reports
Facility administrator
- Audit-ready documentation on every wound
- Consistent records across nurses and shifts
- Statistics and reporting across the facility
The build log
- 01
Distance you can trust: ARKit depth streams fused through a Kalman filter with adaptive EMA smoothing, holding camera-to-wound distance stable to sub-centimetre precision while the clinician's hand moves — with live guidance to hold the right range.
- 02
Contours from pixels: OpenCV extracts the wound boundary, normalizes the region of interest and calibrates pixels to millimetres dynamically from the measured distance, yielding area, perimeter and volume per capture.
- 03
Clinician in control: interactive 2D canvas tools built on React Native Skia for freehand selection and polygon ROI editing — the algorithm proposes, the human corrects, and only the corrected result is saved.
- 04
Structured documentation by construction: guided, dropdown-driven evaluation capturing the mandatory minimum dataset for every wound, so a record can't be saved incomplete and every record reads the same way.
- 05
Healing over time: a Ghost Contour Overlay engine superimposes earlier visits' contours on the live view, and RYB (red-yellow-black) tissue classification with real-time controls quantifies granulation, slough and necrosis per visit.
- 06
Foundations and safety: media handling refactored into centralized ImageService and AssetManager modules (one transform pipeline for both platforms), with captures encrypted in transit and no patient data persisted on the device.
Chapter 03
Turning points
The moments that shaped the build, and the roads not taken.
Depth from a hand-held phone drifts as the clinician moves, and the clinician is holding the phone one-handed over a patient.
- Challenge
- Deliver a distance reading stable enough to calibrate real-world size, in real time, with live feedback that keeps the capture in the valid range.
- Approach
- Fused ARKit depth through a Kalman filter with adaptive EMA smoothing tuned for hand motion, surfacing guided distance feedback in the capture UI so the nurse knows when the frame is measurable.
- Result
- Sub-centimetre distance stability under hand-held motion — the foundation every downstream metric depends on.
Wounds are irregular, lighting varies by room, and the same wound must produce comparable numbers across visits by different nurses.
- Challenge
- Extract a usable boundary, convert it into clinical metrics, and make progression visible rather than anecdotal.
- Approach
- OpenCV contour extraction with ROI normalization and dynamic pixel-to-mm calibration from the measured distance; the ghost overlay renders prior visits' contours on the live view, and RYB classification quantifies tissue composition per capture.
- Result
- Area, perimeter and volume computed per capture and editable before saving — and healing became something you watch happen on screen, not infer from photo albums.
The measurement feeds a documentation and treatment workflow that regulators and reimbursement reviewers read.
- Challenge
- Guarantee every wound record is complete, structured and consistent — without slowing a nurse down at the bedside.
- Approach
- Replaced free-text capture with a dropdown-guided evaluation that walks the mandatory minimum dataset in the order a nurse actually works, with validation refusing incomplete records and measurements auto-filled from the capture.
- Result
- Records became complete by construction and comparable across the whole facility — structured enough to drive treatment recommendations and progress reporting downstream.
Images were transformed differently on each screen and platform in the inherited codebase.
- Challenge
- Eliminate cross-platform rendering artifacts in a medical imaging app — without a rewrite.
- Approach
- Refactored media handling into centralized ImageService and AssetManager modules, unified aspect-ratio canvas transforms behind one pipeline, and removed untyped global image state.
- Result
- Consistent rendering on iOS and Android, one place to fix any future transform bug, and a materially smaller surface area for the class of defect this app can least afford.
Forks in the road
Signal processing
Took this road
Kalman filter with adaptive EMA
Not this one
Raw depth readings
Hand-held depth jitters by centimetres, and every millimetre of distance error multiplies into the area calculation. Filtering made distance stable enough to calibrate pixels to millimetres reliably — the measurement is only as good as this layer.
Clinical trust
Took this road
Algorithm proposes, clinician edits
Not this one
Fully automatic contour
Wound boundaries are genuinely ambiguous — undermining, maceration and lighting all blur the edge. Editable ROI tools kept clinical judgement in the loop, which made the numbers defensible in an audit instead of a black box's opinion.
Documentation
Took this road
Dropdown-guided structured evaluation
Not this one
Free-text wound notes
Structured choices capture the mandatory minimum dataset every time and make records comparable across nurses, shifts and facilities. Free text reads nicely once; structured data trends, audits and drives treatment logic forever.
Canvas
Took this road
React Native Skia for the drawing surface
Not this one
WebView or native canvas per platform
One high-performance surface for both platforms, with direct control over transforms and overlays — essential when the overlay is a previous visit's contour that must align to the millimetre.
Data safety
Took this road
Zero patient data at rest on the device
Not this one
Local caching for convenience
A nurse's shared facility phone is the worst possible place for PHI. Encrypting in transit and persisting nothing locally removed the entire lost-device threat class at the cost of requiring connectivity to commit a visit — the right trade in a clinical setting.
Chapter 04
Did it work?
What changed, what shipped, and the notes I kept for next time.
Ruler vs. instrument
| Dimension | Manual assessment | AR / LiDAR app |
|---|---|---|
| Distance | Hand-held guess | Depth-sensed, Kalman-filtered, sub-cm stable |
| Boundary | Eyeballed with a ruler | OpenCV contour, editable by the clinician |
| Metrics | Length × width | Area, perimeter and volume calibrated to mm |
| Documentation | Free-text notes, fields missing | Dropdown-guided minimum dataset, complete by construction |
| Progression | Compare photos side by side | Ghost contour overlay + RYB tissue trend per visit |
| Patient data | Photos in camera rolls | Encrypted in transit, nothing stored on the device |
Shipped and standing
- Repeatable wound measurement (area, perimeter, volume) from a phone, replacing ruler estimates with calibrated, editable instrument readings.
- Sub-centimetre distance stability under hand-held motion — the sensor-fusion layer held up at the bedside, not just on the bench.
- Documentation became structured and complete by construction, feeding treatment recommendations and audit-ready reporting downstream.
- Clinician-editable contours and the ghost overlay made healing progression visible per visit, with RYB tissue trends quantifying the stage.
- Zero patient data at rest on the device — the lost-phone threat class eliminated by design.
- Media infrastructure centralized into typed ImageService and AssetManager modules, removing cross-platform artifacts for good.
Notes to self
Note 01
Sensor data is a starting point, not an answer. The filtering is the product — everything clinical sits on top of that stability.
Note 02
Keep the expert in the loop. Editable results earned clinical trust faster than a perfect-looking black box would have, and they're what makes the numbers defensible.
Note 03
Structured capture beats free text everywhere it matters: completeness, comparability, and the ability to build logic downstream.
Note 04
Refactor the foundation before adding the feature; the media modules made every later screen cheaper and safer.
Note 05
React Native can host serious computer vision when native modules and a fast canvas are used where they belong.