Multimodal AI assistant for datacenter engineers. Real-time visual reasoning through Meta Ray-Ban smart glasses, grounded in your infrastructure APIs. Zero hallucination.
Helios combines real-time computer vision, voice interaction, and datacenter API integration into a hands-free assistant that rides on your face.
Sees through Meta Ray-Ban glasses at 1fps. Identifies servers, reads LEDs, recognizes cable types, and inspects rack health in real-time via Gemini 2.5 multimodal.
Techs have their hands full. Helios speaks and listens through the glasses -- no phone, no screen, no extra hardware. Natural conversation with automatic speech detection.
Pulls live device inventory, rack positions, cable maps, and IP assignments from your NetBox instance. Every answer is grounded in your actual infrastructure data.
Real-time CPU temperatures, fan speeds, power consumption, and system health from Redfish BMC APIs. Correlates visual cues with telemetry data for accurate diagnosis.
Continuous JSON state engine tracks task progression, urgency levels, and time-to-action predictions using linear regression. No context is ever lost between frames.
One-way video stream from glasses to a browser dashboard via WebRTC. Remote teams see exactly what the technician sees with room-code based signaling.
From photons hitting the glasses camera to voice response in the technician's ear -- every hop in under 100ms.
AVAudioEngine pipelineHelios continuously correlates what the camera sees with live telemetry from Redfish and NetBox. When a server overheats, a fan fails, or a cable is misrouted -- the technician knows immediately through voice and visual alerts.
Urgency escalates linearly; immediate voice alert at 0.9+
Exact U-position, device name, and health status per slot
Camera observations cross-referenced with Redfish telemetry
Gemini outputs a JSON state object with every response. The app's TaskStateManager tracks urgency, predicts time-to-action, and detects acceleration -- all grounded in what the camera actually sees.
{
"task": "inspect rack A-07 cooling",
"work_type": "datacenter",
"stage": "diagnosing_thermal_issue",
"urgency": 0.92,
"cues": [
"amber LED on U12 server",
"fan 3 audibly slower",
"heat shimmer above chassis"
],
"action": "replace fan module immediately",
"seconds_est": 30,
"confidence": 0.97
}
The urgency field (0.0 - 1.0) drives the AI's communication style and the app's alert behavior. Linear regression predicts how fast urgency is climbing.
Brief positive confirmation. "All looks good here."
Direct callout with specifics. "U12 is running warm, 71 degrees."
Immediate alert. "Stop -- that server needs to be shut down now."
Helios ships with two domain-specific AI configurations. Each domain has its own system prompt, state schema, and urgency model.
Infrastructure monitoring and technician guidance powered by NetBox inventory and Redfish health data.
Real-time sous-chef guidance with visual state tracking, countdown timers, and anti-burn safeguards.
From putting on the glasses to getting an AI-powered answer -- every step in the pipeline.
Meta Ray-Ban smart glasses pair via the Meta DAT SDK. Video streams at up to 720x1280 in HEVC/H.264.
The iOS app decodes video frames and sends 1 JPEG per second (50% quality) to the Gemini WebSocket alongside 16kHz audio.
Each frame is annotated with previous state JSON, urgency predictions, datacenter inventory, and health telemetry from NetBox and Redfish.
Gemini 2.5 Flash processes the multimodal input and returns audio speech, text transcription, structured JSON state, and optional tool calls.
24kHz PCM audio plays through the glasses speakers. The technician hears the AI's assessment hands-free while continuing work.
The TaskStateManager updates urgency, predicts time-to-action via linear regression, and feeds this back into the next frame's context.