Now taking design partners · Patent pending
The math
checks itself.
Silent data corruption is found late and fixed by rolling back. PhaseGuard catches it at the step where it happens and puts it right while the run keeps going.
Chips get math wrong without telling anyone.
At fleet scale it happens often enough to matter. Google estimates one silent data corruption event every week or two while training Gemini. The wrong value passes every check, ends up in the weights, and the run can look healthy for weeks.
The usual fix is to notice the loss looks off, roll back to an old checkpoint and pay for the lost time again.
AMD, Arm, Google, Intel, Meta, Microsoft and NVIDIA now work on the problem together through the Open Compute Project.
Google estimate: Ma et al., Understanding Silent Data Corruption in LLM Training, Harvard and AWS, 2025. Open Compute Project, Silent Data Corruption in AI, 2025.
100% caught and put right.
We trained a model on a Google TPU and injected hardware faults throughout the run, in the math, in the data passed between layers and in the stored weights.
PhaseGuard caught every one and fixed every one. The run ended bit for bit identical to a clean run. Without PhaseGuard the same faults changed the model.
Checksum methods in use today fix one wrong value at a time. In a single multiply PhaseGuard fixed thousands. We know of no other product that does this.
How it works
Every guarded multiply carries its own checks, computed alongside the real work. A wrong value is caught at the step where it appears and corrected before the result is used.
checking
Animation of what PhaseGuard does. It does not run the method itself.
What it guards
Math in flight
Every matrix multiply in the forward and backward pass, checked as it runs.
Fixed in place
Wrong values are corrected and training keeps going. No rollback to an old checkpoint.
Between layers
Data handed from one layer to the next is checked when it is read.
Weights at rest
A flipped bit in a stored weight is found and restored bit for bit.
Checkpoints and files
Saved models and files carry their own repair data. Damaged files are repaired.
Where it runs
PyTorch on GPUs and JAX on TPUs. On quantum computers it guards the control side, tested on a 156-qubit IBM machine.
Design partners
We’re looking for a few AI training teams to run PhaseGuard on real jobs while we fit it to their stack.
What you get
- Setup on your training stack, done with your engineers
- Our engineers, whenever you need them
- Your feedback sets what we build next
- No cost during the pilot
- Access before public release
- Design-partner pricing when PhaseGuard goes on sale
What we ask
- Run it on real training jobs
- Share what it caught and what it cost in speed
- Regular check-ins while it runs