TL;DR
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
Q Labs Research describes Dust, a zeroth-order method that estimates training updates by perturbing token-level activations rather than calculating gradients with backpropagation. The authors report competitive results in experiments, but say Dust approximates backprop more closely with larger populations, which require substantially more compute; the findings do not establish that it can replace backprop at large-scale training.
Q Labs Research has reported experiments with Dust, a method for pretraining transformer language models without backpropagation, the technique used to calculate gradients in most modern neural-network training. The researchers say Dust produced competitive results in their tests by perturbing token-level activations and using the resulting changes in loss to estimate updates, but they have not shown that it can replace backprop at production scale.
Dust is a zeroth-order optimization method: rather than calculating a loss gradient directly, it introduces changes to a model’s activations and uses how those changes affect the loss to estimate which update may help. The report says perturbations are applied independently at each token. This makes each token a member of a virtual population, allowing one forward pass to evaluate many perturbations in parallel instead of separately constructing and running a model for each candidate.
In its October 2026 report, Q Labs says Dust’s estimates become more closely aligned with backpropagation as the population grows, and remained aligned across the scales tested, up to 1 billion tokens. The researchers also report that a 243-million-parameter model outperformed a model 120 times smaller at most population sizes. These are findings from the authors’ experiments, not evidence that the method is generally superior to backpropagation.
The report says Dust can approximate backprop closely at large population sizes and sometimes exceed it in the settings tested. That comes with a compute trade-off: larger populations require more computation. Q Labs also estimates that, from 1 million tokens onward, Dust is roughly 1,000 to 10,000 times more efficient than a transformer implementation of EGGROLL, an evolution-strategy method. The authors describe this comparison as based on extrapolations, not a direct demonstration of equivalent cost and performance in large-scale deployments.
A Different Cost Trade-Off for Training
If the reported approach can be made practical at larger scales, it could broaden the ways researchers train neural networks by reducing reliance on differentiable computation and backpropagation. That possibility is relevant because training methods shape which models and hardware configurations researchers can use. Dust’s design also addresses a specific cost of evolution strategies: evaluating many separate weight-perturbed models. Its virtual population instead shares a forward pass across token-level activation perturbations.
But the results do not yet show a cheaper or better route to training large language models overall. The report’s own account ties closer agreement with backprop to larger populations and more compute. Its efficiency comparison is against EGGROLL and rests partly on extrapolation; it does not establish a cost advantage over conventional backprop-based training. For now, the significance is a research result about a possible alternative, not a demonstrated replacement.
Popular right now
Shark Rocket Ultra-Light Corded Stick Vacuum, HV301, Gray/Orange
Shark PowerPro Reveal Plus Cordless Vacuum Cleaner, Charcoal, IZ376H
Dyson Airwrap i.d. Multi-Styler for Straight and Wavy Hair
As an affiliate, we earn on qualifying purchases.
From Weight Search to Token Perturbations
Evolution strategies search for useful parameter updates by changing model weights and measuring how each change affects performance. Q Labs says this approach becomes expensive as the population grows, since candidate models must be evaluated. Dust follows the broader idea of search but shifts perturbations from weight space to activation space, applying them at individual tokens and pooling their loss-based feedback.
The report frames the work against the field’s dependence on backpropagation, which computes first-order gradients and has guided the design of neural-network architectures, training algorithms and hardware. Earlier node-perturbation methods also changed activations, but Q Labs presents Dust’s token-level virtual population as a way to make such search work within transformer pretraining. The report is a research account from Q Labs; the supplied material does not identify it as a peer-reviewed publication.
“We present the first zeroth-order method that is competitive with backprop at pretraining transformer language models.”
— Q Labs Research, in the report’s abstract
Scale, Costs and Independent Checks
The supplied report summary does not establish how Dust performs in a full-scale language-model training run, how its end-to-end compute and energy costs compare with backprop, or whether its reported results hold across different model designs and datasets. The authors’ statement that performance remains aligned with backprop up to 1 billion tokens describes the scales they tested; it is not proof of behavior at larger training scales.
It is also unclear how the reported comparisons were measured in detail from the supplied material, including the exact conditions behind cases where Dust exceeded backprop. The claimed efficiency advantage over EGGROLL is explicitly an extrapolation. Independent replication, full experimental details and evidence from larger runs would help clarify how broadly the findings apply.
Replication and Larger Training Runs
The next steps for evaluating Dust are independent replication and tests at larger training scales, with enough detail to compare both model quality and total compute against backprop-based baselines. Those comparisons would need to account for the population sizes Dust uses, since its gradient estimates improve as the population grows and the report links that improvement to substantially more computation.
The supplied source does not announce a release schedule, a planned external evaluation or a specific follow-up experiment. Until those details or larger-scale results are available, Dust is best understood as a proposed training method with promising reported experiments and unresolved questions about practical cost and scale.
Key Questions
What is Dust?
Dust is a zeroth-order method that estimates model updates by perturbing activations and measuring their effect on loss, rather than calculating gradients through backpropagation.
How does Dust use tokens as a population?
Q Labs says it perturbs activations independently at each token. Each token acts as a virtual population member, so a forward pass can evaluate many perturbations in parallel.
Did the researchers show Dust beats backprop?
The report says Dust sometimes exceeded backprop in the settings tested and approximated it more closely with larger populations. Those claims do not establish that Dust is broadly superior or can replace backprop in large-scale training.
How does Dust compare with EGGROLL?
Q Labs estimates Dust is about 1,000 to 10,000 times more efficient than a transformer implementation of EGGROLL from 1 million tokens onward. The report describes this as an extrapolation, not a confirmed advantage over backpropagation.
What remains to be tested?
Larger training runs and independent replications are needed to establish how Dust performs across models and datasets, and how its total compute costs compare with backprop-based training.
Source: hn
