Rendered at 15:16:45 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
delichon 52 minutes ago [-]
If you want to believe that the success of Kimi is about distillation attacks, ignore this.
WhitneyLand 14 minutes ago [-]
False dichotomy right?
Are Chinese labs impressively innovating? Clearly.
However this doesn’t rule out possible gains due to distillation.
I don’t know the degree of the latter but both things could certainly be true.
api 2 minutes ago [-]
“Distillation” is just indirectly pirating the largely pirated training data used to train the original model.
“You stole my warez!”
moralestapia 48 seconds ago [-]
Well said.
The distillation theory does not even make sense as Fable was only around for days (effectively) before Kimi was released.
Aurornis 13 minutes ago [-]
You can’t build a frontier model with one single thing. This is an incremental improvement but it doesn’t explain the entire success of the model. The training set is immensely important, regardless of how you feel about distillation.
Parfait__ 43 minutes ago [-]
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
fwip 32 minutes ago [-]
Anthropic is claiming that training an LLM to mimic another LLM is materially different and worse than slurping up stuff written by humans (even if that material is stolen).
Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
blints 15 minutes ago [-]
Their claim is even stronger than that, they have complaints about their models being used as a validation step for other model output, which is standard practice in the industry.
verdverm 16 minutes ago [-]
Google is apparently taking a different stance and offering distillation as a paid product
>To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.
This is just awesome.
pooyamo 14 minutes ago [-]
Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled?
Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them, but in the modern LLM scene it seems people are in a race to scaling up, experimenting empirically and hoping the same algorithm/architecture comes to a solution.
IanCal 1 minutes ago [-]
It might be that what we consider a basic and very hard puzzle are extremely close together on a more absolute scale. The difference is often for us what proportion of humans can solve it. And the low end of that is still quite high up - animals that can solve things that are very basic for the vast majority of humans are pretty rare and known about, yet are capable of quite complex actions and learning and aren’t wildly different in scale of neurons to us.
Going from 1m to 1T params is also a scaling of a million times. It’s like going from a human brain down to one percent in size in each direction or just a few mm.
jlamberts 3 minutes ago [-]
This is actually a well-known phenomenon in ML, called "The Bitter Lesson".
> One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
Let's say there's some circuit that does problem solving of the kind we call intelligence.
We dont know what this circuit looks like, but it exists in our brain.
Doing regression on outputs from the brain (e.g. internet text) with enough parameters, we can "fit" our model to this circuit.
But if you try to fit it with fewer parameters than it needs, you're just going to get some linear approximation.
pornel 5 minutes ago [-]
IANAMLE, but there is "grokking" that makes models learn to actually generalize, even after you give them enough parameters that would let them memorize the dataset:
High-dimensional gradient descent behaves very differently than the simplified 3d visualisations we use to demonstrate it, and has lots of ways out of local minima:
it's certainly not a definite procedure for determining if an arbitrary mathematical statement is true or not. it's more like educated guess and check which definitely scales up
senko 51 minutes ago [-]
Old but relevant: if you read the recently-released Kimi K3 paper[0], you'll see that it's heavily based on Kimi Linear discussed here, scaling it up and adding a bunch more things (like native vision and RL improvements).
I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.
jasonjmcghee 2 hours ago [-]
(2025)
As it's 9 months old and they just had a major model release
The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.
mcbuilder 1 hours ago [-]
Compared to the Opus 5 "model card", which read like a standard Anthropic set of alignment principles and safety concerns, this presents a plethora of useful technical details that advances the state of the art.
throwa356262 1 hours ago [-]
Same with DeepSeek papers, they are a joy to read.
senko 50 minutes ago [-]
Not an expert, but looks like they did a lot more work on the RL part (9 expert models, full sandbox access for agentic tasks, etc)?
Are Chinese labs impressively innovating? Clearly.
However this doesn’t rule out possible gains due to distillation.
I don’t know the degree of the latter but both things could certainly be true.
“You stole my warez!”
The distillation theory does not even make sense as Fable was only around for days (effectively) before Kimi was released.
Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
https://docs.cloud.google.com/gemini-enterprise-agent-platfo...
This is just awesome.
Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them, but in the modern LLM scene it seems people are in a race to scaling up, experimenting empirically and hoping the same algorithm/architecture comes to a solution.
Going from 1m to 1T params is also a scaling of a million times. It’s like going from a human brain down to one percent in size in each direction or just a few mm.
> One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
The full essay is worth a read, it's pretty short http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Let's say there's some circuit that does problem solving of the kind we call intelligence.
We dont know what this circuit looks like, but it exists in our brain.
Doing regression on outputs from the brain (e.g. internet text) with enough parameters, we can "fit" our model to this circuit.
But if you try to fit it with fewer parameters than it needs, you're just going to get some linear approximation.
https://en.wikipedia.org/wiki/Grokking_(machine_learning)
High-dimensional gradient descent behaves very differently than the simplified 3d visualisations we use to demonstrate it, and has lots of ways out of local minima:
https://www.youtube.com/watch?v=NrO20Jb-hy0
so it seems like there is a benefit to giving models more space to learn in rather than forcing them to compress the knowledge from the start.
[0] https://arxiv.org/abs/2607.24653
As it's 9 months old and they just had a major model release
The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.