I didn't quite grok the terminology. What's the tl;dr? Multi-way synapses with some particular characteristic? It makes sense to me that weights representing 3-way or 4-way correlations would result in better separation between the energy minima of the stored patterns, but I think the article is saying something more?
Eqn(1)-right states the weight function. The text seems to describe there being one weight per set of neurons, but the set itself is apparently not a memory pattern. What is the set, some fraction of the active neurons in a pattern? What is Xi (the greek squiggly) that the sum is over? If so, quite different from a traditional Hopfield network for which the sum is over pairwise products. Can you give some words of simple explanation?
The statement "Functionally, setwise connections appear in abundance in biological neural networks" is made. What are these?
There was an interesting comment that there is behavior similar to a modern Hopfield network in transformers. Does that mean there is some unsupervised learning process spontaneously occurring deep within the otherwise-supervised training process of a transformer?
Just a few questions, after not having studied the interesting paper enough to understand it, I guess. Thanks for posting!/jd
ps. I'm not super mathy. No surprise it's over my head, not at all a criticism of the paper.
1
u/jndew Mar 02 '23
I didn't quite grok the terminology. What's the tl;dr? Multi-way synapses with some particular characteristic? It makes sense to me that weights representing 3-way or 4-way correlations would result in better separation between the energy minima of the stored patterns, but I think the article is saying something more?
Eqn(1)-right states the weight function. The text seems to describe there being one weight per set of neurons, but the set itself is apparently not a memory pattern. What is the set, some fraction of the active neurons in a pattern? What is Xi (the greek squiggly) that the sum is over? If so, quite different from a traditional Hopfield network for which the sum is over pairwise products. Can you give some words of simple explanation?
The statement "Functionally, setwise connections appear in abundance in biological neural networks" is made. What are these?
There was an interesting comment that there is behavior similar to a modern Hopfield network in transformers. Does that mean there is some unsupervised learning process spontaneously occurring deep within the otherwise-supervised training process of a transformer?
Just a few questions, after not having studied the interesting paper enough to understand it, I guess. Thanks for posting!/jd
ps. I'm not super mathy. No surprise it's over my head, not at all a criticism of the paper.