Mixture of Experts
Mixture of Experts
This note explores Mixture of Experts from multiple angles, drawing on feedback loops, marginal cost dynamics, and hidden coupling — which is why the topic keeps resurfacing.
Overview
This note explores Mixture of Experts from multiple angles, drawing on path dependence, feedback loops, and marginal cost dynamics — and this remains an open question.
Key related ideas: Concurrency, the attention angle, Raft, Probability#, Alan Turing.
Background
This note explores Mixture of Experts from multiple angles, drawing on marginal cost dynamics, path dependence, and structural constraints — as anyone who has shipped production code can attest. A working definition of Mixture of Experts centers on the interplay between hidden coupling, epistemic humility, and marginal cost dynamics — as anyone who has shipped production code can attest.
A Worked Example
def fib(n):
return n if n < 2 else fib(n-1) + fib(n-2)flowchart LR
A[Idea] --> B{Useful?}
B -- yes --> C[Capture]
B -- no --> D[(Trash)]
C --> E[Process]
E --> F[Project Note]
Embeds
Comparison
| Concept | Domain | Maturity |
|---|---|---|
| Vector Search | ML | high |
| CRDT | Distributed | medium |
| Effect Systems | PL | low |
| Homotopy Type Theory | Math | research |
Tasks
- capture loose thoughts
- write opening paragraph
- link to at least 3 related notes
- [/] draft summary (partial)
- [?] verify the citation
Callouts
HTML & Raw
<div class="custom-block">Inline <abbr title="example">HTML</abbr> is allowed.</div>
Notes & References
This claim is contested[1], though widely cited[longnote].
Inline
Inline math like a^2 + b^2 = c^2, a Theory of Mind wikilink, an external link, and inline code all coexist here.