AI Heresy 101

Written By Chris Goldman

You probably shouldn’t start . . .

But if you do, then start here

I want to talk about how AI works. I mean really works.

I want to know the underlying principles that govern AI. I don’t want to tinker anymore! I want to be able to calculate the predicted performance of a model before I train it. The same way I can use fluid dynamics to calculate the lift generated by a wing, or use quantum mechanics to predict band gap energies in semiconductors - I want to use a fundamental theory of AI to predict how well a model will perform, when it will generalize, and how to improve it.

I think we can do this.

Here are the key ideas:

  • AI systems are adaptive non-linear dynamical systems.
  • Learning can be understood as a non-linear synchronization process between the model and the system creating the training data.
  • Generalization occurs as a natural consequence of feature stability during training (think competition not optimization).

My intuition is that this will let us tackle the following:

  • Online learning
  • Model efficiency
  • Model reliability and safety

A Warning

If you are new to the field and looking for a great introduction to AI, then you should go somewhere else. Sorry, I’m not “gate-keeping” - I’m protecting you. These posts won’t explain what an embedding is or how QKV matrices work. These posts could possibly mislead the novice AI practitioner. Some of the ideas are speculative, heretical, and may even contain minor spelling errors. The last thing I want is for you to repeat this stuff in a job interview and sound like some sort of AI Flat Earther. You have been warned.

But… After you have put in the work and spent a decade or so learning the field - but still feel like something is missing, then welcome! Perhaps these ideas will be useful, or at least entertaining (in a cognitive train wreck sort of way).

If You’re Still Interested . . .

Let’s get started. Here is the outline of the topics I’m going to try and cover. Links will be added as the posts are written.

The Roadmap of Rantings

  • Justifications, Rationalizations, and Excuses

    • We Should Eventually Get Around to Understanding How AI Works
    • The Important Stuff We Are Missing
    • The Extra Parts Problem
    • Dynamics is All You Need
    • Some Background About Me

  • The Big Ideas

    • Synchronization Model of Learning
    • Mechanics of Generalization
    • The Dream of Continuous Learning

  • What is Next?

    • Quantitatively predicting generalization and “grokking”
    • Experiments with online learning
Chris Goldman