DeepThoughts on Evidence for the Scaling Hypothesis

If this is your first time here, take a second to click?subscribe! Feel free to send questions, comments, and suggestions for improvement to?[email protected].

Evidence for the Scaling Hypothesis ??

How would the output of an artificial intelligence system improve if unlimited computational resources were available? Following the scaling hypothesis put forward by Gwern Branwen, it improves following a power law distribution, eventually reaching human or even super-human levels.?

In a recent publication, researchers have extended the scaling hypothesis to reinforcement learning which was long assumed but now there is some proof: Training AlphaZero, an AI developed by DeepMind, to play the games ‘Connect Four’ and ‘Pentago,’ the player strength improves with the number of neural network parameters following a power law, if not bottlenecked by the available computing power. This scaling hypothesis was sort of expected but not demonstrated previously. It means that the machine learning model gets smarter the more computing power it has and the larger it gets.?

Interesting: "This scaling law implies that previously published state-of-the-art game-playing models are significantly smaller than their optimal size, given the respective compute budgets."?

Implication: We can expect to see much larger reinforcement learning models with unpredictable but likely breakthroughs in performance on some tasks. It’s another driver for energy-efficient hardware (not that we needed another one) and increases confidence that “huge models” won’t be limited to language.

Reminder to founders ??

If you're a technologist, cryptographer, hacker, or founder building something big, we’d love to speak to you! Reach out at?[email protected]

Interesting Reads ??

How do venture capitalists make decisions? – A recent study in the Journal of Financial Economics surveyed 885 institutional venture capitalists (VCs) at 681 firms. Here you can read about the results.?

Please just tell me what you do – A rant by Evan J. Conrad: “If you're going to "revolutionize", "create the operating system for", or "build at the intersection between", the universe should pause, rewind, and let you explain again.”?

Discovering novel algorithms with AlphaTensor – DeepMind presents AlphaTensor, the first artificial intelligence system for “discovering novel, efficient, and provably correct algorithms for fundamental tasks such as matrix multiplication.”

Fun Stuff???