Michael Bukatin - Transformer Revolution

July 15, 2020

At the end of May 2020, the next stage of Transformer revolution began.

The field experienced a qualitative jump, with the OpenAI presentation of code generator assistant demo during Microsoft Build 2020 event on May 20, 2020 (see, e.g. https://twitter.com/matvelloso/status/1263193089310461952) and then with GPT-3 paper Language Models are Few-Shot Learners on May 28, 2020 and OpenAI API private beta on June 11, 2020.

We are in the middle of July and we have seen reports from enough people using GPT-3 via that private beta, and we can confidently say that a new epoch has started, the transition at least as consequential as AlexNet had been in September-December 2012 has just happened. It's interesting that this coincides with a period of unusually intense social, political, and economic turbulence (somehow, everything is coming to head all at once).

Technologically speaking, we are at a point when things might start changing at arbitrarily fast rate at any moment, in particular because people are ready to hybridize all kinds of approaches with Transformers, just like they hybridized all kinds of approaches with "deep" nets in recent years. As Jürgen Schmidhuber is saying in recent years, "we are almost there". Now this has literally become true.


See this for a nice collection of usage examples: https://twitter.com/xuenay/status/1283312640199196673