Transformers, LLMs, and the (R)evolution of the Moonfire Tech Stack ππ₯
DDVC #37: Where venture capital and data intersect. Every week.
πΒ Hi, Iβm Andre and welcome to my weekly newsletter, Data-driven VC. Every Thursday I cover hands-on insights into data-driven innovation in venture capital and connect the dots between the latest research, reviews of novel tools and datasets, deep dives into various VC tech stacks, interviews with experts and the implications for all stakeholders. Follow along to understand how data-driven approaches change the game, why it matters, and what it means for you.
Current subscribers:Β 9,760, +346 since last week
Brought to you by VESTBERRY - A tool designed specifically for data-driven VCs
Request your FREE demo account to explore the potential of establishing live data links from over 300 diverse data sources, apps, and platforms, paving the way for distinctive VC insights.
Iβm incredibly excited to have one of the top 20 thought leaders from the βData-driven VC Landscape 2023β contribute todayβs guest post. An extremely talented engineer who founded an infrastructure analytics company and previously worked at Etsy and Facebook, Mike Arpaia (the nice guy on the left in the photo below with his data team) is a Partner at Moonfire.
For those who donβt know, Moonfire is an innovative UK-based venture firm that understands itself as a tech team doing venture, not the other way around. Just yesterday, they announced a fresh $ 115m in capital across a second-generation $ 90m early-stage fund and a $ 25m growth opportunity fund. They are known as one of the most data-driven VC firms leveraging a modern tech stack and AI across the value chain.
Thank you, Mike, for sharing this unique behind-the-scenes look and your valuable thoughts on the impact of AI on VC ππ»
In this article, we delve into the journey of deep learning and its impact on the landscape of artificial intelligence (AI), with a specific focus on the venture capital sector. We'll explore the evolution of transformer models, such as GPT-4, that have become pivotal in today's state-of-the-art AI.
Through the lens of our technology stack here at Moonfire, we'll examine how these innovations are streamlining processes and aiding decision-making in venture capital. This article serves as an exploration of where we are, how we got here, and the potential direction of AI in venture capital, offering a pragmatic glimpse into an evolving future.
Transformers
The βstate-of-the-artβ (SOTA) in AI has shifted and evolved over the course of history. Deep learning-based approaches (neural approaches) to AI first became dominant in vision. When the convolutional neural network AlexNet was released in 2012, it not only popularised deep learning in computer vision, which dominated the domain for years but sparked a deep learning revolution.
Then came βAttention is All you Needβ β the famous, and aptly named, 2017 paper that introduced the transformer model to the world. This model enabled the training of state-of-the-art models for language tasks, like translation and summarisation, without needing sequential data processing. All the powerhouse models today β GPT-4, BERT, PaLM β are transformers, using the same architecture but trained in different ways to do different things.
And thatβs set us off on this crazy trajectory. I got really excited about transformers quite early on and ever since then, Iβve been using them to play around with professional data like people, companies, learning content, etc. Transformers are really good at analyzing language, and thereβs a lot of language in professional data. I worked on this problem space of deep learning for natural language applied to the professional domain at Workday and I got a feel for how powerful these models were for dealing with this sort of thing. At Moonfire, I wanted to take this tech and apply it to VC.
My colleague Jonas and I have already written about how weβre using text embeddings at Moonfire to find companies and founders that align with our investment thesis, but weβve been consistently working towards a grander vision for where we want to take AI in venture.
For the first two and half years at Moonfire, we basically partitioned the venture process into all of its various components, like sourcing, screening, company evaluation, founder evaluation, portfolio simulation, and portfolio support, similar as described in Andreβs article here. Given that decomposition of the problem space, we then built transformer-based models to focus on the specific tasks within that ecosystem.
Weβve gotten pretty good at that! But then GPT-4 came along. We were already using GPT-3.5, but GPT-4 was a step change. You can ask it a question and the output is often better than a smaller model trained for that specific task.
GPT-4 was a significant leap in performance
For example, we have one small model in our infrastructure called the venture scale classifier, which aims to answer the question, βIs this company a venture-scale business?β So is it a business that is venture investable from the perspective of growth, type of business, and stuff like that? Itβs a subtle problem and itβs challenging to get a lot of useful training data. But itβs an important model for us because itβs where we filter out a lot of the noise early on. So, if we can make iterative improvements to this model, it has a meaningful effect.
Jonas spent a quarter working on this model, before GPT-4, and improved it by 5%, which was incredible. Then GPT-4 comes out, he spends an afternoon with it, and gets a 20% improvement on top of that. We decided then and there that a critical objective for us would be to go through the whole stack and try that for every model.