Research Update: December 2023

In our update for December, we covered several key industry updates in AI, including developments from companies like OpenAI, Google, Apple, and Anthropic. We also introduced a new topic we're exploring around mechanistic interpretability in AI models. Additionally, we discussed the relationship between memory and cost margins for large language models, explaining why memory is the “enemy of margins.” We previewed an upcoming public webinar on talking with teens about AI.

Below you'll find key points, videos, and a pdf of slides. Please reach out with any questions.

Industry Updates:

• Amazon released an AI assistant “Alexa” that was found to have severe hallucinations and data leaks.
• Elon Musk’s X AI released “Grok” which he acknowledged will likely have problems.
• Anthropic released Claude v2, doubling the context window while claiming to reduce hallucinations.
• The Mistral.ai startup released an open source language model trained on sparse expert models, aimed at efficiency.
• Multiple companies like IBM and Meta launched an AI alliance for advancing open science.

Key Companies:

OpenAI:

• CEO fired and rehired by the board within days, showing instability.
• Released GPT-4 Turbo model which is smaller, cheaper, and more up-to-date.
• Users found ways to access private training data through the model.
• Allowed users to create customized models but data could still be easily accessed by others.
• Diversity issue—new board is all white men (for now).
• Key takeaway: Volatile leadership, while profits are prioritized over responsibility. Models remain impressive but suffer bugs and data leaks.

Google:

• Released Gemini, a next-gen multi-modal foundation model surpassing GPT-4 in key benchmarks.
• Gemini uses reasoning to understand user intent and generate interfaces dynamically.
• Reestablishes Google as a key player with impressive innovation.
• Key takeaway: Gemini establishes Google as a leader again with innovations in multimodal reasoning and dynamic workflows.

Apple:

• Quietly released developer tools to accelerate open source AI models on Apple silicon.
• Focused approach fits Apple’s strategy of enabling developers to use AI locally.
• Key takaway: Quietly accelerating open source models on its chips, positioning itself in generative AI through a mobile focus.

Mechanistic Interpretability:

• An emerging method to peer inside AI models at the neural circuit level to understand how they work for purposes of safety and reducing unwanted behaviors.
• Involves identifying “features” that models aim to represent, as well as the “circuits” of algorithms that transform features.
• Autoencoders are a new approach introduced for mechanistic interpretation this year.
• Key milestone of discovering how neural generalization and “grokking” emerges during training.

Memory versus Margins:

• Memory capacity increases costs for large language models exponentially due to tokens needing to be repriced each exchange.
• Examples given of how costs can jump from thousands to millions for long conversations.
• On-device AI computation offers a potential solution for reducing margins.

Talking to Teens About AI Webinar:
On December 22nd at 11 AM Pacific time, we will hold a public webinar on topics like:

• Positive and negative aspects of AI
• Using AI properly for education
• Effects on mental health
• Social implications
• Future career impacts

Please let us know if you have any other questions! We look forward to continuing the dialogue.