So OpenAI is the leading AI company and of course their recent iteration of models the 01 series is by far the most advanced AI that we currently have access to. Now incredibly this AI model has been shrouded with secrecy to the point that if you ever dare to ask the model what it was thinking about during the process it was giving you a response, the model gives you a response where it tells you to never ask a question like that again and if you do it too many times you can actually get banned. from using OpenAI service.
And now the reason that this is shrouded in so much secrecy is because this is a big step towards AGI and many are thinking that OpenAI are quite likely to be the first company to achieve it. Now, with that being said, many have wanted to know exactly how this system works and there have been many different ways. OpenAI have, of course, published a few different publications, but nothing to the point where we truly understand what's going on beneath the hood. However, there has been a... recent research paper from a group of researchers in china and we are now asking ourselves if they just managed to crack the code did they figure out how o1 works and release a roadmap to build something similar so this is the paper scaling of search and learning a roadmap to reproduce o1 from reinforcement learning perspective and this is the paper that could change everything because if this is true that it means the playing field is leveled and it means it's only a matter of time before many other companies start to produce their AI models that are going to be on par with OpenAI.
Now, I'm actually going to break this down into four parts, but let's actually first understand the basics of how this AI thing even works. So one of the first things that we do have is we have, of course, reinforcement learning with AI. So essentially, we can use a game analogy.
So imagine you're trying to teach a dog a trick. So you would give this dog a treat, which is the reward when it does something right. And it then learns to repeat those actions to get more treats. And that is basically reinforcement learning.
Now, with AI, the dog is essentially a program and the treat is a digital reward. And the trick could be anything from winning a game to writing code. Now, why is reinforcement learning important for the 01 series? And this is because OpenAI seems to believe that reinforcement learning is the key to making 01 so smart.
It's basically how 01 learns to reason and solve complex problems through trial and error. Now, there are four pillars. of this according to the paper. You can see right here, they give us an overview of how O1 essentially works.
We've got the policy initialization. This is the starting point of the model. This sets up the model's initial reasoning abilities using pre-training or fine-tuning. And this is basically the foundation of the model.
We've got reward design, which is, of course, how the model is rewarded, which we just spoke about. I'm going to speak about that in more detail. And then, of course, we've got search, which is where during the inference time where the model is, quote, unquote, thinking.
This is how the model searches through different possibilities. And of course, we have learning. And this is where you improve the model by analyzing the data generated during the search process. And then you use different techniques such as reinforcement learning to make the model better over time. And essentially, the central idea is reinforcement learning.
And the core mechanism ties these components together. The model, which is the policy, interacts with its environment. Data flows from search results into the learning process. and the improved policy is fed back into the search creating a continuous improvement loop and the diagram basically emphasizes the cyclic nature of the process search generates data for learning learning updates the policy and yada yada so if we want to actually understand how this works we have to actually understand the policy so this is the basics this is the foundation of the model so imagine you're basically teaching someone to play a complex game like chess you wouldn't throw them into a match against a grandmaster on their first day right you'd start by teaching them the basics how the pieces move, basic strategies, and maybe some common opening moves. That's essentially what policy initialization is for AI.
Now, in the context of a powerful AI like O1, policy initialization is essentially giving the AI just a very strong foundation and reasoning before it even starts trying to solve really hard problems. It's about equipping it with a basic set of skills and knowledge that it can then build upon through reinforcement learning. The paper suggests that for O1, this head start likely comes in two main phases number one the pre-training which we can see here which is you know where you train it on massive text data think of this like letting the ai read the entirety of the internet or at least a huge chunk of it and by doing this the ai learns how language works how words relate to each other and gains a vast amount of general knowledge about the world think of it like learning grammar vocabulary the basic facts before trying to write a novel and it will also learn basic reasoning abilities by training on this data and then this is where we get to the important bit which is where we get the fine tuning with instructions and human-like reasoning and this is where we actually the AI more specific lessons on how to reason and solve problems. And this involves two key techniques, which we can see right here, prompt engineering and supervised fine tuning. So prompt engineering is where essentially, you know, you give the AI carefully crafted instructions or examples to guide its behavior.
And the paper mentions behaviors like problem analysis, which is where you restate the problem to make sure it's understood. Task decomposition, like breaking down a complex problem into smaller, easier steps, which is where you literally say, you know, first think step by step. And of course, with supervised fine tuning, which is right here, SFT, this involves training the AI on examples of human solving problems, like basically showing it the right way to think and reason.
It could involve showing examples of experts explaining their thought process step by step. So in a nutshell, policy initialization is about giving AI a solid foundation and language, knowledge and basic reasoning skills, setting up. setting it up for success in the later stages of learning and problem solving.
And this phase of O1 is essentially crucial for developing human-like reasoning behaviors in AI, enabling them to think systematically and explore solution spaces efficiently. Next, we get to something super interesting. This is where we get to reward design.
So this image that you can see on the screen illustrates two types of reward systems used in reinforcement learning. Outcome reward modeling, which is ORM over here. And then we've got process reward modeling, which is PRM. Now, as for the explanation, it's actually pretty straightforward.
So outcome reward modeling is something that only evaluates the solution based on the final result. So if the final answer is incorrect, the entire solution is marked as wrong, even if the steps right here, or even if most steps are correct. And in this example, there are some steps that are actually correct, but due to the fact that the final output is incorrect, the entire thing is just marked as wrong. But this is where we actually use process reward modeling, which is much better. So with process reward modeling, this evaluates each step in the solution individually.
This is where we reward the correct steps and we penalize the incorrect ones. And this one actually provides more granular feedback, which helps guide improvements during training. So we can see that steps one, two and three are correct and then they receive the rewards. And steps four and five are incorrect and are thus flagged as errors. And this approach is far better because it pinpoints the exact errors in the process rather than discarding the entire solution.
And this diagram basically emphasizes the importance of process rewards in tasks that involve multi-step reasoning, as it allows for iterative improvements and better learning outcomes, which is essentially what they believe O1 is using. Now, this is where we get into the really interesting thing, because this is where we get to search. And many have heralded search as the thing that could take us to super intelligence in fact i did recently see a tweet that just stated that i'm sure i'll manage to add that on screen so when we decide to break this down this is essentially where we have the ai thinking so you know when you have a powerful ai like a one it needs time to think to explore different possibilities and find the best solution this thinking process is what the paper refers to as search so thinking more is where they say that you know one way you could improve the performance is by thinking more during the inference, which means that instead of just generating one answer, it explores multiple possible solutions before picking the best one. So, you know, let's say you think about writing an essay. You don't just write the first draft and submit it, right?
You brainstorm ideas, you write multiple drafts, you revise and edit until you're happy with the final product. And that is essentially a form of search too. So there are two main strategies that are in the search area and the paper highlights these strategies that O1 might be using. for this thinking process so coming in at number one we have the tree search so imagine a branching tree where a branch represents a different choice or you know action that the ai could potentially take research is like you know exploring the tree following different paths to see where they lead for example in a game of chess an ai might consider all the possible moves that it could make then all the possible responses its opponent could make and then build on this tree of possibilities and then it uses a certain kind of criteria to decide which branch to explore further and which to prune focusing on the most promising path.
Basically thinking about where you're going to go, what decisions you're going to make, and which one yields the best rewards. It's kind of like a gardener selectively trimming branches to help a tree grow in the right direction. A simple example of this is best of N sampling where the model generates N possible solutions and then picks the best one based on some kind of criteria. Now on the bottom right here, this is where we have sequential revisions. This is like writing that essay we talked about earlier and the AI starts with an initial attempt.
that a solution, then refines it step by step along the way, making improvements. For example, an AI might generate an initial answer to a math problem, and then it might check its work, then identify the errors, and then revise its solution accordingly. It's kind of like editing your essay, catching the mistakes, and then making it better with every time you review it. So you have to also think about, you know, how does the AI decide?
which paths to explore in the tree search or how to even, you know, revise the solution in sequential revision. So the paper mentions two types of guidance. So we have internal guidance, and this is where you've got the AI using its own internal knowledge and calculations to guide its search. And one example is, of course, you know, model uncertainty.
And this is where the model can actually estimate how confident it is in certain parts of its solution. It might focus on areas where it's less certain, exploring alternatives or making revisions. It's kind of like double checking your work when you're not really sure if you've made a mistake.
Another example of this is, of course, you know, self-evaluation. This is where, you know, the AI can be trained to assess its own work, identifying potential errors or areas for improvement. It's kind of like having an internal editor that reviews your writing and suggest changes.
Then we've got external. Guidance, and this is like getting feedback from the outside world to guide the search. So one example is environmental feedback, which is where in some cases, AI can interact with a real or simulated environment and get feedback on its actions.
For example, a robot learning to navigate a maze might get feedback on whether it's moving closer to or farther from the goal. And another example of this is using a reward model, which we discussed earlier. The reward model can provide feedback on the quality.
of different solutions or actions guiding the AI towards better outcomes. It's kind of like having a teacher who grades your work and tells you what you did well and tells you where you need to improve. In essence, the search element and the process by which O1 explores different possibilities and refines its solution is guided by both its internal knowledge and its external feedback.
And this is a crucial part of what makes O1 so good at complex reasoning tasks. So of course, search is how the AI thinks about a problem, but how does it actually get better at solving problems over time? This is where learning comes in. So the paper suggests that O1 uses a powerful technique called reinforcement learning to improve its performance.
So search generates the training data. So remember how we talked about search generating multiple possible solutions? Well, those solutions, along with the feedback from internal or external guidance, become value training, become valuable training data for the AI. Think of it like a student practicing for an exam.
They might try and solve many different practice problems, getting feedback on their answers and learning from their mistakes. Each attempt, whether successful or not, provides valuable information that actually helps them learn and improve. Now, we've got two main learning methods and the paper focuses on two main methods that O1 might be using to learn from during this search generated data. Number one is policy gradient methods like PPO and these methods are a little bit more but the basic idea is that the AI adjusts its internal policy, which is the strategy for choosing its actions based on the reward that it achieves.
And actions that lead to high rewards are made more likely, while actions that lead to low rewards are made less likely. It's kind of like fine-tuning the AI's decision-making process based on its own experiences. Then we've got PPO, which is essentially Proximal Policy Optimization, which is a popular policy gradient method that is known for its stability and efficiency. It's like having a careful and methodical way of updating the AI strategy, making sure it doesn't change too drastically in its response to any single experience. Then, of course, here we have behavior cloning.
And this is a simpler method where the AI learns to mimic successful solutions. It's like learning via imitation. If the search process finds a really good solution, one that gets a high reward, the AI can learn to copy that solution in similar situations. It's like a student learning to solve a math problem.
So, by studying a worked example and the paper suggests that o1 might use behavior cloning to learn from the very best solutions found during search effectively adding them to its repertoire of successful strategies or it could be used as an initial way to warm up the model before using more complex methods like ppo now of course we've got iterative search and learning and the real power of this approach comes from combining search and of course learning in an iterative loop so the ai searches for solutions, learns from the results, then uses improved knowledge to conduct even better searches in the future. It's like a continuous cycle of practice, feedback, and improvement. And the paper suggests that this iterative progress is key to O1's ability to achieve superhuman performance on certain tasks. By continuously searching and learning, the AI can surpass the limitations of its initial training data, potentially discover new and better solutions than humans.
haven't thought of. So with all that being said about how O1 works, and now that you know the basics, the four key pillars, do you guys think we are close to superintelligence? After reading this research paper and understanding the key granular details about how O1 works, I think I really do understand why the wider AI community is saying that superintelligence isn't that far away. If an AI can search for solutions, then learn from those results and use that improved knowledge to conduct even better searches in the future.
having a continuous cycle of practice feedback and improvement achieving superhuman performance would be possible in theory so maybe artificial super intelligence isn't that far away with that being said i'd love to know your thoughts and hopefully you guys have a