How to best set up AI organizationally? Specialists or embedded in the team?

Chris, how to best set up AI organizationally? Specialists or embedded in the team?

This is highly dependent on the stage of the company, as well as the nature of the technical stack and product of the technical team. For example, for larger tech companies that make products for a large audience, then most Eng will need to use AI (clearly not who is asking this question though). Alternatively, a very large more traditional company (e.g. a large financial services company) that has a sophisticated internal development team to build custom solutions for the business units is likely to benefit from engagement teams with cross-functional skills (PM, UX, Data Scientist, AI Eng, F/E Eng, B/E Eng).

However, let’s assume that the person asking this question is from an early or mid-stage growth tech company with say 10-300 engineers. They probably have a product with PMF they are looking to expand from a single product to a richer product line. The reality is that engineers skilled in AI are in high demand, and will be expensive, so as the question implies, we need to think of them as a specialized resource and ask how they can have the most leverage on the product. We want to ensure we get the most ROI from this sparse resource, which means they need to work on hard problems that are difficult for competitors to emulate and/or that have large impact for the business.

I believe that this leverage will best come from building hard and novel horizontal capabilities that can be leveraged across the product and company. For example, if you are selling software to help with supply chain management, then if you build a model that can suggest new production plans in response to changes in material delivery, you could use this model in multiple places across the product (e.g. in a module for counterfactual scenario planning; in a module to test the robustness of plans/inventory/supply chain; in an app for responding to an emergency material shortage). These hard problems require a team of AI specialists to be able to sustain focus on a single problem for months, especially when it comes to using LLMs because we still don’t have a great playbook for the use of this technology and the technology is changing rapidly, necessitating a lot of trial and error.

Furthermore, the more experienced AI Eng managers capable of guiding the solving of hard problems are especially expensive, and another benefit of having a dedicated team of AI specialists is that it can be staffed with more junior AI Engs and a senior leader. If one took the approach of spreading the Eng out across teams, each AI Eng would need to be more capable (and thus higher cost per capita / much harder to hire given unprecedented demand).

However, more work needs to be done in order to ensure this team of AI specialists can have the most impact and leverage. The most important thing to focus on is that this AI team is prioritizing the right problems. For example, making decisions based on technical feasibility, as well as cross-company impact. The AI specialist team will need a PM who understands the capabilities of AI technology (what is easy vs hard to do), and who can work with other PMs and executives across the company to prioritize problems for the AI team and define what capabilities to build. The other thing to be mindful of is that by virtue of working on horizontal capabilities, this AI team will be one step removed from the actual product and users – they will be unable to build solutions independently because they lack the F/E and B/E Eng and UX; will struggle to collect good feedback and build intuition on what to build next; and could be demotivated because it will be harder to measure and be celebrated for their impact. This can be addressed by forming limited duration teams (AI specialist team + standard product development team) to jointly solve a problem and share in the credit. For example, in the supply chain example above, I would create a limited project between the AI specialist team and the team in charge of the scenario planning module to develop the horizontal capability with that module as the example use case, and once that was launched, move the AI team on to deploying the capability in other products.